2024-03-24T06:10:14.976 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T06:10:14.977 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026 --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/orig.config.yaml 2024-03-24T06:10:15.032 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T06:10:15.202 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026 branch: wip-batrick-testing-20240323.003144-squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7619026' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 21b3bd672163092b4e542f772797fcba02d3d4de subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeF8n/SZuNgyZPs9XNwUg7rkA6uJdvep8icuSigElRcFLKOlLNXoAVa5l2pMWfh7LEbC34ir7PUfGKJAkySGVE= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGL7KZ8QT6pbke7ylbxXfXTJCLGVuszRe9QoGfwhyP9exiv911wQH3qxyX9WFRNUBy3v+I2/yeHuwvgIuWg0ew= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvQFjcEQjn+KQcODW9amDgNmp77l705RsazjDnv+bCR0hwxl3qwLlMi++O/VweM8c8l6R31x24acrcOB7Vy3Lw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T06:10:15.202 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T06:10:15.203 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T06:10:15.203 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T06:10:15.205 INFO:teuthology.task.internal:Checking packages... 2024-03-24T06:10:15.226 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T06:10:15.226 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T06:10:15.226 INFO:teuthology.packaging:ref: None 2024-03-24T06:10:15.226 INFO:teuthology.packaging:tag: None 2024-03-24T06:10:15.226 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T06:10:15.226 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:10:15.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T06:10:15.444 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T06:10:15.445 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T06:10:15.451 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T06:10:15.451 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T06:10:15.464 INFO:teuthology.task.internal:Saving configuration 2024-03-24T06:10:15.484 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T06:10:15.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T06:10:15.507 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.520772', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeF8n/SZuNgyZPs9XNwUg7rkA6uJdvep8icuSigElRcFLKOlLNXoAVa5l2pMWfh7LEbC34ir7PUfGKJAkySGVE='} 2024-03-24T06:10:15.526 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.521899', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGL7KZ8QT6pbke7ylbxXfXTJCLGVuszRe9QoGfwhyP9exiv911wQH3qxyX9WFRNUBy3v+I2/yeHuwvgIuWg0ew='} 2024-03-24T06:10:15.547 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.518033', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvQFjcEQjn+KQcODW9amDgNmp77l705RsazjDnv+bCR0hwxl3qwLlMi++O/VweM8c8l6R31x24acrcOB7Vy3Lw='} 2024-03-24T06:10:15.547 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T06:10:15.553 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-24T06:10:15.553 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-24T06:10:15.553 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-24T06:10:15.553 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T06:10:15.640 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7eff308dd160>, signals=[15]) 2024-03-24T06:10:15.641 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T06:10:15.647 INFO:teuthology.task.internal:Opening connections... 2024-03-24T06:10:15.647 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:10:15.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:10:15.752 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T06:10:15.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:10:15.838 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-24T06:10:15.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:10:15.924 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T06:10:15.931 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-24T06:10:15.949 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-24T06:10:15.949 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T06:10:16.006 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-24T06:10:16.007 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T06:10:16.007 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T06:10:16.007 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T06:10:16.007 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-24T06:10:16.034 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-03-24T06:10:16.052 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-03-24T06:10:16.052 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T06:10:16.109 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T06:10:16.110 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-03-24T06:10:16.133 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-24T06:10:16.151 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-24T06:10:16.151 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-24T06:10:16.209 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T06:10:16.210 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T06:10:16.210 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T06:10:16.210 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-24T06:10:16.233 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T06:10:16.243 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T06:10:16.249 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T06:10:16.249 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-24T06:10:16.251 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-03-24T06:10:16.253 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-24T06:10:16.269 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T06:10:16.274 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T06:10:16.274 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-24T06:10:16.309 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-03-24T06:10:16.311 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-24T06:10:16.423 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T06:10:16.494 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T06:10:16.507 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-24T06:10:16.507 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-24T06:10:16.507 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-24T06:10:16.507 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T06:10:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:10:16.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-24T06:10:16.825 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T06:10:16.826 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T06:10:16.826 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T06:10:16.844 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-24T06:10:16.844 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T06:10:16.844 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T06:10:16.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum upgrade -y linux-firmware 2024-03-24T06:10:19.029 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T06:10:20.545 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-24T06:10:24.053 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-24T06:10:25.796 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-24T06:10:30.113 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.1 MB 00:00 2024-03-24T06:10:31.480 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-24T06:14:55.434 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 80 kB/s | 21 MB 04:23 2024-03-24T06:15:00.509 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-24T06:15:02.773 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 1 Package 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 375 M 2024-03-24T06:15:02.784 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T06:15:09.551 INFO:teuthology.orchestra.run.smithi022.stdout:linux-firmware-20240219-143.el9.noarch.rpm 55 MB/s | 375 MB 00:06 2024-03-24T06:15:09.552 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T06:15:09.553 INFO:teuthology.orchestra.run.smithi022.stdout:Total 55 MB/s | 375 MB 00:06 2024-03-24T06:15:09.553 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T06:15:09.655 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T06:15:09.655 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T06:15:12.121 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T06:15:12.122 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T06:15:18.555 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T06:15:18.772 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T06:15:18.862 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T06:15:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T06:15:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:15:20.525 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T06:15:20.808 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-24T06:15:20.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T06:15:20.957 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-24T06:15:20.957 DEBUG:teuthology.orchestra.run.smithi022:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-24T06:15:20.960 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi022.front.sepia.ceph.com to complete... 2024-03-24T06:15:32.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24 06:15:32 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-24T06:15:32.371 INFO:teuthology.orchestra.run.smithi022.stderr:FINISHED --2024-03-24 06:15:32-- 2024-03-24T06:15:32.371 INFO:teuthology.orchestra.run.smithi022.stderr:Total wall clock time: 11s 2024-03-24T06:15:32.371 INFO:teuthology.orchestra.run.smithi022.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-24T06:15:32.373 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-24T06:15:32.374 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-24T06:15:34.408 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying... ######################################## 2024-03-24T06:15:34.461 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing... ######################################## 2024-03-24T06:15:34.547 INFO:teuthology.orchestra.run.smithi022.stdout:Updating / installing... 2024-03-24T06:15:42.413 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-24T06:15:58.576 INFO:teuthology.orchestra.run.smithi022.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T06:16:21.731 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-24T06:16:21.731 DEBUG:teuthology.orchestra.run.smithi022:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-24T06:16:21.802 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:21.802 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-24T06:16:21.802 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-24T06:16:21.802 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-24T06:16:21.802 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T06:16:21.803 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T06:16:21.804 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T06:16:21.805 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T06:16:21.806 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T06:16:21.807 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T06:16:21.808 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T06:16:21.809 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T06:16:21.810 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T06:16:21.811 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T06:16:21.812 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-24T06:16:21.813 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T06:16:21.814 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-24T06:16:21.815 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-24T06:16:21.816 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-24T06:16:21.817 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T06:16:21.818 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T06:16:21.819 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T06:16:21.820 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T06:16:21.821 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T06:16:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T06:16:21.823 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T06:16:21.824 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T06:16:21.825 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T06:16:21.826 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T06:16:21.827 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T06:16:21.828 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T06:16:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T06:16:21.830 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T06:16:21.831 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T06:16:21.832 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T06:16:21.833 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T06:16:21.834 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-24T06:16:21.835 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T06:16:21.836 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T06:16:21.837 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-24T06:16:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T06:16:21.839 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T06:16:21.840 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T06:16:21.841 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T06:16:21.842 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-24T06:16:21.844 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-24T06:16:21.844 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T06:16:21.845 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T06:16:21.846 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-24T06:16:21.847 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-24T06:16:21.848 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-24T06:16:21.849 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-24T06:16:21.850 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-24T06:16:21.851 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-24T06:16:21.852 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T06:16:21.853 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-24T06:16:21.854 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-24T06:16:21.855 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T06:16:21.856 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-24T06:16:21.857 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T06:16:21.858 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T06:16:21.859 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T06:16:21.860 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-24T06:16:21.861 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-24T06:16:21.862 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T06:16:21.863 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T06:16:21.864 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T06:16:21.865 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T06:16:21.866 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T06:16:21.867 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T06:16:21.868 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-24T06:16:21.869 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T06:16:21.870 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T06:16:21.871 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T06:16:21.872 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T06:16:21.873 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T06:16:21.874 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T06:16:21.875 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T06:16:21.876 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T06:16:21.877 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-24T06:16:21.878 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T06:16:21.879 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T06:16:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T06:16:21.881 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T06:16:21.882 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-24T06:16:21.883 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-24T06:16:21.884 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T06:16:21.885 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T06:16:21.886 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-24T06:16:21.887 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T06:16:21.888 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-24T06:16:21.889 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T06:16:21.890 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-24T06:16:21.891 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T06:16:21.892 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T06:16:21.893 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T06:16:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-24T06:16:21.895 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T06:16:21.896 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T06:16:21.897 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-24T06:16:21.898 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T06:16:21.899 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T06:16:21.900 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T06:16:21.901 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T06:16:21.902 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T06:16:21.903 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T06:16:21.904 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T06:16:21.905 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-24T06:16:21.906 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T06:16:21.907 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T06:16:21.908 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T06:16:21.909 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T06:16:21.910 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T06:16:21.911 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T06:16:21.912 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T06:16:21.913 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T06:16:21.914 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T06:16:21.915 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T06:16:21.916 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T06:16:21.917 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T06:16:21.918 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-24T06:16:21.919 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T06:16:21.920 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T06:16:21.921 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-24T06:16:21.922 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T06:16:21.923 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T06:16:21.924 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T06:16:21.925 INFO:teuthology.orchestra.run.smithi022.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T06:16:21.928 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:21.929 DEBUG:teuthology.orchestra.run.smithi022:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-24T06:16:21.952 INFO:teuthology.orchestra.run.smithi022.stdout:postinstall scriptlet (using /bin/sh): 2024-03-24T06:16:21.952 INFO:teuthology.orchestra.run.smithi022.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-24T06:16:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-24T06:16:21.952 INFO:teuthology.orchestra.run.smithi022.stdout:fi 2024-03-24T06:16:21.952 INFO:teuthology.orchestra.run.smithi022.stdout:for file in vmlinuz System.map config; do 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: fi 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:done 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:fi 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:21.953 INFO:teuthology.orchestra.run.smithi022.stdout:fi 2024-03-24T06:16:21.954 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T06:16:22.036 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:16:22.037 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T06:16:22.038 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T06:16:22.038 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:22.038 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T06:16:22.635 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-24T06:16:24.007 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T06:16:24.038 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-24T06:16:24.040 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-24T06:16:24.055 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.AGcftycfqR 2024-03-24T06:16:24.055 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AGcftycfqR 2024-03-24T06:16:24.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.AGcftycfqR 2024-03-24T06:16:24.269 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.AGcftycfqR is 6KB 2024-03-24T06:16:24.317 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.AGcftycfqR 2024-03-24T06:16:24.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T06:16:24.406 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T06:16:24.406 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T06:16:24.406 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T06:16:24.406 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-24T06:16:24.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-24T06:16:24.569 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-24T06:16:54.572 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-24T06:16:54.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T06:16:54.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:17:54.575 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T06:18:03.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T06:18:03.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:18:04.130 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T06:18:04.824 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T06:18:04.825 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T06:18:04.825 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T06:18:04.866 INFO:teuthology.orchestra.run.smithi022.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T06:18:04.866 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T06:18:04.866 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T06:18:04.866 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T06:18:05.869 DEBUG:teuthology.parallel:result is None 2024-03-24T06:18:05.869 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T06:18:05.876 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T06:18:05.876 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T06:18:05.884 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T06:18:05.887 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T06:18:05.927 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T06:18:05.934 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T06:18:05.940 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T06:18:05.940 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T06:18:05.946 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T06:18:05.948 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T06:18:06.080 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T06:18:06.086 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T06:18:06.086 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T06:18:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:18:06.103 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T06:18:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:18:06.120 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T06:18:06.125 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T06:18:06.146 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T06:18:06.196 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.217 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.228 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.247 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.307 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T06:18:06.311 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T06:18:06.319 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T06:18:06.319 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T06:18:06.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T06:18:06.327 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T06:18:06.396 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T06:18:06.406 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T06:18:06.406 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T06:18:06.412 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T06:18:06.415 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T06:18:06.455 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:06.576 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:06.660 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:06.729 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:06.826 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:18:06.826 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T06:18:06.929 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:06.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:07.030 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:07.110 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:07.158 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:18:07.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T06:18:07.228 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:07.270 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T06:18:07.330 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:07.409 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T06:18:07.460 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:18:07.461 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T06:18:07.530 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-24T06:18:07.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-03-24T06:18:07.540 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-24T06:18:07.573 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T06:18:07.609 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T06:18:07.642 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T06:18:08.073 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T06:18:08.082 INFO:teuthology.task.internal:Starting timer... 2024-03-24T06:18:08.082 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T06:18:08.093 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T06:18:08.102 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T06:18:08.103 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-24T06:18:08.209 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-24T06:18:08.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-03-24T06:18:08.268 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-03-24T06:18:08.270 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-24T06:18:08.333 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-24T06:18:08.335 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T06:18:08.335 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-24T06:18:08.384 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-24T06:18:08.384 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-03-24T06:18:08.404 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-03-24T06:18:08.404 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-24T06:18:08.428 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-24T06:18:08.428 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-03-24T06:18:08.428 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T06:18:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:18:08.507 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T06:18:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:18:08.537 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T06:18:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:18:08.567 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T06:18:08.568 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-24T06:18:08.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-03-24T06:18:08.689 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-24T06:18:08.729 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T06:18:08.742 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T06:18:08.750 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T06:18:08.896 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T06:18:08.910 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T06:18:08.911 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi143.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T06:24:22.205 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-03-24T06:24:22.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T06:24:22.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:24:22.354 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T06:24:22.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T06:24:22.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T06:24:22.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:24:22.605 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-24T06:24:22.675 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T06:24:22.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T06:24:22.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T06:24:22.765 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-24T06:24:22.834 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T06:24:22.835 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T06:24:22.845 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T06:24:22.845 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T06:24:22.846 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T06:24:22.852 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T06:24:22.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T06:24:22.855 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T06:24:22.855 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T06:24:22.888 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T06:24:22.906 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T06:24:22.909 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T06:24:22.926 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T06:24:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-03-24T06:24:22.951 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-03-24T06:24:22.953 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T06:24:22.957 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-24T06:24:22.968 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T06:24:22.971 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-24T06:24:22.984 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T06:24:22.988 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T06:24:23.004 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T06:24:23.007 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T06:24:23.028 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T06:24:23.031 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T06:24:23.031 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T06:24:23.031 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.031 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.031 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.032 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.048 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-24T06:24:23.051 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T06:24:23.051 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-24T06:24:23.052 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.052 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.052 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.052 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.102 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-24T06:24:23.146 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-24T06:24:23.200 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T06:24:23.256 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T06:24:23.399 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.409 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T06:24:23.411 INFO:teuthology.run_tasks:Running task install... 2024-03-24T06:24:23.420 DEBUG:teuthology.task.install:project ceph 2024-03-24T06:24:23.420 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T06:24:23.420 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T06:24:23.420 INFO:teuthology.task.install:Using flavor: default 2024-03-24T06:24:23.426 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T06:24:23.426 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T06:24:23.426 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T06:24:23.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:24:23.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T06:24:23.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:24:23.430 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T06:24:23.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:24:23.577 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T06:24:23.577 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T06:24:23.581 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T06:24:23.581 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T06:24:23.647 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T06:24:23.648 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T06:24:23.751 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T06:24:23.751 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:24:23.752 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T06:24:23.769 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T06:24:23.770 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:24:23.770 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T06:24:23.774 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T06:24:23.774 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:24:23.774 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T06:24:23.782 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T06:24:23.782 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T06:24:23.804 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T06:24:23.804 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T06:24:23.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T06:24:23.872 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T06:24:23.873 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T06:24:23.875 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T06:24:23.933 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T06:24:23.957 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T06:24:24.001 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-03-24T06:24:24.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-03-24T06:24:24.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T06:24:24.025 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-24T06:24:24.026 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-24T06:24:24.163 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T06:24:24.238 INFO:teuthology.orchestra.run.smithi143.stdout:74 files removed 2024-03-24T06:24:24.255 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2024-03-24T06:24:24.260 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2024-03-24T06:24:24.262 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-24T06:24:24.265 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T06:24:24.289 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T06:24:24.623 INFO:teuthology.orchestra.run.smithi022.stdout:74 files removed 2024-03-24T06:24:24.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T06:24:24.930 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 238 kB/s | 83 kB 00:00 2024-03-24T06:24:24.967 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-03-24T06:24:25.289 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-24T06:24:25.384 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-24T06:24:25.596 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-03-24T06:24:25.690 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-24T06:24:25.760 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-24T06:24:26.162 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T06:24:26.188 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-24T06:24:26.334 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T06:24:26.722 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-24T06:24:27.350 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T06:24:27.612 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-24T06:24:27.993 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-24T06:24:28.985 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-24T06:24:29.094 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-24T06:24:29.778 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-24T06:24:31.063 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-03-24T06:24:32.132 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-03-24T06:24:32.854 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-03-24T06:24:35.977 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-24T06:24:37.570 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-24T06:24:37.696 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-24T06:24:39.264 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-24T06:24:40.096 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 21 MB 00:02 2024-03-24T06:24:41.051 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-24T06:24:45.187 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 6.4 kB/s | 1.7 kB 00:00 2024-03-24T06:24:45.512 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-24T06:24:45.948 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T06:24:47.515 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:24:47.516 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:24:47.528 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T06:24:47.529 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T06:24:47.530 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T06:24:47.531 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T06:24:47.532 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T06:24:47.535 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T06:24:47.536 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T06:24:47.614 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T06:24:47.638 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-24T06:24:47.638 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T06:24:47.638 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-24T06:24:47.638 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T06:24:47.638 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T06:24:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T06:24:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T06:24:47.639 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T06:24:47.641 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T06:24:47.641 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T06:24:47.641 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T06:24:47.641 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T06:24:47.641 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T06:24:47.642 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T06:24:47.643 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T06:24:47.644 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T06:24:47.645 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T06:24:47.646 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T06:24:47.647 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T06:24:47.648 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T06:24:47.648 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T06:24:47.648 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T06:24:47.648 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T06:24:47.649 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T06:24:47.649 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T06:24:47.649 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T06:24:47.649 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T06:24:47.650 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T06:24:47.651 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T06:24:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T06:24:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T06:24:47.654 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T06:24:47.655 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T06:24:47.656 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T06:24:47.657 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T06:24:47.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T06:24:47.659 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-24T06:24:47.660 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout:Install 150 Packages 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 24 Packages 2024-03-24T06:24:47.661 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:24:47.662 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 224 M 2024-03-24T06:24:47.662 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T06:24:48.279 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:24:48.279 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:24:48.293 INFO:teuthology.orchestra.run.smithi143.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T06:24:48.295 INFO:teuthology.orchestra.run.smithi143.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T06:24:48.296 INFO:teuthology.orchestra.run.smithi143.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T06:24:48.297 INFO:teuthology.orchestra.run.smithi143.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T06:24:48.299 INFO:teuthology.orchestra.run.smithi143.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T06:24:48.302 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T06:24:48.303 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T06:24:48.381 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T06:24:48.399 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T06:24:48.400 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T06:24:48.401 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T06:24:48.402 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T06:24:48.403 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T06:24:48.404 INFO:teuthology.orchestra.run.smithi143.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T06:24:48.405 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T06:24:48.406 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T06:24:48.407 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T06:24:48.408 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T06:24:48.409 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T06:24:48.410 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T06:24:48.411 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T06:24:48.412 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T06:24:48.413 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T06:24:48.414 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T06:24:48.415 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T06:24:48.416 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T06:24:48.417 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout:Install 150 Packages 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 24 Packages 2024-03-24T06:24:48.418 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:24:48.419 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 224 M 2024-03-24T06:24:48.419 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-24T06:24:48.438 INFO:teuthology.orchestra.run.smithi155.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-03-24T06:24:48.789 INFO:teuthology.orchestra.run.smithi155.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.5 MB/s | 888 kB 00:00 2024-03-24T06:24:48.847 INFO:teuthology.orchestra.run.smithi155.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 2.4 MB/s | 145 kB 00:00 2024-03-24T06:24:49.065 INFO:teuthology.orchestra.run.smithi143.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-24T06:24:49.340 INFO:teuthology.orchestra.run.smithi143.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 3.2 MB/s | 888 kB 00:00 2024-03-24T06:24:49.381 INFO:teuthology.orchestra.run.smithi155.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 4.4 MB/s | 2.4 MB 00:00 2024-03-24T06:24:49.391 INFO:teuthology.orchestra.run.smithi143.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 2.8 MB/s | 145 kB 00:00 2024-03-24T06:24:49.531 INFO:teuthology.orchestra.run.smithi155.stdout:(5/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 4.1 MB/s | 5.3 MB 00:01 2024-03-24T06:24:49.731 INFO:teuthology.orchestra.run.smithi155.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-03-24T06:24:49.966 INFO:teuthology.orchestra.run.smithi143.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 4.1 MB/s | 2.4 MB 00:00 2024-03-24T06:24:50.041 INFO:teuthology.orchestra.run.smithi143.stdout:(5/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 4.7 MB/s | 5.3 MB 00:01 2024-03-24T06:24:50.333 INFO:teuthology.orchestra.run.smithi143.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 4.1 MB/s | 1.5 MB 00:00 2024-03-24T06:24:50.365 INFO:teuthology.orchestra.run.smithi155.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 5.6 MB/s | 4.7 MB 00:00 2024-03-24T06:24:50.514 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-24T06:24:51.191 INFO:teuthology.orchestra.run.smithi143.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 4.1 MB/s | 4.7 MB 00:01 2024-03-24T06:24:52.415 INFO:teuthology.orchestra.run.smithi155.stdout:(8/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el9 4.8 MB/s | 9.8 MB 00:02 2024-03-24T06:24:52.539 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-24T06:24:52.623 INFO:teuthology.orchestra.run.smithi155.stdout:(9/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 4.7 MB/s | 21 MB 00:04 2024-03-24T06:24:52.649 INFO:teuthology.orchestra.run.smithi155.stdout:(10/174): ceph-selinux-19.0.0-1855.g21b3bd67.el 108 kB/s | 25 kB 00:00 2024-03-24T06:24:52.733 INFO:teuthology.orchestra.run.smithi143.stdout:(8/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x86 7.0 MB/s | 17 MB 00:02 2024-03-24T06:24:52.784 INFO:teuthology.orchestra.run.smithi143.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 502 kB/s | 25 kB 00:00 2024-03-24T06:24:52.824 INFO:teuthology.orchestra.run.smithi155.stdout:(11/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 5.4 MB/s | 17 MB 00:03 2024-03-24T06:24:52.849 INFO:teuthology.orchestra.run.smithi155.stdout:(12/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.1 MB/s | 219 kB 00:00 2024-03-24T06:24:52.908 INFO:teuthology.orchestra.run.smithi155.stdout:(13/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 325 kB/s | 27 kB 00:00 2024-03-24T06:24:52.924 INFO:teuthology.orchestra.run.smithi155.stdout:(14/174): libcephfs-devel-19.0.0-1855.g21b3bd67 423 kB/s | 32 kB 00:00 2024-03-24T06:24:53.016 INFO:teuthology.orchestra.run.smithi155.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 163 kB 00:00 2024-03-24T06:24:53.083 INFO:teuthology.orchestra.run.smithi155.stdout:(16/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 4.1 MB/s | 744 kB 00:00 2024-03-24T06:24:53.117 INFO:teuthology.orchestra.run.smithi155.stdout:(17/174): librados-devel-19.0.0-1855.g21b3bd67. 1.2 MB/s | 127 kB 00:00 2024-03-24T06:24:53.276 INFO:teuthology.orchestra.run.smithi155.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 2.5 MB/s | 499 kB 00:00 2024-03-24T06:24:53.276 INFO:teuthology.orchestra.run.smithi143.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 4.7 MB/s | 9.8 MB 00:02 2024-03-24T06:24:53.367 INFO:teuthology.orchestra.run.smithi143.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.3 MB/s | 219 kB 00:00 2024-03-24T06:24:53.368 INFO:teuthology.orchestra.run.smithi155.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 493 kB/s | 45 kB 00:00 2024-03-24T06:24:53.443 INFO:teuthology.orchestra.run.smithi143.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 360 kB/s | 27 kB 00:00 2024-03-24T06:24:53.476 INFO:teuthology.orchestra.run.smithi155.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.2 MB/s | 131 kB 00:00 2024-03-24T06:24:53.510 INFO:teuthology.orchestra.run.smithi143.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 476 kB/s | 32 kB 00:00 2024-03-24T06:24:53.576 INFO:teuthology.orchestra.run.smithi155.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.6 MB/s | 161 kB 00:00 2024-03-24T06:24:53.685 INFO:teuthology.orchestra.run.smithi155.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.9 MB/s | 320 kB 00:00 2024-03-24T06:24:53.710 INFO:teuthology.orchestra.run.smithi143.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 3.6 MB/s | 744 kB 00:00 2024-03-24T06:24:53.777 INFO:teuthology.orchestra.run.smithi155.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.2 MB/s | 298 kB 00:00 2024-03-24T06:24:53.799 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-24T06:24:53.802 INFO:teuthology.orchestra.run.smithi143.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 163 kB 00:00 2024-03-24T06:24:53.860 INFO:teuthology.orchestra.run.smithi155.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.2 MB/s | 99 kB 00:00 2024-03-24T06:24:53.893 INFO:teuthology.orchestra.run.smithi143.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.4 MB/s | 127 kB 00:00 2024-03-24T06:24:53.961 INFO:teuthology.orchestra.run.smithi155.stdout:(25/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 6.1 MB/s | 5.1 MB 00:00 2024-03-24T06:24:53.978 INFO:teuthology.orchestra.run.smithi155.stdout:(26/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 730 kB/s | 85 kB 00:00 2024-03-24T06:24:54.002 INFO:teuthology.orchestra.run.smithi143.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 4.5 MB/s | 499 kB 00:00 2024-03-24T06:24:54.070 INFO:teuthology.orchestra.run.smithi155.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T06:24:54.178 INFO:teuthology.orchestra.run.smithi155.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 290 kB/s | 31 kB 00:00 2024-03-24T06:24:54.287 INFO:teuthology.orchestra.run.smithi155.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.2 MB/s | 138 kB 00:00 2024-03-24T06:24:54.487 INFO:teuthology.orchestra.run.smithi155.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 5.9 MB/s | 3.1 MB 00:00 2024-03-24T06:24:54.671 INFO:teuthology.orchestra.run.smithi155.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 4.6 MB/s | 1.8 MB 00:00 2024-03-24T06:24:54.771 INFO:teuthology.orchestra.run.smithi155.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.4 MB/s | 248 kB 00:00 2024-03-24T06:24:54.855 INFO:teuthology.orchestra.run.smithi155.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 592 kB/s | 49 kB 00:00 2024-03-24T06:24:54.930 INFO:teuthology.orchestra.run.smithi155.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 204 kB/s | 15 kB 00:00 2024-03-24T06:24:55.022 INFO:teuthology.orchestra.run.smithi155.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.8 MB/s | 264 kB 00:00 2024-03-24T06:24:55.197 INFO:teuthology.orchestra.run.smithi155.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-24T06:24:55.281 INFO:teuthology.orchestra.run.smithi155.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 219 kB/s | 18 kB 00:00 2024-03-24T06:24:55.369 INFO:teuthology.orchestra.run.smithi143.stdout:(18/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-03-24T06:24:55.423 INFO:teuthology.orchestra.run.smithi155.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-24T06:24:55.427 INFO:teuthology.orchestra.run.smithi143.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 776 kB/s | 45 kB 00:00 2024-03-24T06:24:55.473 INFO:teuthology.orchestra.run.smithi155.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-24T06:24:55.494 INFO:teuthology.orchestra.run.smithi143.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.9 MB/s | 131 kB 00:00 2024-03-24T06:24:55.507 INFO:teuthology.orchestra.run.smithi155.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-24T06:24:55.557 INFO:teuthology.orchestra.run.smithi155.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T06:24:55.561 INFO:teuthology.orchestra.run.smithi143.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 2.4 MB/s | 161 kB 00:00 2024-03-24T06:24:55.590 INFO:teuthology.orchestra.run.smithi155.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 950 kB/s | 32 kB 00:00 2024-03-24T06:24:55.678 INFO:teuthology.orchestra.run.smithi143.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.7 MB/s | 320 kB 00:00 2024-03-24T06:24:55.682 INFO:teuthology.orchestra.run.smithi155.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-24T06:24:55.766 INFO:teuthology.orchestra.run.smithi155.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-03-24T06:24:55.770 INFO:teuthology.orchestra.run.smithi143.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.2 MB/s | 298 kB 00:00 2024-03-24T06:24:55.791 INFO:teuthology.orchestra.run.smithi155.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 158 kB/s | 17 kB 00:00 2024-03-24T06:24:55.817 INFO:teuthology.orchestra.run.smithi155.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-24T06:24:55.829 INFO:teuthology.orchestra.run.smithi143.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.7 MB/s | 99 kB 00:00 2024-03-24T06:24:55.842 INFO:teuthology.orchestra.run.smithi155.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T06:24:55.867 INFO:teuthology.orchestra.run.smithi155.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-24T06:24:55.887 INFO:teuthology.orchestra.run.smithi143.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-24T06:24:55.892 INFO:teuthology.orchestra.run.smithi155.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T06:24:55.918 INFO:teuthology.orchestra.run.smithi155.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-24T06:24:55.943 INFO:teuthology.orchestra.run.smithi155.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-03-24T06:24:55.968 INFO:teuthology.orchestra.run.smithi155.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T06:24:55.996 INFO:teuthology.orchestra.run.smithi155.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.5 MB/s | 187 kB 00:00 2024-03-24T06:24:56.021 INFO:teuthology.orchestra.run.smithi155.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.0 MB/s | 217 kB 00:00 2024-03-24T06:24:56.046 INFO:teuthology.orchestra.run.smithi155.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-24T06:24:56.121 INFO:teuthology.orchestra.run.smithi143.stdout:(26/174): ceph-common-19.0.0-1855.g21b3bd67.el9 2.8 MB/s | 21 MB 00:07 2024-03-24T06:24:56.139 INFO:teuthology.orchestra.run.smithi155.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-24T06:24:56.172 INFO:teuthology.orchestra.run.smithi155.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-24T06:24:56.205 INFO:teuthology.orchestra.run.smithi143.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-24T06:24:56.206 INFO:teuthology.orchestra.run.smithi155.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-24T06:24:56.231 INFO:teuthology.orchestra.run.smithi155.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T06:24:56.288 INFO:teuthology.orchestra.run.smithi143.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 379 kB/s | 31 kB 00:00 2024-03-24T06:24:56.340 INFO:teuthology.orchestra.run.smithi155.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 43 MB/s | 5.7 MB 00:00 2024-03-24T06:24:56.363 INFO:teuthology.orchestra.run.smithi143.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.8 MB/s | 138 kB 00:00 2024-03-24T06:24:56.374 INFO:teuthology.orchestra.run.smithi155.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.2 MB/s | 1.0 MB 00:00 2024-03-24T06:24:56.399 INFO:teuthology.orchestra.run.smithi155.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T06:24:56.516 INFO:teuthology.orchestra.run.smithi155.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-24T06:24:56.522 INFO:teuthology.orchestra.run.smithi143.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 4.9 MB/s | 3.1 MB 00:00 2024-03-24T06:24:56.541 INFO:teuthology.orchestra.run.smithi155.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-24T06:24:56.567 INFO:teuthology.orchestra.run.smithi155.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-24T06:24:56.592 INFO:teuthology.orchestra.run.smithi155.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-24T06:24:56.617 INFO:teuthology.orchestra.run.smithi155.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-24T06:24:56.717 INFO:teuthology.orchestra.run.smithi155.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-24T06:24:56.743 INFO:teuthology.orchestra.run.smithi155.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-24T06:24:56.768 INFO:teuthology.orchestra.run.smithi155.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-24T06:24:56.793 INFO:teuthology.orchestra.run.smithi155.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T06:24:56.819 INFO:teuthology.orchestra.run.smithi155.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-24T06:24:56.831 INFO:teuthology.orchestra.run.smithi143.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 3.8 MB/s | 1.8 MB 00:00 2024-03-24T06:24:56.844 INFO:teuthology.orchestra.run.smithi155.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-24T06:24:56.869 INFO:teuthology.orchestra.run.smithi155.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T06:24:56.894 INFO:teuthology.orchestra.run.smithi155.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T06:24:56.919 INFO:teuthology.orchestra.run.smithi155.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-24T06:24:56.945 INFO:teuthology.orchestra.run.smithi155.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-24T06:24:56.948 INFO:teuthology.orchestra.run.smithi143.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.1 MB/s | 248 kB 00:00 2024-03-24T06:24:57.014 INFO:teuthology.orchestra.run.smithi143.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 742 kB/s | 49 kB 00:00 2024-03-24T06:24:57.062 INFO:teuthology.orchestra.run.smithi155.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 338 kB/s | 39 kB 00:00 2024-03-24T06:24:57.096 INFO:teuthology.orchestra.run.smithi143.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 190 kB/s | 15 kB 00:00 2024-03-24T06:24:57.221 INFO:teuthology.orchestra.run.smithi143.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.1 MB/s | 264 kB 00:00 2024-03-24T06:24:57.262 INFO:teuthology.orchestra.run.smithi155.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-24T06:24:57.287 INFO:teuthology.orchestra.run.smithi155.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 184 kB/s | 41 kB 00:00 2024-03-24T06:24:57.312 INFO:teuthology.orchestra.run.smithi155.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T06:24:57.338 INFO:teuthology.orchestra.run.smithi155.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T06:24:57.380 INFO:teuthology.orchestra.run.smithi155.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 937 kB/s | 63 kB 00:00 2024-03-24T06:24:57.405 INFO:teuthology.orchestra.run.smithi155.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 662 kB/s | 44 kB 00:00 2024-03-24T06:24:57.430 INFO:teuthology.orchestra.run.smithi155.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T06:24:57.454 INFO:teuthology.orchestra.run.smithi143.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 3.1 MB/s | 748 kB 00:00 2024-03-24T06:24:57.456 INFO:teuthology.orchestra.run.smithi155.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T06:24:57.489 INFO:teuthology.orchestra.run.smithi155.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T06:24:57.515 INFO:teuthology.orchestra.run.smithi155.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T06:24:57.540 INFO:teuthology.orchestra.run.smithi155.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-24T06:24:57.574 INFO:teuthology.orchestra.run.smithi155.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-24T06:24:57.888 INFO:teuthology.orchestra.run.smithi143.stdout:(37/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 9.3 MB/s | 47 MB 00:05 2024-03-24T06:24:57.905 INFO:teuthology.orchestra.run.smithi143.stdout:(38/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 40 kB/s | 18 kB 00:00 2024-03-24T06:24:57.999 INFO:teuthology.orchestra.run.smithi155.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 408 kB/s | 187 kB 00:00 2024-03-24T06:24:58.005 INFO:teuthology.orchestra.run.smithi143.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 5.0 MB/s | 7.4 MB 00:01 2024-03-24T06:24:58.164 INFO:teuthology.orchestra.run.smithi143.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-24T06:24:58.191 INFO:teuthology.orchestra.run.smithi155.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-24T06:24:58.197 INFO:teuthology.orchestra.run.smithi143.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 920 kB 00:00 2024-03-24T06:24:58.223 INFO:teuthology.orchestra.run.smithi143.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-24T06:24:58.239 INFO:teuthology.orchestra.run.smithi143.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 767 kB/s | 32 kB 00:00 2024-03-24T06:24:58.258 INFO:teuthology.orchestra.run.smithi155.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-24T06:24:58.274 INFO:teuthology.orchestra.run.smithi143.stdout:(44/174): boost-program-options-1.75.0-8.el9.x8 489 kB/s | 106 kB 00:00 2024-03-24T06:24:58.298 INFO:teuthology.orchestra.run.smithi143.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 297 kB/s | 17 kB 00:00 2024-03-24T06:24:58.300 INFO:teuthology.orchestra.run.smithi155.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-24T06:24:58.357 INFO:teuthology.orchestra.run.smithi143.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-24T06:24:58.374 INFO:teuthology.orchestra.run.smithi143.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 10 kB 00:00 2024-03-24T06:24:58.399 INFO:teuthology.orchestra.run.smithi143.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-24T06:24:58.424 INFO:teuthology.orchestra.run.smithi143.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-24T06:24:58.450 INFO:teuthology.orchestra.run.smithi143.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-24T06:24:58.483 INFO:teuthology.orchestra.run.smithi143.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 578 kB 00:00 2024-03-24T06:24:58.500 INFO:teuthology.orchestra.run.smithi143.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 46 kB 00:00 2024-03-24T06:24:58.525 INFO:teuthology.orchestra.run.smithi143.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T06:24:58.550 INFO:teuthology.orchestra.run.smithi143.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-24T06:24:58.567 INFO:teuthology.orchestra.run.smithi143.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-24T06:24:58.592 INFO:teuthology.orchestra.run.smithi143.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-24T06:24:58.685 INFO:teuthology.orchestra.run.smithi143.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-24T06:24:58.726 INFO:teuthology.orchestra.run.smithi143.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 9.7 MB/s | 1.5 MB 00:00 2024-03-24T06:24:58.741 INFO:teuthology.orchestra.run.smithi155.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 58 kB/s | 25 kB 00:00 2024-03-24T06:24:58.760 INFO:teuthology.orchestra.run.smithi143.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.8 MB/s | 811 kB 00:00 2024-03-24T06:24:58.767 INFO:teuthology.orchestra.run.smithi155.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 42 kB/s | 21 kB 00:00 2024-03-24T06:24:58.785 INFO:teuthology.orchestra.run.smithi143.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-24T06:24:58.860 INFO:teuthology.orchestra.run.smithi143.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-24T06:24:58.952 INFO:teuthology.orchestra.run.smithi143.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-24T06:24:58.977 INFO:teuthology.orchestra.run.smithi143.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 405 kB/s | 78 kB 00:00 2024-03-24T06:24:59.094 INFO:teuthology.orchestra.run.smithi143.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-24T06:24:59.119 INFO:teuthology.orchestra.run.smithi143.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-03-24T06:24:59.145 INFO:teuthology.orchestra.run.smithi143.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 148 kB/s | 24 kB 00:00 2024-03-24T06:24:59.170 INFO:teuthology.orchestra.run.smithi143.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-24T06:24:59.195 INFO:teuthology.orchestra.run.smithi143.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T06:24:59.271 INFO:teuthology.orchestra.run.smithi143.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-24T06:24:59.296 INFO:teuthology.orchestra.run.smithi143.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-24T06:24:59.321 INFO:teuthology.orchestra.run.smithi143.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-24T06:24:59.338 INFO:teuthology.orchestra.run.smithi143.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-24T06:24:59.363 INFO:teuthology.orchestra.run.smithi143.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 992 kB/s | 41 kB 00:00 2024-03-24T06:24:59.389 INFO:teuthology.orchestra.run.smithi143.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T06:24:59.414 INFO:teuthology.orchestra.run.smithi143.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-24T06:24:59.439 INFO:teuthology.orchestra.run.smithi143.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T06:24:59.464 INFO:teuthology.orchestra.run.smithi143.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 632 kB/s | 47 kB 00:00 2024-03-24T06:24:59.489 INFO:teuthology.orchestra.run.smithi143.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 570 kB/s | 43 kB 00:00 2024-03-24T06:24:59.514 INFO:teuthology.orchestra.run.smithi143.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-03-24T06:24:59.648 INFO:teuthology.orchestra.run.smithi143.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 258 kB/s | 41 kB 00:00 2024-03-24T06:24:59.724 INFO:teuthology.orchestra.run.smithi143.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-24T06:24:59.873 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-24T06:24:59.890 INFO:teuthology.orchestra.run.smithi143.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-24T06:24:59.907 INFO:teuthology.orchestra.run.smithi143.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 220 kB/s | 56 kB 00:00 2024-03-24T06:24:59.932 INFO:teuthology.orchestra.run.smithi143.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 303 kB/s | 63 kB 00:00 2024-03-24T06:24:59.958 INFO:teuthology.orchestra.run.smithi143.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-24T06:24:59.983 INFO:teuthology.orchestra.run.smithi143.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-24T06:25:00.008 INFO:teuthology.orchestra.run.smithi143.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-24T06:25:00.033 INFO:teuthology.orchestra.run.smithi143.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T06:25:00.075 INFO:teuthology.orchestra.run.smithi143.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-24T06:25:00.100 INFO:teuthology.orchestra.run.smithi143.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-24T06:25:00.126 INFO:teuthology.orchestra.run.smithi143.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-24T06:25:00.184 INFO:teuthology.orchestra.run.smithi155.stdout:(97/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 6.3 MB/s | 47 MB 00:07 2024-03-24T06:25:00.266 INFO:teuthology.orchestra.run.smithi143.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-24T06:25:00.291 INFO:teuthology.orchestra.run.smithi143.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-24T06:25:00.325 INFO:teuthology.orchestra.run.smithi143.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T06:25:00.408 INFO:teuthology.orchestra.run.smithi143.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-24T06:25:00.542 INFO:teuthology.orchestra.run.smithi155.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 138 kB/s | 245 kB 00:01 2024-03-24T06:25:00.559 INFO:teuthology.orchestra.run.smithi143.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 109 kB/s | 25 kB 00:00 2024-03-24T06:25:00.834 INFO:teuthology.orchestra.run.smithi143.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 40 kB/s | 21 kB 00:00 2024-03-24T06:25:01.034 INFO:teuthology.orchestra.run.smithi143.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 516 kB/s | 245 kB 00:00 2024-03-24T06:25:01.118 INFO:teuthology.orchestra.run.smithi143.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 4.1 MB 00:00 2024-03-24T06:25:01.159 INFO:teuthology.orchestra.run.smithi155.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 351 kB/s | 216 kB 00:00 2024-03-24T06:25:01.176 INFO:teuthology.orchestra.run.smithi143.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 599 kB/s | 35 kB 00:00 2024-03-24T06:25:01.201 INFO:teuthology.orchestra.run.smithi143.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 675 kB/s | 248 kB 00:00 2024-03-24T06:25:01.209 INFO:teuthology.orchestra.run.smithi155.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 701 kB/s | 35 kB 00:00 2024-03-24T06:25:01.235 INFO:teuthology.orchestra.run.smithi155.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 236 kB/s | 248 kB 00:01 2024-03-24T06:25:01.235 INFO:teuthology.orchestra.run.smithi143.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-03-24T06:25:01.268 INFO:teuthology.orchestra.run.smithi155.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-03-24T06:25:01.302 INFO:teuthology.orchestra.run.smithi143.stdout:(103/174): qatlib-23.11.0-1.el9.x86_64.rpm 810 kB/s | 216 kB 00:00 2024-03-24T06:25:01.319 INFO:teuthology.orchestra.run.smithi143.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-24T06:25:01.344 INFO:teuthology.orchestra.run.smithi155.stdout:(103/174): openssl-devel-3.0.7-27.el9.x86_64.rp 1.6 MB/s | 4.1 MB 00:02 2024-03-24T06:25:01.353 INFO:teuthology.orchestra.run.smithi143.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 620 kB/s | 94 kB 00:00 2024-03-24T06:25:01.461 INFO:teuthology.orchestra.run.smithi143.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-03-24T06:25:01.503 INFO:teuthology.orchestra.run.smithi155.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-03-24T06:25:01.690 INFO:teuthology.orchestra.run.smithi155.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-24T06:25:01.995 INFO:teuthology.orchestra.run.smithi143.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 36 kB/s | 19 kB 00:00 2024-03-24T06:25:02.186 INFO:teuthology.orchestra.run.smithi155.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 803 kB/s | 548 kB 00:00 2024-03-24T06:25:02.219 INFO:teuthology.orchestra.run.smithi143.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 609 kB/s | 548 kB 00:00 2024-03-24T06:25:02.286 INFO:teuthology.orchestra.run.smithi155.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 519 kB/s | 309 kB 00:00 2024-03-24T06:25:02.312 INFO:teuthology.orchestra.run.smithi155.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-24T06:25:02.337 INFO:teuthology.orchestra.run.smithi143.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-03-24T06:25:02.395 INFO:teuthology.orchestra.run.smithi143.stdout:(110/174): gperftools-libs-2.9.1-2.el9.x86_64.r 296 kB/s | 309 kB 00:01 2024-03-24T06:25:02.412 INFO:teuthology.orchestra.run.smithi143.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-24T06:25:02.420 INFO:teuthology.orchestra.run.smithi155.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-24T06:25:02.479 INFO:teuthology.orchestra.run.smithi143.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-24T06:25:02.537 INFO:teuthology.orchestra.run.smithi155.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-24T06:25:02.548 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:25:02.549 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T06:25:02.571 INFO:teuthology.orchestra.run.smithi022.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T06:25:02.572 INFO:teuthology.orchestra.run.smithi022.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T06:25:02.574 INFO:teuthology.orchestra.run.smithi022.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T06:25:02.575 INFO:teuthology.orchestra.run.smithi022.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T06:25:02.577 INFO:teuthology.orchestra.run.smithi022.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T06:25:02.579 INFO:teuthology.orchestra.run.smithi155.stdout:(111/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 70 kB/s | 94 kB 00:01 2024-03-24T06:25:02.581 INFO:teuthology.orchestra.run.smithi022.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T06:25:02.583 INFO:teuthology.orchestra.run.smithi022.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T06:25:02.638 INFO:teuthology.orchestra.run.smithi155.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-24T06:25:02.675 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T06:25:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T06:25:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T06:25:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T06:25:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T06:25:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T06:25:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T06:25:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T06:25:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T06:25:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T06:25:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T06:25:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T06:25:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T06:25:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout:Installing weak dependencies: 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-24T06:25:02.711 INFO:teuthology.orchestra.run.smithi022.stdout:Install 150 Packages 2024-03-24T06:25:02.712 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 24 Packages 2024-03-24T06:25:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:25:02.712 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 224 M 2024-03-24T06:25:02.712 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T06:25:02.771 INFO:teuthology.orchestra.run.smithi155.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-24T06:25:02.847 INFO:teuthology.orchestra.run.smithi155.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-24T06:25:02.939 INFO:teuthology.orchestra.run.smithi155.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-03-24T06:25:02.980 INFO:teuthology.orchestra.run.smithi155.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 326 kB/s | 43 kB 00:00 2024-03-24T06:25:03.056 INFO:teuthology.orchestra.run.smithi155.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-24T06:25:03.098 INFO:teuthology.orchestra.run.smithi155.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-24T06:25:03.148 INFO:teuthology.orchestra.run.smithi155.stdout:(119/174): luarocks-3.9.2-1.el9.noarch.rpm 266 kB/s | 151 kB 00:00 2024-03-24T06:25:03.173 INFO:teuthology.orchestra.run.smithi155.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-24T06:25:03.240 INFO:teuthology.orchestra.run.smithi155.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-24T06:25:03.290 INFO:teuthology.orchestra.run.smithi155.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-24T06:25:03.366 INFO:teuthology.orchestra.run.smithi155.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-24T06:25:03.391 INFO:teuthology.orchestra.run.smithi155.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.2 MB/s | 303 kB 00:00 2024-03-24T06:25:03.408 INFO:teuthology.orchestra.run.smithi022.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-24T06:25:03.458 INFO:teuthology.orchestra.run.smithi155.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-24T06:25:03.483 INFO:teuthology.orchestra.run.smithi155.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-24T06:25:03.509 INFO:teuthology.orchestra.run.smithi155.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-24T06:25:03.576 INFO:teuthology.orchestra.run.smithi155.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-24T06:25:03.601 INFO:teuthology.orchestra.run.smithi155.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-24T06:25:03.626 INFO:teuthology.orchestra.run.smithi155.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-24T06:25:03.693 INFO:teuthology.orchestra.run.smithi155.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-24T06:25:03.742 INFO:teuthology.orchestra.run.smithi022.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.6 MB/s | 888 kB 00:00 2024-03-24T06:25:03.752 INFO:teuthology.orchestra.run.smithi155.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-03-24T06:25:03.785 INFO:teuthology.orchestra.run.smithi155.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-03-24T06:25:03.810 INFO:teuthology.orchestra.run.smithi155.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-03-24T06:25:03.833 INFO:teuthology.orchestra.run.smithi022.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.6 MB/s | 145 kB 00:00 2024-03-24T06:25:03.869 INFO:teuthology.orchestra.run.smithi155.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2024-03-24T06:25:03.911 INFO:teuthology.orchestra.run.smithi155.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-24T06:25:03.928 INFO:teuthology.orchestra.run.smithi155.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-03-24T06:25:03.978 INFO:teuthology.orchestra.run.smithi155.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-03-24T06:25:04.020 INFO:teuthology.orchestra.run.smithi155.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-24T06:25:04.037 INFO:teuthology.orchestra.run.smithi155.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-03-24T06:25:04.122 INFO:teuthology.orchestra.run.smithi155.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-03-24T06:25:04.146 INFO:teuthology.orchestra.run.smithi155.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-24T06:25:04.163 INFO:teuthology.orchestra.run.smithi155.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-03-24T06:25:04.246 INFO:teuthology.orchestra.run.smithi155.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-03-24T06:25:04.280 INFO:teuthology.orchestra.run.smithi155.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-24T06:25:04.283 INFO:teuthology.orchestra.run.smithi022.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 5.2 MB/s | 2.4 MB 00:00 2024-03-24T06:25:04.305 INFO:teuthology.orchestra.run.smithi155.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 630 kB/s | 90 kB 00:00 2024-03-24T06:25:04.384 INFO:teuthology.orchestra.run.smithi022.stdout:(5/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 4.2 MB/s | 5.3 MB 00:01 2024-03-24T06:25:04.389 INFO:teuthology.orchestra.run.smithi155.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-24T06:25:04.414 INFO:teuthology.orchestra.run.smithi155.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-03-24T06:25:04.429 INFO:teuthology.orchestra.run.smithi143.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 33 kB/s | 67 kB 00:02 2024-03-24T06:25:04.439 INFO:teuthology.orchestra.run.smithi155.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-24T06:25:04.546 INFO:teuthology.orchestra.run.smithi143.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 71 kB/s | 151 kB 00:02 2024-03-24T06:25:04.551 INFO:teuthology.orchestra.run.smithi022.stdout:(6/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 5.6 MB/s | 1.5 MB 00:00 2024-03-24T06:25:04.581 INFO:teuthology.orchestra.run.smithi155.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-03-24T06:25:04.698 INFO:teuthology.orchestra.run.smithi155.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-24T06:25:04.763 INFO:teuthology.orchestra.run.smithi143.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 367 kB/s | 838 kB 00:02 2024-03-24T06:25:04.789 INFO:teuthology.orchestra.run.smithi143.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 180 kB/s | 43 kB 00:00 2024-03-24T06:25:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:(7/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 11 MB/s | 21 MB 00:01 2024-03-24T06:25:05.118 INFO:teuthology.orchestra.run.smithi022.stdout:(8/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 6.4 MB/s | 4.7 MB 00:00 2024-03-24T06:25:05.193 INFO:teuthology.orchestra.run.smithi022.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 342 kB/s | 25 kB 00:00 2024-03-24T06:25:05.232 INFO:teuthology.orchestra.run.smithi155.stdout:(152/174): librados2-19.0.0-1855.g21b3bd67.el9. 4.1 MB/s | 3.4 MB 00:00 2024-03-24T06:25:05.272 INFO:teuthology.orchestra.run.smithi143.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 63 kB/s | 32 kB 00:00 2024-03-24T06:25:05.297 INFO:teuthology.orchestra.run.smithi143.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 29 kB/s | 15 kB 00:00 2024-03-24T06:25:05.557 INFO:teuthology.orchestra.run.smithi155.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.7 MB/s | 2.3 MB 00:00 2024-03-24T06:25:05.657 INFO:teuthology.orchestra.run.smithi155.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 420 kB/s | 42 kB 00:00 2024-03-24T06:25:05.766 INFO:teuthology.orchestra.run.smithi155.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-24T06:25:05.866 INFO:teuthology.orchestra.run.smithi155.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 713 kB/s | 71 kB 00:00 2024-03-24T06:25:05.966 INFO:teuthology.orchestra.run.smithi155.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 365 kB/s | 36 kB 00:00 2024-03-24T06:25:06.075 INFO:teuthology.orchestra.run.smithi155.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-24T06:25:06.108 INFO:teuthology.orchestra.run.smithi155.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-03-24T06:25:06.152 INFO:teuthology.orchestra.run.smithi022.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 8.0 MB/s | 9.8 MB 00:01 2024-03-24T06:25:06.184 INFO:teuthology.orchestra.run.smithi155.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-24T06:25:06.243 INFO:teuthology.orchestra.run.smithi155.stdout:(161/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-03-24T06:25:06.244 INFO:teuthology.orchestra.run.smithi022.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.4 MB/s | 219 kB 00:00 2024-03-24T06:25:06.268 INFO:teuthology.orchestra.run.smithi155.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 145 kB/s | 23 kB 00:00 2024-03-24T06:25:06.293 INFO:teuthology.orchestra.run.smithi155.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 994 kB/s | 108 kB 00:00 2024-03-24T06:25:06.311 INFO:teuthology.orchestra.run.smithi022.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 410 kB/s | 27 kB 00:00 2024-03-24T06:25:06.377 INFO:teuthology.orchestra.run.smithi155.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-24T06:25:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 484 kB/s | 32 kB 00:00 2024-03-24T06:25:06.402 INFO:teuthology.orchestra.run.smithi155.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 589 kB/s | 64 kB 00:00 2024-03-24T06:25:06.477 INFO:teuthology.orchestra.run.smithi155.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 287 kB/s | 29 kB 00:00 2024-03-24T06:25:06.536 INFO:teuthology.orchestra.run.smithi155.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-03-24T06:25:06.639 INFO:teuthology.orchestra.run.smithi143.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 126 kB/s | 172 kB 00:01 2024-03-24T06:25:06.703 INFO:teuthology.orchestra.run.smithi155.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.6 MB/s | 2.2 MB 00:00 2024-03-24T06:25:06.736 INFO:teuthology.orchestra.run.smithi155.stdout:(169/174): libfdisk-2.37.4-18.el9.x86_64.rpm 314 kB/s | 155 kB 00:00 2024-03-24T06:25:06.756 INFO:teuthology.orchestra.run.smithi143.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 246 kB/s | 358 kB 00:01 2024-03-24T06:25:06.798 INFO:teuthology.orchestra.run.smithi143.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-03-24T06:25:06.820 INFO:teuthology.orchestra.run.smithi155.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.1 MB/s | 465 kB 00:00 2024-03-24T06:25:06.831 INFO:teuthology.orchestra.run.smithi143.stdout:(122/174): python3-asyncssh-2.13.2-5.el9.noarch 228 kB/s | 548 kB 00:02 2024-03-24T06:25:06.845 INFO:teuthology.orchestra.run.smithi022.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 1.6 MB/s | 744 kB 00:00 2024-03-24T06:25:06.857 INFO:teuthology.orchestra.run.smithi143.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-24T06:25:06.870 INFO:teuthology.orchestra.run.smithi155.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 234 kB/s | 31 kB 00:00 2024-03-24T06:25:06.937 INFO:teuthology.orchestra.run.smithi155.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-24T06:25:07.007 INFO:teuthology.orchestra.run.smithi143.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 964 kB/s | 144 kB 00:00 2024-03-24T06:25:07.033 INFO:teuthology.orchestra.run.smithi143.stdout:(125/174): python3-google-auth-2.28.2-1.el9.noa 866 kB/s | 203 kB 00:00 2024-03-24T06:25:07.070 INFO:teuthology.orchestra.run.smithi022.stdout:(15/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 6.7 MB/s | 17 MB 00:02 2024-03-24T06:25:07.074 INFO:teuthology.orchestra.run.smithi143.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-24T06:25:07.103 INFO:teuthology.orchestra.run.smithi022.stdout:(16/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 634 kB/s | 163 kB 00:00 2024-03-24T06:25:07.124 INFO:teuthology.orchestra.run.smithi143.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-24T06:25:07.129 INFO:teuthology.orchestra.run.smithi022.stdout:(17/174): librados-devel-19.0.0-1855.g21b3bd67. 2.2 MB/s | 127 kB 00:00 2024-03-24T06:25:07.149 INFO:teuthology.orchestra.run.smithi143.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-03-24T06:25:07.175 INFO:teuthology.orchestra.run.smithi143.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-24T06:25:07.179 INFO:teuthology.orchestra.run.smithi022.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 6.6 MB/s | 499 kB 00:00 2024-03-24T06:25:07.179 INFO:teuthology.orchestra.run.smithi155.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 241 kB/s | 74 kB 00:00 2024-03-24T06:25:07.200 INFO:teuthology.orchestra.run.smithi143.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-24T06:25:07.242 INFO:teuthology.orchestra.run.smithi143.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-24T06:25:07.246 INFO:teuthology.orchestra.run.smithi022.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 682 kB/s | 45 kB 00:00 2024-03-24T06:25:07.329 INFO:teuthology.orchestra.run.smithi022.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.6 MB/s | 131 kB 00:00 2024-03-24T06:25:07.413 INFO:teuthology.orchestra.run.smithi022.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.9 MB/s | 161 kB 00:00 2024-03-24T06:25:07.497 INFO:teuthology.orchestra.run.smithi022.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 3.8 MB/s | 320 kB 00:00 2024-03-24T06:25:07.579 INFO:teuthology.orchestra.run.smithi155.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.2 MB/s | 2.3 MB 00:01 2024-03-24T06:25:07.582 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T06:25:07.583 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 224 MB 00:19 2024-03-24T06:25:07.588 INFO:teuthology.orchestra.run.smithi022.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.2 MB/s | 298 kB 00:00 2024-03-24T06:25:07.680 INFO:teuthology.orchestra.run.smithi022.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.1 MB/s | 99 kB 00:00 2024-03-24T06:25:07.764 INFO:teuthology.orchestra.run.smithi022.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-24T06:25:07.864 INFO:teuthology.orchestra.run.smithi022.stdout:(26/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 7.0 MB/s | 5.1 MB 00:00 2024-03-24T06:25:07.965 INFO:teuthology.orchestra.run.smithi022.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-24T06:25:08.057 INFO:teuthology.orchestra.run.smithi022.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 347 kB/s | 31 kB 00:00 2024-03-24T06:25:08.066 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T06:25:08.148 INFO:teuthology.orchestra.run.smithi022.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-24T06:25:08.273 INFO:teuthology.orchestra.run.smithi022.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 6.1 MB/s | 3.1 MB 00:00 2024-03-24T06:25:08.341 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T06:25:08.341 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T06:25:08.725 INFO:teuthology.orchestra.run.smithi143.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 53 kB/s | 79 kB 00:01 2024-03-24T06:25:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:(31/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 13 MB/s | 47 MB 00:03 2024-03-24T06:25:08.767 INFO:teuthology.orchestra.run.smithi143.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 657 kB/s | 1.0 MB 00:01 2024-03-24T06:25:08.841 INFO:teuthology.orchestra.run.smithi022.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 3.0 MB/s | 248 kB 00:00 2024-03-24T06:25:08.891 INFO:teuthology.orchestra.run.smithi022.stdout:(33/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 2.4 MB/s | 1.8 MB 00:00 2024-03-24T06:25:08.916 INFO:teuthology.orchestra.run.smithi022.stdout:(34/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 664 kB/s | 49 kB 00:00 2024-03-24T06:25:08.958 INFO:teuthology.orchestra.run.smithi022.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1855.g2 233 kB/s | 15 kB 00:00 2024-03-24T06:25:08.992 INFO:teuthology.orchestra.run.smithi022.stdout:(36/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 3.5 MB/s | 264 kB 00:00 2024-03-24T06:25:09.059 INFO:teuthology.orchestra.run.smithi022.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 277 kB/s | 18 kB 00:00 2024-03-24T06:25:09.176 INFO:teuthology.orchestra.run.smithi022.stdout:(38/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-24T06:25:09.309 INFO:teuthology.orchestra.run.smithi022.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 7.2 MB/s | 7.4 MB 00:01 2024-03-24T06:25:09.409 INFO:teuthology.orchestra.run.smithi022.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 662 kB 00:00 2024-03-24T06:25:09.452 INFO:teuthology.orchestra.run.smithi022.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-03-24T06:25:09.477 INFO:teuthology.orchestra.run.smithi022.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 648 kB/s | 106 kB 00:00 2024-03-24T06:25:09.510 INFO:teuthology.orchestra.run.smithi022.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.2 MB/s | 307 kB 00:00 2024-03-24T06:25:09.536 INFO:teuthology.orchestra.run.smithi022.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 571 kB/s | 32 kB 00:00 2024-03-24T06:25:09.677 INFO:teuthology.orchestra.run.smithi022.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-24T06:25:09.694 INFO:teuthology.orchestra.run.smithi022.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 97 kB/s | 17 kB 00:00 2024-03-24T06:25:09.711 INFO:teuthology.orchestra.run.smithi022.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 59 kB/s | 10 kB 00:00 2024-03-24T06:25:09.745 INFO:teuthology.orchestra.run.smithi022.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-24T06:25:09.804 INFO:teuthology.orchestra.run.smithi022.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 44 kB 00:00 2024-03-24T06:25:09.825 INFO:teuthology.orchestra.run.smithi143.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 52 kB/s | 58 kB 00:01 2024-03-24T06:25:09.837 INFO:teuthology.orchestra.run.smithi022.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 5.3 MB/s | 661 kB 00:00 2024-03-24T06:25:09.879 INFO:teuthology.orchestra.run.smithi022.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 578 kB 00:00 2024-03-24T06:25:09.896 INFO:teuthology.orchestra.run.smithi022.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 887 kB/s | 46 kB 00:00 2024-03-24T06:25:09.921 INFO:teuthology.orchestra.run.smithi022.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.6 MB/s | 282 kB 00:00 2024-03-24T06:25:09.980 INFO:teuthology.orchestra.run.smithi022.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 187 kB 00:00 2024-03-24T06:25:10.005 INFO:teuthology.orchestra.run.smithi022.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 424 kB/s | 34 kB 00:00 2024-03-24T06:25:10.025 INFO:teuthology.orchestra.run.smithi143.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 82 kB/s | 16 kB 00:00 2024-03-24T06:25:10.047 INFO:teuthology.orchestra.run.smithi022.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.7 MB/s | 217 kB 00:00 2024-03-24T06:25:10.051 INFO:teuthology.orchestra.run.smithi143.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 212 kB/s | 272 kB 00:01 2024-03-24T06:25:10.076 INFO:teuthology.orchestra.run.smithi143.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-24T06:25:10.143 INFO:teuthology.orchestra.run.smithi143.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-03-24T06:25:10.143 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T06:25:10.144 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T06:25:10.160 INFO:teuthology.orchestra.run.smithi143.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-03-24T06:25:10.205 INFO:teuthology.orchestra.run.smithi022.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-24T06:25:10.281 INFO:teuthology.orchestra.run.smithi022.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.8 MB/s | 1.5 MB 00:00 2024-03-24T06:25:10.322 INFO:teuthology.orchestra.run.smithi022.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.0 MB/s | 811 kB 00:00 2024-03-24T06:25:10.348 INFO:teuthology.orchestra.run.smithi022.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.0 MB/s | 134 kB 00:00 2024-03-24T06:25:10.448 INFO:teuthology.orchestra.run.smithi022.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.2 MB/s | 1.0 MB 00:00 2024-03-24T06:25:10.590 INFO:teuthology.orchestra.run.smithi022.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-24T06:25:10.615 INFO:teuthology.orchestra.run.smithi022.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 301 kB/s | 78 kB 00:00 2024-03-24T06:25:10.757 INFO:teuthology.orchestra.run.smithi022.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-24T06:25:10.782 INFO:teuthology.orchestra.run.smithi022.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 24 kB 00:00 2024-03-24T06:25:10.807 INFO:teuthology.orchestra.run.smithi022.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 103 kB/s | 18 kB 00:00 2024-03-24T06:25:10.841 INFO:teuthology.orchestra.run.smithi022.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.0 MB/s | 32 kB 00:00 2024-03-24T06:25:10.883 INFO:teuthology.orchestra.run.smithi022.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T06:25:11.008 INFO:teuthology.orchestra.run.smithi022.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-24T06:25:11.033 INFO:teuthology.orchestra.run.smithi022.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-24T06:25:11.058 INFO:teuthology.orchestra.run.smithi022.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-24T06:25:11.121 INFO:teuthology.orchestra.run.smithi022.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-24T06:25:11.142 INFO:teuthology.orchestra.run.smithi022.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 503 kB/s | 41 kB 00:00 2024-03-24T06:25:11.176 INFO:teuthology.orchestra.run.smithi022.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 238 kB 00:00 2024-03-24T06:25:11.209 INFO:teuthology.orchestra.run.smithi022.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-24T06:25:11.234 INFO:teuthology.orchestra.run.smithi022.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-03-24T06:25:11.251 INFO:teuthology.orchestra.run.smithi022.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 664 kB/s | 47 kB 00:00 2024-03-24T06:25:11.302 INFO:teuthology.orchestra.run.smithi022.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 647 kB/s | 43 kB 00:00 2024-03-24T06:25:11.310 INFO:teuthology.orchestra.run.smithi143.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 162 kB/s | 188 kB 00:01 2024-03-24T06:25:11.410 INFO:teuthology.orchestra.run.smithi022.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 275 kB/s | 39 kB 00:00 2024-03-24T06:25:11.419 INFO:teuthology.orchestra.run.smithi143.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 47 kB/s | 59 kB 00:01 2024-03-24T06:25:11.435 INFO:teuthology.orchestra.run.smithi143.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-03-24T06:25:11.494 INFO:teuthology.orchestra.run.smithi143.stdout:(143/174): python3-grpcio-1.46.7-10.el9.x86_64. 447 kB/s | 2.0 MB 00:04 2024-03-24T06:25:11.619 INFO:teuthology.orchestra.run.smithi022.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-24T06:25:11.644 INFO:teuthology.orchestra.run.smithi022.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 125 kB/s | 41 kB 00:00 2024-03-24T06:25:11.708 INFO:teuthology.orchestra.run.smithi022.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 300 kB 00:00 2024-03-24T06:25:11.736 INFO:teuthology.orchestra.run.smithi022.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 715 kB/s | 63 kB 00:00 2024-03-24T06:25:11.761 INFO:teuthology.orchestra.run.smithi022.stdout:(84/174): uuid-1.6.2-55.el9.x86_64.rpm 498 kB/s | 56 kB 00:00 2024-03-24T06:25:11.787 INFO:teuthology.orchestra.run.smithi022.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 572 kB/s | 44 kB 00:00 2024-03-24T06:25:11.803 INFO:teuthology.orchestra.run.smithi022.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-24T06:25:11.846 INFO:teuthology.orchestra.run.smithi022.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2024-03-24T06:25:11.887 INFO:teuthology.orchestra.run.smithi022.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T06:25:11.913 INFO:teuthology.orchestra.run.smithi022.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-24T06:25:11.946 INFO:teuthology.orchestra.run.smithi022.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 124 kB 00:00 2024-03-24T06:25:11.971 INFO:teuthology.orchestra.run.smithi022.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 187 kB 00:00 2024-03-24T06:25:12.921 INFO:teuthology.orchestra.run.smithi022.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 206 kB/s | 195 kB 00:00 2024-03-24T06:25:12.969 INFO:teuthology.orchestra.run.smithi143.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 33 kB/s | 51 kB 00:01 2024-03-24T06:25:12.980 INFO:teuthology.orchestra.run.smithi022.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 176 kB/s | 187 kB 00:01 2024-03-24T06:25:13.011 INFO:teuthology.orchestra.run.smithi143.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 146 kB/s | 230 kB 00:01 2024-03-24T06:25:13.037 INFO:teuthology.orchestra.run.smithi143.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-03-24T06:25:13.080 INFO:teuthology.orchestra.run.smithi022.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 218 kB/s | 21 kB 00:00 2024-03-24T06:25:13.095 INFO:teuthology.orchestra.run.smithi143.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-24T06:25:13.139 INFO:teuthology.orchestra.run.smithi022.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 444 kB/s | 25 kB 00:00 2024-03-24T06:25:13.387 INFO:teuthology.orchestra.run.smithi143.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 547 kB/s | 192 kB 00:00 2024-03-24T06:25:13.456 INFO:teuthology.orchestra.run.smithi022.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-03-24T06:25:13.514 INFO:teuthology.orchestra.run.smithi022.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.2 MB/s | 245 kB 00:00 2024-03-24T06:25:13.565 INFO:teuthology.orchestra.run.smithi022.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-03-24T06:25:13.615 INFO:teuthology.orchestra.run.smithi022.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.3 MB/s | 216 kB 00:00 2024-03-24T06:25:13.665 INFO:teuthology.orchestra.run.smithi022.stdout:(100/174): libgfortran-11.4.1-3.el9.x86_64.rpm 474 kB/s | 803 kB 00:01 2024-03-24T06:25:13.690 INFO:teuthology.orchestra.run.smithi022.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 466 kB/s | 35 kB 00:00 2024-03-24T06:25:13.741 INFO:teuthology.orchestra.run.smithi022.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-03-24T06:25:13.774 INFO:teuthology.orchestra.run.smithi022.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 415 kB/s | 44 kB 00:00 2024-03-24T06:25:13.825 INFO:teuthology.orchestra.run.smithi022.stdout:(104/174): python3-requests-2.25.1-8.el9.noarch 138 kB/s | 125 kB 00:00 2024-03-24T06:25:13.850 INFO:teuthology.orchestra.run.smithi022.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-03-24T06:25:13.908 INFO:teuthology.orchestra.run.smithi022.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-24T06:25:13.921 INFO:teuthology.orchestra.run.smithi143.stdout:(149/174): librados2-19.0.0-1855.g21b3bd67.el9. 6.3 MB/s | 3.4 MB 00:00 2024-03-24T06:25:13.967 INFO:teuthology.orchestra.run.smithi022.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 340 kB/s | 19 kB 00:00 2024-03-24T06:25:14.025 INFO:teuthology.orchestra.run.smithi022.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-03-24T06:25:14.059 INFO:teuthology.orchestra.run.smithi022.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-24T06:25:14.085 INFO:teuthology.orchestra.run.smithi022.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 448 kB/s | 26 kB 00:00 2024-03-24T06:25:14.110 INFO:teuthology.orchestra.run.smithi022.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 983 kB/s | 49 kB 00:00 2024-03-24T06:25:14.135 INFO:teuthology.orchestra.run.smithi022.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-24T06:25:14.160 INFO:teuthology.orchestra.run.smithi022.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-24T06:25:14.171 INFO:teuthology.orchestra.run.smithi143.stdout:(150/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 13 MB/s | 3.1 MB 00:00 2024-03-24T06:25:14.213 INFO:teuthology.orchestra.run.smithi143.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T06:25:14.219 INFO:teuthology.orchestra.run.smithi022.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 9.3 MB/s | 548 kB 00:00 2024-03-24T06:25:14.261 INFO:teuthology.orchestra.run.smithi022.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-03-24T06:25:14.286 INFO:teuthology.orchestra.run.smithi022.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 658 kB/s | 43 kB 00:00 2024-03-24T06:25:14.312 INFO:teuthology.orchestra.run.smithi022.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 651 kB/s | 32 kB 00:00 2024-03-24T06:25:14.328 INFO:teuthology.orchestra.run.smithi022.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 365 kB/s | 15 kB 00:00 2024-03-24T06:25:14.362 INFO:teuthology.orchestra.run.smithi022.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-24T06:25:14.396 INFO:teuthology.orchestra.run.smithi022.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-03-24T06:25:14.429 INFO:teuthology.orchestra.run.smithi022.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-24T06:25:14.455 INFO:teuthology.orchestra.run.smithi022.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 612 kB/s | 35 kB 00:00 2024-03-24T06:25:14.488 INFO:teuthology.orchestra.run.smithi022.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-24T06:25:14.531 INFO:teuthology.orchestra.run.smithi022.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.5 MB/s | 144 kB 00:00 2024-03-24T06:25:14.589 INFO:teuthology.orchestra.run.smithi022.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 186 kB/s | 11 kB 00:00 2024-03-24T06:25:14.622 INFO:teuthology.orchestra.run.smithi022.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 545 kB/s | 18 kB 00:00 2024-03-24T06:25:14.714 INFO:teuthology.orchestra.run.smithi022.stdout:(127/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-24T06:25:14.738 INFO:teuthology.orchestra.run.smithi143.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.4 MB/s | 2.3 MB 00:00 2024-03-24T06:25:14.740 INFO:teuthology.orchestra.run.smithi022.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 199 kB/s | 23 kB 00:00 2024-03-24T06:25:14.765 INFO:teuthology.orchestra.run.smithi022.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 394 kB/s | 19 kB 00:00 2024-03-24T06:25:14.790 INFO:teuthology.orchestra.run.smithi022.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 399 kB/s | 20 kB 00:00 2024-03-24T06:25:14.857 INFO:teuthology.orchestra.run.smithi022.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.1 MB/s | 2.0 MB 00:00 2024-03-24T06:25:14.882 INFO:teuthology.orchestra.run.smithi022.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 508 kB/s | 46 kB 00:00 2024-03-24T06:25:14.899 INFO:teuthology.orchestra.run.smithi022.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-03-24T06:25:14.933 INFO:teuthology.orchestra.run.smithi022.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-24T06:25:14.966 INFO:teuthology.orchestra.run.smithi022.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-24T06:25:14.992 INFO:teuthology.orchestra.run.smithi022.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 285 kB/s | 16 kB 00:00 2024-03-24T06:25:15.017 INFO:teuthology.orchestra.run.smithi022.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-24T06:25:15.042 INFO:teuthology.orchestra.run.smithi022.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 573 kB/s | 28 kB 00:00 2024-03-24T06:25:15.067 INFO:teuthology.orchestra.run.smithi022.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 623 kB/s | 31 kB 00:00 2024-03-24T06:25:15.101 INFO:teuthology.orchestra.run.smithi022.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-24T06:25:15.121 INFO:teuthology.orchestra.run.smithi143.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-03-24T06:25:15.152 INFO:teuthology.orchestra.run.smithi022.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 713 kB/s | 59 kB 00:00 2024-03-24T06:25:15.172 INFO:teuthology.orchestra.run.smithi143.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 845 kB/s | 42 kB 00:00 2024-03-24T06:25:15.177 INFO:teuthology.orchestra.run.smithi022.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-03-24T06:25:15.219 INFO:teuthology.orchestra.run.smithi022.stdout:(143/174): python3-kubernetes-26.1.0-2.el9.noar 2.3 MB/s | 1.0 MB 00:00 2024-03-24T06:25:15.253 INFO:teuthology.orchestra.run.smithi022.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-24T06:25:15.255 INFO:teuthology.orchestra.run.smithi143.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-24T06:25:15.278 INFO:teuthology.orchestra.run.smithi022.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-24T06:25:15.280 INFO:teuthology.orchestra.run.smithi143.stdout:(156/174): python3-websocket-client-1.2.3-2.el9 24 kB/s | 90 kB 00:03 2024-03-24T06:25:15.306 INFO:teuthology.orchestra.run.smithi143.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-24T06:25:15.312 INFO:teuthology.orchestra.run.smithi022.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-24T06:25:15.337 INFO:teuthology.orchestra.run.smithi022.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-03-24T06:25:15.348 INFO:teuthology.orchestra.run.smithi143.stdout:(158/174): thrift-0.15.0-2.el9.x86_64.rpm 719 kB/s | 1.6 MB 00:02 2024-03-24T06:25:15.371 INFO:teuthology.orchestra.run.smithi022.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-24T06:25:15.373 INFO:teuthology.orchestra.run.smithi143.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 396 kB/s | 36 kB 00:00 2024-03-24T06:25:15.398 INFO:teuthology.orchestra.run.smithi143.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-24T06:25:15.405 INFO:teuthology.orchestra.run.smithi022.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-24T06:25:15.424 INFO:teuthology.orchestra.run.smithi143.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-24T06:25:15.449 INFO:teuthology.orchestra.run.smithi143.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-24T06:25:15.472 INFO:teuthology.orchestra.run.smithi022.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-24T06:25:15.474 INFO:teuthology.orchestra.run.smithi143.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-24T06:25:15.507 INFO:teuthology.orchestra.run.smithi143.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-24T06:25:15.514 INFO:teuthology.orchestra.run.smithi022.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-24T06:25:15.533 INFO:teuthology.orchestra.run.smithi143.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-24T06:25:15.558 INFO:teuthology.orchestra.run.smithi143.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 763 kB/s | 64 kB 00:00 2024-03-24T06:25:15.583 INFO:teuthology.orchestra.run.smithi143.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-24T06:25:15.864 INFO:teuthology.orchestra.run.smithi022.stdout:(152/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 6.8 MB/s | 3.1 MB 00:00 2024-03-24T06:25:15.867 INFO:teuthology.orchestra.run.smithi143.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.0 MB/s | 2.2 MB 00:00 2024-03-24T06:25:15.917 INFO:teuthology.orchestra.run.smithi143.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 3.2 MB/s | 1.2 MB 00:00 2024-03-24T06:25:15.931 INFO:teuthology.orchestra.run.smithi022.stdout:(153/174): librados2-19.0.0-1855.g21b3bd67.el9. 6.1 MB/s | 3.4 MB 00:00 2024-03-24T06:25:15.976 INFO:teuthology.orchestra.run.smithi143.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-24T06:25:16.001 INFO:teuthology.orchestra.run.smithi143.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 373 kB/s | 31 kB 00:00 2024-03-24T06:25:16.043 INFO:teuthology.orchestra.run.smithi143.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.0 MB/s | 2.3 MB 00:00 2024-03-24T06:25:16.135 INFO:teuthology.orchestra.run.smithi143.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-24T06:25:16.335 INFO:teuthology.orchestra.run.smithi143.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.0 MB/s | 1.1 MB 00:00 2024-03-24T06:25:16.339 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-24T06:25:16.339 INFO:teuthology.orchestra.run.smithi143.stdout:Total 8.0 MB/s | 224 MB 00:27 2024-03-24T06:25:16.590 INFO:teuthology.orchestra.run.smithi022.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 64 kB/s | 42 kB 00:00 2024-03-24T06:25:16.794 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-24T06:25:17.063 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T06:25:17.067 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-24T06:25:17.067 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-24T06:25:17.166 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T06:25:17.200 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T06:25:17.239 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:17.249 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:17.297 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T06:25:17.350 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T06:25:17.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T06:25:17.419 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T06:25:17.460 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T06:25:17.531 INFO:teuthology.orchestra.run.smithi022.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 264 kB/s | 248 kB 00:00 2024-03-24T06:25:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T06:25:17.844 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:17.947 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:17.970 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:18.060 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:18.091 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:25:18.132 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T06:25:18.265 INFO:teuthology.orchestra.run.smithi022.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 97 kB/s | 71 kB 00:00 2024-03-24T06:25:18.308 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T06:25:18.357 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T06:25:18.386 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:18.471 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:18.495 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:18.542 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:18.574 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T06:25:18.597 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:18.612 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:18.756 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:18.786 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-24T06:25:18.787 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:18.787 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-24T06:25:18.827 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T06:25:18.857 INFO:teuthology.orchestra.run.smithi022.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 627 kB/s | 1.8 MB 00:02 2024-03-24T06:25:18.935 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T06:25:18.982 INFO:teuthology.orchestra.run.smithi022.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 51 kB/s | 36 kB 00:00 2024-03-24T06:25:19.046 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:19.089 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:19.155 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:19.249 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:19.424 INFO:teuthology.orchestra.run.smithi022.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 604 kB/s | 2.3 MB 00:03 2024-03-24T06:25:19.461 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:19.572 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T06:25:19.691 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T06:25:19.721 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T06:25:19.748 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T06:25:19.783 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T06:25:19.857 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T06:25:19.912 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T06:25:19.958 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T06:25:20.015 INFO:teuthology.orchestra.run.smithi022.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 166 kB/s | 192 kB 00:01 2024-03-24T06:25:20.022 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T06:25:20.057 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T06:25:20.100 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T06:25:20.133 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T06:25:20.149 INFO:teuthology.orchestra.run.smithi022.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 35 kB/s | 23 kB 00:00 2024-03-24T06:25:20.161 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T06:25:20.216 INFO:teuthology.orchestra.run.smithi022.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 207 kB/s | 256 kB 00:01 2024-03-24T06:25:20.239 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T06:25:20.275 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T06:25:20.318 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T06:25:20.347 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T06:25:20.417 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T06:25:20.461 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T06:25:20.544 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T06:25:20.676 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T06:25:20.731 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T06:25:20.933 INFO:teuthology.orchestra.run.smithi022.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 118 kB/s | 108 kB 00:00 2024-03-24T06:25:21.033 INFO:teuthology.orchestra.run.smithi022.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 175 kB/s | 155 kB 00:00 2024-03-24T06:25:21.075 INFO:teuthology.orchestra.run.smithi022.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 158 kB/s | 136 kB 00:00 2024-03-24T06:25:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T06:25:21.335 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T06:25:21.406 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T06:25:21.446 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:21.455 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:21.486 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T06:25:21.527 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T06:25:21.565 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T06:25:21.594 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T06:25:21.667 INFO:teuthology.orchestra.run.smithi022.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 45 kB/s | 29 kB 00:00 2024-03-24T06:25:21.735 INFO:teuthology.orchestra.run.smithi022.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 80 kB/s | 64 kB 00:00 2024-03-24T06:25:21.839 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T06:25:22.003 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T06:25:22.056 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T06:25:22.100 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T06:25:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T06:25:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T06:25:22.350 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T06:25:22.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T06:25:22.454 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T06:25:22.487 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T06:25:22.534 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T06:25:22.570 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T06:25:22.612 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T06:25:22.655 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T06:25:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T06:25:22.827 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T06:25:22.858 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T06:25:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T06:25:22.993 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T06:25:23.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T06:25:23.073 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T06:25:23.099 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T06:25:23.123 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T06:25:23.196 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T06:25:23.325 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T06:25:23.418 INFO:teuthology.orchestra.run.smithi022.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 529 kB/s | 1.2 MB 00:02 2024-03-24T06:25:23.423 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T06:25:23.474 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T06:25:23.522 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T06:25:23.578 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T06:25:23.764 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T06:25:23.811 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T06:25:23.941 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T06:25:23.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T06:25:24.016 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T06:25:24.048 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T06:25:24.076 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T06:25:24.110 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T06:25:24.149 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T06:25:24.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T06:25:24.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T06:25:24.287 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T06:25:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T06:25:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T06:25:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T06:25:24.431 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T06:25:24.520 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T06:25:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T06:25:24.651 INFO:teuthology.orchestra.run.smithi022.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 378 kB/s | 465 kB 00:01 2024-03-24T06:25:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T06:25:24.685 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T06:25:24.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T06:25:24.831 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T06:25:24.879 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T06:25:25.076 INFO:teuthology.orchestra.run.smithi022.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 648 kB/s | 2.2 MB 00:03 2024-03-24T06:25:25.285 INFO:teuthology.orchestra.run.smithi022.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 49 kB/s | 31 kB 00:00 2024-03-24T06:25:25.512 INFO:teuthology.orchestra.run.smithi022.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 619 kB/s | 2.3 MB 00:03 2024-03-24T06:25:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T06:25:25.635 INFO:teuthology.orchestra.run.smithi022.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 213 kB/s | 74 kB 00:00 2024-03-24T06:25:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T06:25:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T06:25:25.759 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T06:25:25.776 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-24T06:25:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T06:25:25.882 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T06:25:25.886 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T06:25:25.908 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T06:25:25.956 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:25.965 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:26.005 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T06:25:26.058 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T06:25:26.063 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T06:25:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T06:25:26.093 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T06:25:26.118 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T06:25:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T06:25:26.177 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T06:25:26.504 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T06:25:26.561 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:26.664 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:26.687 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:26.760 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:26.791 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:25:26.801 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T06:25:26.837 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T06:25:26.916 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T06:25:27.007 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T06:25:27.064 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T06:25:27.093 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:27.195 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:27.219 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:27.269 INFO:teuthology.orchestra.run.smithi022.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 502 kB/s | 1.1 MB 00:02 2024-03-24T06:25:27.274 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:27.279 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T06:25:27.280 INFO:teuthology.orchestra.run.smithi022.stdout:Total 9.1 MB/s | 224 MB 00:24 2024-03-24T06:25:27.322 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T06:25:27.345 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:27.361 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:27.553 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:27.584 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:27.625 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T06:25:27.730 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T06:25:27.819 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:27.863 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:27.931 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:27.989 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:28.151 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T06:25:28.207 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:28.218 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T06:25:28.249 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T06:25:28.281 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T06:25:28.308 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T06:25:28.310 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T06:25:28.405 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T06:25:28.444 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T06:25:28.470 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T06:25:28.531 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T06:25:28.605 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T06:25:28.611 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T06:25:28.654 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:28.659 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T06:25:28.698 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T06:25:28.745 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:28.756 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T06:25:28.761 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T06:25:28.777 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T06:25:28.806 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T06:25:28.813 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T06:25:28.849 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T06:25:28.881 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T06:25:28.910 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T06:25:28.978 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T06:25:29.023 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T06:25:29.066 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T06:25:29.095 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T06:25:29.155 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T06:25:29.155 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T06:25:29.163 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T06:25:29.209 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T06:25:29.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T06:25:29.281 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T06:25:29.299 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:29.371 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:29.402 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T06:25:29.429 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T06:25:29.469 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T06:25:29.985 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T06:25:30.073 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T06:25:30.144 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T06:25:30.194 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:30.202 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:30.234 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T06:25:30.283 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T06:25:30.321 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T06:25:30.351 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T06:25:30.569 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T06:25:30.728 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T06:25:30.804 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T06:25:30.848 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T06:25:30.978 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T06:25:31.031 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T06:25:31.085 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T06:25:31.149 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T06:25:31.181 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T06:25:31.218 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T06:25:31.264 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T06:25:31.301 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T06:25:31.344 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T06:25:31.386 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T06:25:31.519 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T06:25:31.567 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T06:25:31.597 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T06:25:31.619 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:31.638 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T06:25:31.657 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:31.725 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T06:25:31.726 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T06:25:31.732 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T06:25:31.764 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T06:25:31.796 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T06:25:31.822 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T06:25:31.846 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T06:25:31.919 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T06:25:32.046 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T06:25:32.145 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T06:25:32.188 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T06:25:32.194 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:32.235 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:25:32.237 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T06:25:32.291 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T06:25:32.299 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:25:32.299 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T06:25:32.299 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T06:25:32.300 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:32.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:25:32.477 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T06:25:32.526 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T06:25:32.663 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T06:25:32.708 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T06:25:32.740 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T06:25:32.771 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T06:25:32.799 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T06:25:32.833 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T06:25:32.872 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T06:25:32.904 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T06:25:32.934 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T06:25:33.010 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T06:25:33.053 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T06:25:33.086 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T06:25:33.127 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T06:25:33.162 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T06:25:33.250 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T06:25:33.333 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T06:25:33.377 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T06:25:33.416 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T06:25:33.463 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T06:25:33.561 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T06:25:33.609 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T06:25:34.331 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T06:25:34.382 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T06:25:34.416 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T06:25:34.464 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T06:25:34.551 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T06:25:34.592 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T06:25:34.769 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T06:25:34.798 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T06:25:34.875 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T06:25:35.506 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T06:25:35.620 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T06:25:36.818 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T06:25:36.899 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T06:25:36.930 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T06:25:36.970 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T06:25:36.997 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T06:25:37.309 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T06:25:37.336 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:37.408 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:37.440 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T06:25:37.476 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T06:25:37.938 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T06:25:37.964 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:38.035 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:38.057 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T06:25:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T06:25:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T06:25:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T06:25:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T06:25:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T06:25:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T06:25:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T06:25:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T06:25:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T06:25:40.237 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:40.271 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T06:25:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:40.659 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:25:40.693 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:40.806 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:25:40.816 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:25:40.849 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:25:40.867 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:25:40.922 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:25:40.922 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T06:25:40.922 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T06:25:40.922 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:40.949 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T06:25:40.950 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:25:41.259 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T06:25:41.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T06:25:41.339 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:41.491 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:25:41.557 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:41.649 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T06:25:41.702 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T06:25:41.734 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:41.799 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T06:25:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T06:25:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T06:25:42.552 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T06:25:42.699 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T06:25:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:43.403 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T06:25:43.601 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T06:25:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T06:25:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T06:25:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T06:25:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T06:25:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T06:25:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T06:25:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T06:25:44.250 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T06:25:44.252 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:25:44.252 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-24T06:25:44.253 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:44.298 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T06:25:44.362 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T06:25:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T06:25:44.417 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:25:44.433 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:25:44.433 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:44.433 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T06:25:44.433 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T06:25:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T06:25:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T06:25:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T06:25:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T06:25:44.843 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T06:25:44.910 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T06:25:45.024 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T06:25:45.214 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T06:25:45.294 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T06:25:45.922 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:25:45.947 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:25:45.948 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:45.948 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T06:25:45.948 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:25:45.948 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:25:45.948 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:46.027 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T06:25:46.129 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T06:25:46.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:25:46.204 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:46.245 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T06:25:46.299 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:46.325 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T06:25:46.392 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T06:25:46.466 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T06:25:46.521 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T06:25:46.567 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T06:25:46.648 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:25:46.671 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:25:46.671 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:46.671 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T06:25:46.672 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:25:46.672 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:25:46.672 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T06:25:46.992 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T06:25:47.118 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:25:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:25:47.180 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T06:25:47.228 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T06:25:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T06:25:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:25:47.314 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T06:25:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T06:25:47.496 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:25:47.531 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:25:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T06:25:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T06:25:47.792 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T06:25:47.886 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T06:25:47.925 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T06:25:47.965 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T06:25:48.033 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T06:25:48.072 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T06:25:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T06:25:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T06:25:48.435 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T06:25:48.495 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T06:25:48.497 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:25:48.531 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:25:48.534 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T06:25:48.592 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T06:25:48.634 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:25:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T06:25:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T06:25:48.794 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T06:25:48.825 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:25:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:25:48.849 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T06:25:48.893 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:25:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T06:25:48.904 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:25:48.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T06:25:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T06:25:49.268 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T06:25:49.501 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T06:25:49.557 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T06:25:49.630 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T06:25:49.721 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T06:25:49.805 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T06:25:49.837 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:25:49.860 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:49.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:25:49.926 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:25:49.926 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:49.926 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T06:25:49.926 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:25:49.986 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T06:25:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:25:49.994 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:49.994 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T06:25:49.994 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:25:49.994 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:25:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:50.054 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T06:25:50.263 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T06:25:50.297 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:25:50.319 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:25:50.320 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:50.320 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T06:25:50.320 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:25:50.320 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:25:50.320 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:25:50.522 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T06:25:50.572 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T06:25:50.625 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T06:25:50.679 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T06:25:50.724 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T06:25:50.799 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T06:25:50.839 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T06:25:50.884 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T06:25:51.014 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T06:25:51.080 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T06:25:51.115 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T06:25:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T06:25:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T06:25:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T06:25:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T06:25:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T06:25:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T06:25:51.660 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T06:25:51.813 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T06:25:51.874 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T06:25:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T06:25:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T06:25:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T06:25:52.992 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-03-24T06:25:52.993 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T06:25:53.150 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:25:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T06:25:53.165 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:25:53.165 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:53.165 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T06:25:53.165 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T06:25:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T06:25:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T06:25:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T06:25:54.401 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T06:25:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T06:25:54.653 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:25:54.678 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:25:54.678 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:54.678 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T06:25:54.679 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:25:54.679 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:25:54.679 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:54.718 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:25:54.761 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T06:25:54.792 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T06:25:54.819 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T06:25:54.912 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:25:54.934 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:25:54.934 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:54.934 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T06:25:54.934 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:25:54.935 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:25:54.935 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T06:25:55.110 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T06:25:55.151 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T06:25:55.199 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T06:25:55.252 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T06:25:55.303 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T06:25:55.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T06:25:55.359 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T06:25:55.379 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:25:55.389 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T06:25:55.401 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:25:55.401 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:55.402 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T06:25:55.402 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:25:55.402 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:25:55.402 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:55.443 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T06:25:55.508 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T06:25:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T06:25:55.563 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T06:25:55.851 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:25:55.881 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:25:55.913 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T06:25:55.974 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T06:25:55.996 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:25:56.143 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:25:56.160 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T06:25:56.178 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:25:56.198 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:25:56.227 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:25:56.256 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:25:56.275 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T06:25:56.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T06:25:56.345 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T06:25:56.376 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T06:25:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T06:25:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:56.957 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T06:25:56.990 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T06:25:57.024 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T06:25:57.047 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T06:25:57.107 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T06:25:57.148 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T06:25:57.223 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:25:57.238 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T06:25:57.248 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:25:57.269 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T06:25:57.294 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T06:25:57.319 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T06:25:57.346 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T06:25:57.346 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:25:57.347 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:25:57.417 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:25:57.447 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T06:25:57.447 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:25:57.502 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T06:25:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:25:57.532 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:25:57.547 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:25:57.547 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:57.548 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T06:25:57.548 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:25:57.548 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:25:57.548 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:57.566 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T06:25:57.593 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:25:57.600 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T06:25:57.602 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T06:25:57.603 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:25:57.625 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T06:25:57.635 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:57.650 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T06:25:57.656 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T06:25:57.683 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T06:25:57.731 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T06:25:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T06:25:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T06:25:57.772 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T06:25:57.797 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T06:25:57.822 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T06:25:57.847 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:25:57.873 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T06:25:57.879 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:25:57.879 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:25:58.051 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:25:58.051 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:25:58.142 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:25:58.167 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:25:58.199 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T06:25:58.226 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T06:25:58.251 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T06:25:58.316 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:25:58.512 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T06:25:58.545 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:25:58.567 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:58.612 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:25:58.634 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:25:58.634 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:58.634 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T06:25:58.634 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:58.679 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:25:58.701 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:25:59.022 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:25:59.044 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:25:59.044 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:25:59.044 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T06:25:59.044 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:25:59.045 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:25:59.045 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:26:00.447 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:26:00.494 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:26:01.172 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T06:26:01.213 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:26:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T06:26:01.354 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T06:26:01.354 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T06:26:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:26:03.160 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:26:03.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T06:26:03.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:26:03.161 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T06:26:03.162 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T06:26:03.163 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T06:26:03.163 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T06:26:03.163 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T06:26:03.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T06:26:03.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T06:26:03.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T06:26:03.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T06:26:03.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T06:26:03.186 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T06:26:03.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T06:26:03.188 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T06:26:03.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T06:26:03.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T06:26:03.191 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T06:26:03.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T06:26:03.193 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T06:26:03.194 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T06:26:03.195 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T06:26:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T06:26:03.197 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:26:03.197 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T06:26:03.197 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T06:26:03.197 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T06:26:03.485 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:26:03.534 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T06:26:03.565 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T06:26:03.593 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T06:26:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T06:26:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T06:26:03.941 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T06:26:03.980 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T06:26:04.057 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T06:26:04.092 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T06:26:04.126 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T06:26:04.158 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T06:26:04.188 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T06:26:04.241 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T06:26:04.302 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T06:26:04.336 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T06:26:04.880 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:04.917 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:04.937 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:05.745 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T06:26:05.813 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T06:26:05.853 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T06:26:05.934 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T06:26:05.967 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T06:26:06.000 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T06:26:06.025 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T06:26:06.052 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T06:26:06.052 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:26:06.140 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:26:06.170 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T06:26:06.170 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:06.176 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.177 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.179 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T06:26:06.180 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.181 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T06:26:06.182 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T06:26:06.183 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T06:26:06.184 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T06:26:06.185 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.187 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:26:06.188 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T06:26:06.255 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:26:06.280 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T06:26:06.314 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T06:26:06.339 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T06:26:06.364 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T06:26:06.397 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T06:26:06.445 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T06:26:06.486 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T06:26:06.511 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T06:26:06.537 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T06:26:06.562 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:26:06.587 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T06:26:06.593 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:06.594 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:06.692 DEBUG:teuthology.parallel:result is None 2024-03-24T06:26:06.774 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:06.774 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:26:06.840 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:26:06.865 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:26:06.896 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T06:26:06.916 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T06:26:06.949 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T06:26:07.012 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:26:12.021 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:26:12.022 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T06:26:12.023 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T06:26:12.039 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T06:26:12.040 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T06:26:12.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T06:26:12.042 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T06:26:12.043 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T06:26:12.044 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T06:26:12.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T06:26:12.046 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T06:26:12.047 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T06:26:12.048 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T06:26:12.049 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T06:26:12.050 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /root 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2024-03-24T06:26:14.773 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:26:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T06:26:15.051 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:15.051 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T06:26:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T06:26:15.152 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T06:26:15.153 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T06:26:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: jq-1.6-16.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T06:26:15.156 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T06:26:15.157 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T06:26:15.158 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T06:26:15.159 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T06:26:15.160 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T06:26:15.161 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T06:26:15.162 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T06:26:15.163 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-24T06:26:15.633 DEBUG:teuthology.parallel:result is None 2024-03-24T06:26:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T06:26:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:17.071 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:26:17.148 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T06:26:17.149 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:17.149 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T06:26:17.149 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:26:17.149 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T06:26:17.149 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:17.648 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:26:17.745 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T06:26:17.746 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:17.746 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T06:26:17.746 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:26:17.746 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T06:26:17.746 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:17.867 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T06:26:18.492 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:26:18.534 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T06:26:18.586 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T06:26:18.682 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T06:26:18.713 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:26:19.049 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T06:26:19.080 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:26:20.249 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T06:26:20.280 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:26:20.431 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T06:26:20.612 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T06:26:20.641 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T06:26:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:26:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T06:26:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T06:26:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T06:26:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:26:22.042 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T06:26:22.042 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:22.042 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T06:26:22.042 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:26:22.043 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T06:26:22.043 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:22.094 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:26:22.184 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T06:26:22.184 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:22.184 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T06:26:22.184 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:22.303 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T06:26:22.394 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:22.804 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:26:22.882 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T06:26:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T06:26:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T06:26:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:26:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T06:26:22.883 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:27.961 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:26:28.014 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T06:26:28.075 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T06:26:28.114 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T06:26:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T06:26:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T06:26:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T06:26:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T06:26:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T06:26:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T06:26:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T06:26:28.868 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T06:26:28.920 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T06:26:29.011 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T06:26:29.123 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T06:26:29.168 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T06:26:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T06:26:31.581 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T06:26:31.688 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T06:26:31.748 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T06:26:31.882 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T06:26:31.915 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T06:26:31.948 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T06:26:31.984 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T06:26:32.026 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T06:26:32.026 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:26:32.173 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T06:26:32.214 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T06:26:32.214 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:26:32.335 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T06:26:32.386 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T06:26:32.434 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T06:26:32.500 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T06:26:32.532 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T06:26:32.602 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T06:26:32.698 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T06:26:32.746 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T06:26:32.779 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T06:26:32.813 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T06:26:32.846 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:26:32.879 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T06:26:32.916 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:32.917 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:33.281 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T06:26:33.281 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:26:33.411 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T06:26:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:26:33.495 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T06:26:33.527 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T06:26:33.567 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T06:26:33.690 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:26:33.960 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T06:26:33.960 INFO:teuthology.orchestra.run.smithi022.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-24T06:26:33.961 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T06:26:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T06:26:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T06:26:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T06:26:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T06:26:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T06:26:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T06:26:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T06:26:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T06:26:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T06:26:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T06:26:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T06:26:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T06:26:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T06:26:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T06:26:41.207 INFO:teuthology.orchestra.run.smithi022.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:41.208 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.209 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.210 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-16.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T06:26:41.211 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T06:26:41.212 INFO:teuthology.orchestra.run.smithi022.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T06:26:41.213 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T06:26:41.214 INFO:teuthology.orchestra.run.smithi022.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T06:26:41.215 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T06:26:41.216 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T06:26:41.217 INFO:teuthology.orchestra.run.smithi022.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T06:26:41.218 INFO:teuthology.orchestra.run.smithi022.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T06:26:41.219 INFO:teuthology.orchestra.run.smithi022.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T06:26:41.219 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T06:26:41.219 INFO:teuthology.orchestra.run.smithi022.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T06:26:41.219 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:26:41.219 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T06:26:42.026 DEBUG:teuthology.parallel:result is None 2024-03-24T06:26:42.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:42.167 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T06:26:42.224 INFO:teuthology.orchestra.run.smithi022.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.225 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.225 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T06:26:42.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:42.450 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T06:26:42.476 INFO:teuthology.orchestra.run.smithi143.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.476 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.476 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T06:26:42.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:42.618 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T06:26:42.644 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.644 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T06:26:42.645 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T06:26:42.646 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T06:26:42.647 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:26:42.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T06:26:42.728 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:26:42.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T06:26:42.762 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:26:42.763 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T06:26:42.797 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T06:26:42.798 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:26:42.798 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T06:26:42.877 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T06:26:42.995 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:26:42.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T06:26:43.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T06:26:43.095 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:26:43.095 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T06:26:43.127 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T06:26:43.195 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T06:26:43.195 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:26:43.195 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T06:26:43.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T06:26:43.394 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:26:43.394 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T06:26:43.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T06:26:43.489 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:26:43.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T06:26:43.517 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T06:26:43.584 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T06:26:43.585 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:26:43.585 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T06:26:43.671 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T06:26:43.782 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:26:43.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T06:26:43.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T06:26:43.876 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:26:43.876 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T06:26:43.903 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T06:26:43.970 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T06:26:44.071 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-24T06:26:44.072 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:44.073 INFO:tasks.cephadm:Cluster fsid is 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:26:44.073 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T06:26:44.073 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.22', 'mon.b': '172.21.15.143', 'mon.c': '172.21.15.155'} 2024-03-24T06:26:44.073 INFO:tasks.cephadm:First mon is mon.a on smithi022 2024-03-24T06:26:44.073 INFO:tasks.cephadm:First mgr is x 2024-03-24T06:26:44.073 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T06:26:44.073 DEBUG:teuthology.orchestra.run.smithi022:> sudo hostname $(hostname -s) 2024-03-24T06:26:44.153 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-03-24T06:26:44.180 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-03-24T06:26:44.207 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T06:26:44.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:44.423 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77900/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1855-g21b3bd67', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1855.g21b3bd67'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 04:48:23.862772', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240323.003144-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T06:26:44.570 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-batrick-testing-20240323.003144-squid, sha1 21b3bd672163092b4e542f772797fcba02d3d4de from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:26:44.571 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm 2024-03-24T06:26:44.571 DEBUG:teuthology.orchestra.run.smithi022:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:45.249 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 06:26 /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:45.249 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:45.729 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 06:26 /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:45.730 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:46.169 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 06:26 /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:46.169 DEBUG:teuthology.orchestra.run.smithi022:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:46.217 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:46.236 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T06:26:46.270 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de on all hosts... 2024-03-24T06:26:46.271 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T06:26:46.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T06:26:46.278 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T06:26:47.128 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T06:26:47.168 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T06:26:47.697 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout: "repo_digests": [ 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:27:22.372 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:27:28.184 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T06:27:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T06:27:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T06:27:28.185 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-03-24T06:27:28.185 INFO:teuthology.orchestra.run.smithi155.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T06:27:28.185 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-03-24T06:27:28.185 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-24T06:27:30.538 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T06:27:30.560 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph 2024-03-24T06:27:30.672 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-03-24T06:27:30.701 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-03-24T06:27:30.737 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /etc/ceph 2024-03-24T06:27:30.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-03-24T06:27:30.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-03-24T06:27:30.891 INFO:tasks.cephadm:Writing seed config... 2024-03-24T06:27:30.892 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T06:27:30.892 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T06:27:30.892 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T06:27:30.892 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T06:27:30.892 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T06:27:30.893 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T06:27:30.894 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T06:27:30.895 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-24T06:27:30.896 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:27:30.896 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T06:27:32.005 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7bca48fe-e9a7-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T06:27:32.006 DEBUG:teuthology.orchestra.run.smithi022:mon.a> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T06:27:32.052 DEBUG:teuthology.orchestra.run.smithi022:mgr.x> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T06:27:32.097 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T06:27:32.098 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de -v bootstrap --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.22 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:27:32.367 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T06:27:32.367 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de', '-v', 'bootstrap', '--fsid', '7bca48fe-e9a7-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.22', '--skip-admin-label'] 2024-03-24T06:27:32.427 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T06:27:32.427 INFO:teuthology.orchestra.run.smithi022.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T06:27:32.428 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying podman|docker is present... 2024-03-24T06:27:32.479 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T06:27:32.480 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying lvm2 is present... 2024-03-24T06:27:32.480 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying time synchronization is in place... 2024-03-24T06:27:32.495 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T06:27:32.496 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T06:27:32.509 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T06:27:32.509 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:27:32.527 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout enabled 2024-03-24T06:27:32.542 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout active 2024-03-24T06:27:32.542 INFO:teuthology.orchestra.run.smithi022.stdout:Unit chronyd.service is enabled and running 2024-03-24T06:27:32.543 INFO:teuthology.orchestra.run.smithi022.stdout:Repeating the final host check... 2024-03-24T06:27:32.592 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T06:27:32.592 INFO:teuthology.orchestra.run.smithi022.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T06:27:32.592 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl is present 2024-03-24T06:27:32.592 INFO:teuthology.orchestra.run.smithi022.stdout:lvcreate is present 2024-03-24T06:27:32.605 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T06:27:32.605 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T06:27:32.618 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T06:27:32.619 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:27:32.633 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout enabled 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout active 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:Unit chronyd.service is enabled and running 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:Host looks OK 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:Cluster fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:Acquiring lock 140319575125968 on /run/cephadm/7bca48fe-e9a7-11ee-95cd-87774f69a715.lock 2024-03-24T06:27:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:Lock 140319575125968 acquired on /run/cephadm/7bca48fe-e9a7-11ee-95cd-87774f69a715.lock 2024-03-24T06:27:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying IP 172.21.15.22 port 3300 ... 2024-03-24T06:27:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying IP 172.21.15.22 port 6789 ... 2024-03-24T06:27:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:Base mon IP(s) is [172.21.15.22:3300, 172.21.15.22:6789], mon addrv is [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T06:27:32.658 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.22 metric 100 2024-03-24T06:27:32.658 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.22 metric 100 2024-03-24T06:27:32.665 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T06:27:32.665 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T06:27:32.665 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T06:27:32.671 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T06:27:32.672 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T06:27:32.672 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T06:27:32.672 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T06:27:32.672 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T06:27:32.672 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T06:27:32.673 INFO:teuthology.orchestra.run.smithi022.stdout:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2024-03-24T06:27:32.673 INFO:teuthology.orchestra.run.smithi022.stdout:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2024-03-24T06:27:32.673 INFO:teuthology.orchestra.run.smithi022.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T06:27:32.673 INFO:teuthology.orchestra.run.smithi022.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T06:27:32.674 INFO:teuthology.orchestra.run.smithi022.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T06:27:36.196 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T06:27:36.196 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T06:27:36.196 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T06:27:36.197 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying blob sha256:30dc77a9f705c02d26e1a1f9e2bfdf9d08cd4086b08b221e8a27d0dd2909924d 2024-03-24T06:27:36.197 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T06:27:36.197 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying config sha256:553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T06:27:36.197 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T06:27:37.791 INFO:teuthology.orchestra.run.smithi022.stdout:ceph: stdout ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T06:27:37.791 INFO:teuthology.orchestra.run.smithi022.stdout:Ceph version: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T06:27:37.791 INFO:teuthology.orchestra.run.smithi022.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T06:27:39.033 INFO:teuthology.orchestra.run.smithi022.stdout:stat: stdout 167 167 2024-03-24T06:27:39.033 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial keys... 2024-03-24T06:27:40.050 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDbx/9lk1BDGRAApVPB28+MB7cU8FA4fbfZ9g== 2024-03-24T06:27:41.514 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDcx/9lKcrxHhAA7VW8s4QISFV43CUcu4OzCQ== 2024-03-24T06:27:42.630 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDdx/9lD2afORAASGeC4La4V8Cd9AFEMGWWRQ== 2024-03-24T06:27:42.634 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial monmap... 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool for a [v2:172.21.15.22:3300,v1:172.21.15.22:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = quincy 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: set fsid to 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T06:27:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:27:43.742 INFO:teuthology.orchestra.run.smithi022.stdout:Creating mon... 2024-03-24T06:27:44.883 INFO:teuthology.orchestra.run.smithi022.stdout:create mon.a on 2024-03-24T06:27:45.256 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T06:27:45.504 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T06:27:45.802 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target. 2024-03-24T06:27:45.802 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target. 2024-03-24T06:27:46.232 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a 2024-03-24T06:27:46.232 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T06:27:46.542 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target.wants/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@.service. 2024-03-24T06:27:47.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:46 smithi022 podman[31610]: 2024-03-24 06:27:46.974372972 +0000 UTC m=+0.170528684 container create c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T06:27:47.254 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 podman[31610]: 2024-03-24 06:27:47.252498336 +0000 UTC m=+0.448654121 container init c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T06:27:47.372 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T06:27:47.372 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T06:27:47.388 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T06:27:47.388 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:27:47.389 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T06:27:47.389 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T06:27:47.391 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mon to start... 2024-03-24T06:27:47.391 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mon... 2024-03-24T06:27:47.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 podman[31610]: 2024-03-24 06:27:47.263262232 +0000 UTC m=+0.459417917 container start c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-24T06:27:47.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 bash[31610]: c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c 2024-03-24T06:27:47.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 ceph-mon[31629]: mkfs 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:47.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 systemd[1]: Started Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:27:47.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:47 smithi022 ceph-mon[31629]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T06:27:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: monmap epoch 1 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: last_changed 2024-03-24T06:27:43.107757+0000 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: min_mon_release 19 (squid) 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: election_strategy: 1 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: fsmap 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: mgrmap e1: no daemons active 2024-03-24T06:27:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:48 smithi022 ceph-mon[31629]: from='client.? 172.21.15.22:0/3177595117' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout id: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout services: 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.785021s) 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T06:27:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T06:27:48.766 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:48.771 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout data: 2024-03-24T06:27:48.771 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.032+0000 7f0e40c00700 1 Processor -- start 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.032+0000 7f0e40c00700 1 -- start start 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.033+0000 7f0e40c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.033+0000 7f0e40c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c109dd0 con 0x7f0e3c1093e0 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.034+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.034+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47518/0 (socket says 172.21.15.22:47518) 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.034+0000 7f0e39e00700 1 -- 172.21.15.22:0/1798183880 learned_addr learned my addr 172.21.15.22:0/1798183880 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:48.772 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.034+0000 7f0e39e00700 1 -- 172.21.15.22:0/1798183880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c107920 con 0x7f0e3c1093e0 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.035+0000 7f0e39e00700 1 --2- 172.21.15.22:0/1798183880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0098a0 tx=0x7f0e2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd38df28cca652bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.035+0000 7f0e38a00700 1 -- 172.21.15.22:0/1798183880 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e2c004160 con 0x7f0e3c1093e0 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.035+0000 7f0e38a00700 1 -- 172.21.15.22:0/1798183880 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e2c0042c0 con 0x7f0e3c1093e0 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.035+0000 7f0e38a00700 1 -- 172.21.15.22:0/1798183880 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e2c0045a0 con 0x7f0e3c1093e0 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 -- 172.21.15.22:0/1798183880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 msgr2=0x7f0e3c109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 --2- 172.21.15.22:0/1798183880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0098a0 tx=0x7f0e2c009bb0 comp rx=0 tx=0).stop 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 -- 172.21.15.22:0/1798183880 shutdown_connections 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 --2- 172.21.15.22:0/1798183880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c109800 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 -- 172.21.15.22:0/1798183880 >> 172.21.15.22:0/1798183880 conn(0x7f0e3c1032a0 msgr2=0x7f0e3c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 -- 172.21.15.22:0/1798183880 shutdown_connections 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.036+0000 7f0e40c00700 1 -- 172.21.15.22:0/1798183880 wait complete. 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.038+0000 7f0e40c00700 1 Processor -- start 2024-03-24T06:27:48.773 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.038+0000 7f0e40c00700 1 -- start start 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.038+0000 7f0e40c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.038+0000 7f0e40c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c19b890 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.039+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.039+0000 7f0e39e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47522/0 (socket says 172.21.15.22:47522) 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.039+0000 7f0e39e00700 1 -- 172.21.15.22:0/3177595117 learned_addr learned my addr 172.21.15.22:0/3177595117 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.039+0000 7f0e39e00700 1 -- 172.21.15.22:0/3177595117 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c009550 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e39e00700 1 --2- 172.21.15.22:0/3177595117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c004000 tx=0x7f0e2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e2c011630 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e2c011790 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c19ba90 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e2c011a30 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c19bf30 con 0x7f0e3c1093e0 2024-03-24T06:27:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0e2c022070 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.040+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0e2c01c450 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.041+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3c194fc0 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.045+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0e2c011b90 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.126+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0e3c107ec0 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.126+0000 7f0e32a00700 1 -- 172.21.15.22:0/3177595117 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f0e2c011d30 con 0x7f0e3c1093e0 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.128+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 msgr2=0x7f0e3c19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.128+0000 7f0e40c00700 1 --2- 172.21.15.22:0/3177595117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c004000 tx=0x7f0e2c004750 comp rx=0 tx=0).stop 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.128+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 shutdown_connections 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.128+0000 7f0e40c00700 1 --2- 172.21.15.22:0/3177595117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0e3c1093e0 0x7f0e3c19b350 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.128+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 >> 172.21.15.22:0/3177595117 conn(0x7f0e3c1032a0 msgr2=0x7f0e3c192100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.129+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 shutdown_connections 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:48.129+0000 7f0e40c00700 1 -- 172.21.15.22:0/3177595117 wait complete. 2024-03-24T06:27:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:mon is available 2024-03-24T06:27:48.776 INFO:teuthology.orchestra.run.smithi022.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T06:27:49.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:49 smithi022 ceph-mon[31629]: from='client.? 172.21.15.22:0/804106590' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T06:27:49.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:49 smithi022 ceph-mon[31629]: from='client.? 172.21.15.22:0/804106590' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T06:27:49.998 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout fsid = 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:49.999 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.360+0000 7f9b43e00700 1 Processor -- start 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.361+0000 7f9b43e00700 1 -- start start 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.361+0000 7f9b43e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.361+0000 7f9b43e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c109dc0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.362+0000 7f9b41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.362+0000 7f9b41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47538/0 (socket says 172.21.15.22:47538) 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.362+0000 7f9b41400700 1 -- 172.21.15.22:0/1569825315 learned_addr learned my addr 172.21.15.22:0/1569825315 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:50.000 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.362+0000 7f9b41400700 1 -- 172.21.15.22:0/1569825315 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c107910 con 0x7f9b3c1093d0 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.363+0000 7f9b41400700 1 --2- 172.21.15.22:0/1569825315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098a0 tx=0x7f9b30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526c787154fe4074 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.363+0000 7f9b3be00700 1 -- 172.21.15.22:0/1569825315 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b30004160 con 0x7f9b3c1093d0 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.363+0000 7f9b3be00700 1 -- 172.21.15.22:0/1569825315 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b300042c0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.363+0000 7f9b3be00700 1 -- 172.21.15.22:0/1569825315 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b300045a0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 -- 172.21.15.22:0/1569825315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 msgr2=0x7f9b3c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 --2- 172.21.15.22:0/1569825315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098a0 tx=0x7f9b30009bb0 comp rx=0 tx=0).stop 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 -- 172.21.15.22:0/1569825315 shutdown_connections 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 --2- 172.21.15.22:0/1569825315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c1097f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 -- 172.21.15.22:0/1569825315 >> 172.21.15.22:0/1569825315 conn(0x7f9b3c1032b0 msgr2=0x7f9b3c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 -- 172.21.15.22:0/1569825315 shutdown_connections 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.364+0000 7f9b43e00700 1 -- 172.21.15.22:0/1569825315 wait complete. 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.365+0000 7f9b43e00700 1 Processor -- start 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.365+0000 7f9b43e00700 1 -- start start 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.366+0000 7f9b43e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:50.001 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.366+0000 7f9b43e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c19dac0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.366+0000 7f9b41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.366+0000 7f9b41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47552/0 (socket says 172.21.15.22:47552) 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.366+0000 7f9b41400700 1 -- 172.21.15.22:0/804106590 learned_addr learned my addr 172.21.15.22:0/804106590 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b41400700 1 -- 172.21.15.22:0/804106590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b30009550 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b41400700 1 --2- 172.21.15.22:0/804106590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9b30004000 tx=0x7f9b30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b30011630 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b30011790 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c19dcc0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b30011a30 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.367+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c19e160 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.368+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9b30022070 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.368+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9b3001c450 con 0x7f9b3c1093d0 2024-03-24T06:27:50.002 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.369+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b3c107c80 con 0x7f9b3c1093d0 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.371+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9b30011b90 con 0x7f9b3c1093d0 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.450+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f9b3c107e10 con 0x7f9b3c1093d0 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.457+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f9b30011de0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.458+0000 7f9b3a000700 1 -- 172.21.15.22:0/804106590 <== mon.0 v2:172.21.15.22:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9b3001c6b0 con 0x7f9b3c1093d0 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 msgr2=0x7f9b3c19d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 --2- 172.21.15.22:0/804106590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9b30004000 tx=0x7f9b30004750 comp rx=0 tx=0).stop 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 shutdown_connections 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 --2- 172.21.15.22:0/804106590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9b3c1093d0 0x7f9b3c19d580 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 >> 172.21.15.22:0/804106590 conn(0x7f9b3c1032b0 msgr2=0x7f9b3c07c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 shutdown_connections 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:49.460+0000 7f9b43e00700 1 -- 172.21.15.22:0/804106590 wait complete. 2024-03-24T06:27:50.003 INFO:teuthology.orchestra.run.smithi022.stdout:Generating new minimal ceph.conf... 2024-03-24T06:27:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:50 smithi022 ceph-mon[31629]: from='client.? 172.21.15.22:0/1129583880' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:27:51.313 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.568+0000 7fd5dac00700 1 Processor -- start 2024-03-24T06:27:51.313 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.569+0000 7fd5dac00700 1 -- start start 2024-03-24T06:27:51.313 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.569+0000 7fd5dac00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:51.313 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.569+0000 7fd5dac00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d410aab0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.569+0000 7fd5d9800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.570+0000 7fd5d9800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47556/0 (socket says 172.21.15.22:47556) 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.570+0000 7fd5d9800700 1 -- 172.21.15.22:0/3668885093 learned_addr learned my addr 172.21.15.22:0/3668885093 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.570+0000 7fd5d9800700 1 -- 172.21.15.22:0/3668885093 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d410aee0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.570+0000 7fd5d9800700 1 --2- 172.21.15.22:0/3668885093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40098a0 tx=0x7fd5c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1034d70eb952e7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5d3e00700 1 -- 172.21.15.22:0/3668885093 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5c4004160 con 0x7fd5d410a0c0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5d3e00700 1 -- 172.21.15.22:0/3668885093 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5c40042c0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5d3e00700 1 -- 172.21.15.22:0/3668885093 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5c4011ac0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 -- 172.21.15.22:0/3668885093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 msgr2=0x7fd5d410a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 --2- 172.21.15.22:0/3668885093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40098a0 tx=0x7fd5c4009bb0 comp rx=0 tx=0).stop 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 -- 172.21.15.22:0/3668885093 shutdown_connections 2024-03-24T06:27:51.314 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 --2- 172.21.15.22:0/3668885093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d410a4e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 -- 172.21.15.22:0/3668885093 >> 172.21.15.22:0/3668885093 conn(0x7fd5d407d290 msgr2=0x7fd5d407d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.571+0000 7fd5dac00700 1 -- 172.21.15.22:0/3668885093 shutdown_connections 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.572+0000 7fd5dac00700 1 -- 172.21.15.22:0/3668885093 wait complete. 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5dac00700 1 Processor -- start 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5dac00700 1 -- start start 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5dac00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5dac00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d419dfe0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5d9800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5d9800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47568/0 (socket says 172.21.15.22:47568) 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.573+0000 7fd5d9800700 1 -- 172.21.15.22:0/1129583880 learned_addr learned my addr 172.21.15.22:0/1129583880 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5d9800700 1 -- 172.21.15.22:0/1129583880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c4009550 con 0x7fd5d410a0c0 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5d9800700 1 --2- 172.21.15.22:0/1129583880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009130 tx=0x7fd5c4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5c4003aa0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.315 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5c4003c00 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5c4024800 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d419e1e0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.574+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d419e680 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.575+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5c402d3f0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.575+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd5c4019760 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.576+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d410e110 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.579+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd5c4024e00 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.657+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd5d41a11e0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.657+0000 7fd5d2000700 1 -- 172.21.15.22:0/1129583880 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fd5c401abd0 con 0x7fd5d410a0c0 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 msgr2=0x7fd5d419daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 --2- 172.21.15.22:0/1129583880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009130 tx=0x7fd5c4011b40 comp rx=0 tx=0).stop 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 shutdown_connections 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 --2- 172.21.15.22:0/1129583880 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5d410a0c0 0x7fd5d419daa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:51.316 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 >> 172.21.15.22:0/1129583880 conn(0x7fd5d407d290 msgr2=0x7fd5d4107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:51.317 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 shutdown_connections 2024-03-24T06:27:51.317 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:50.659+0000 7fd5dac00700 1 -- 172.21.15.22:0/1129583880 wait complete. 2024-03-24T06:27:51.317 INFO:teuthology.orchestra.run.smithi022.stdout:Restarting the monitor... 2024-03-24T06:27:51.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:51 smithi022 systemd[1]: Stopping Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:27:52.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a[31625]: 2024-03-24T06:27:51.796+0000 7f270be00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:27:52.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a[31625]: 2024-03-24T06:27:51.796+0000 7f270be00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T06:27:52.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:51 smithi022 podman[31871]: 2024-03-24 06:27:51.837938149 +0000 UTC m=+0.159901642 container died c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:27:52.643 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:52 smithi022 podman[31871]: 2024-03-24 06:27:52.467629088 +0000 UTC m=+0.789592578 container cleanup c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:27:52.643 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:52 smithi022 bash[31871]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a 2024-03-24T06:27:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:52 smithi022 podman[31885]: 2024-03-24 06:27:52.722788254 +0000 UTC m=+0.864064243 container remove c1e1a0c373f304c2c7343ccfeaaf69550da20d3e5a6608a8fba06e5311353d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T06:27:53.578 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 systemd[1]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T06:27:53.578 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 systemd[1]: Stopped Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:27:53.579 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 systemd[1]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service: Consumed 1.870s CPU time. 2024-03-24T06:27:53.579 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 systemd[1]: Starting Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:27:53.872 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 podman[31984]: 2024-03-24 06:27:53.484885861 +0000 UTC m=+0.040489067 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:27:53.873 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 podman[31984]: 2024-03-24 06:27:53.581982162 +0000 UTC m=+0.137585370 container create c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:27:53.873 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 podman[31984]: 2024-03-24 06:27:53.818557397 +0000 UTC m=+0.374160641 container init c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T06:27:53.873 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 podman[31984]: 2024-03-24 06:27:53.828595145 +0000 UTC m=+0.384198412 container start c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T06:27:53.918 INFO:teuthology.orchestra.run.smithi022.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev), process ceph-mon, pid 2 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: pidfile_write: ignore empty --pid-file 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: load: jerasure load: lrc 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: RocksDB version: 7.9.2 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Git sha 0 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Compile date 2024-03-23 03:28:38 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: DB SUMMARY 2024-03-24T06:27:54.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: DB Session ID: H9XK0ZF80PPQNPMXVQ51 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: CURRENT file: CURRENT 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105178 ; 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.error_if_exists: 0 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.create_if_missing: 0 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.paranoid_checks: 1 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.env: 0x557e24bcbc20 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.info_log: 0x557e26769b00 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T06:27:54.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.statistics: (nil) 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.use_fsync: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_log_file_size: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_fallocate: 1 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.use_direct_reads: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.db_log_dir: 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.wal_dir: 2024-03-24T06:27:54.152 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.write_buffer_manager: 0x557e2673c1e0 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T06:27:54.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.unordered_write: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.row_cache: None 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.wal_filter: None 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T06:27:54.154 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.two_write_queues: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.wal_compression: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.atomic_flush: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.log_readahead_size: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T06:27:54.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_background_jobs: 2 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_background_compactions: -1 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_subcompactions: 1 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T06:27:54.156 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_open_files: -1 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_background_flushes: -1 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Compression algorithms supported: 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kZSTD supported: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kXpressCompression supported: 0 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kZlibCompression supported: 1 2024-03-24T06:27:54.157 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kSnappyCompression supported: 1 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kLZ4Compression supported: 1 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: kBZip2Compression supported: 0 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.merge_operator: 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_filter: None 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T06:27:54.158 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e26769c20) 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: cache_index_and_filter_blocks: 1 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: pin_top_level_index_and_filter: 1 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_type: 0 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: data_block_index_type: 0 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_shortening: 1 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: checksum: 4 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: no_block_cache: 0 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache: 0x557e2676d090 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_name: BinnedLRUCache 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_options: 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: capacity : 536870912 2024-03-24T06:27:54.159 INFO:journalctl@ceph.mon.a.smithi022.stdout: num_shard_bits : 4 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: strict_capacity_limit : 0 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: high_pri_pool_ratio: 0.000 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_cache_compressed: (nil) 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: persistent_cache: (nil) 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_size: 4096 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_size_deviation: 10 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_restart_interval: 16 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: index_block_restart_interval: 1 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: metadata_block_size: 4096 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: partition_filters: 0 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: use_delta_encoding: 1 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: filter_policy: bloomfilter 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: whole_key_filtering: 1 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: verify_compression: 0 2024-03-24T06:27:54.160 INFO:journalctl@ceph.mon.a.smithi022.stdout: read_amp_bytes_per_bit: 0 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: format_version: 5 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: enable_index_compression: 1 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: block_align: 0 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: max_auto_readahead_size: 262144 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: prepopulate_block_cache: 0 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: initial_auto_readahead_size: 8192 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression: NoCompression 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.num_levels: 7 2024-03-24T06:27:54.161 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T06:27:54.162 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:27:54.163 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T06:27:54.164 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T06:27:54.165 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.table_properties_collectors: 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.inplace_update_support: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.bloom_locality: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.max_successive_merges: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.ttl: 2592000 2024-03-24T06:27:54.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enable_blob_files: false 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.min_blob_size: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 293c1683-e87f-45b2-8bce-bd9b38ef8e53 2024-03-24T06:27:54.167 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261673885932, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261673887813, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98145, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14284, "raw_average_key_size": 49, "raw_value_size": 90409, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711261673, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "293c1683-e87f-45b2-8bce-bd9b38ef8e53", "db_session_id": "H9XK0ZF80PPQNPMXVQ51", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261673888064, "job": 1, "event": "recovery_finished"} 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557e2685e000 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: DB pointer 0x557e26848000 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** DB Stats ** 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:27:54.168 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: L0 2/0 99.72 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Sum 2/0 99.72 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 84.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:27:54.169 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Cumulative compaction: 0.00 GB write, 11.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Interval compaction: 0.00 GB write, 11.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T06:27:54.170 INFO:journalctl@ceph.mon.a.smithi022.stdout: Block cache BinnedLRUCache@0x557e2676d090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: starting mon.a rank 0 at public addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] at bind addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???) e1 preinit fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).mds e1 new map 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).mds e1 print_map 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: e1 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: legacy client fscid: -1 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout: No filesystems configured 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:27:54.171 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 bash[31984]: c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 systemd[1]: Started Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: monmap epoch 1 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: last_changed 2024-03-24T06:27:43.107757+0000 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: min_mon_release 19 (squid) 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: election_strategy: 1 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: fsmap 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T06:27:54.172 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:53 smithi022 ceph-mon[32003]: mgrmap e1: no daemons active 2024-03-24T06:27:55.205 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.517+0000 7f52a4000700 1 Processor -- start 2024-03-24T06:27:55.206 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.518+0000 7f52a4000700 1 -- start start 2024-03-24T06:27:55.206 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.518+0000 7f52a4000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:55.206 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.518+0000 7f52a4000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f529c10b0a0 con 0x7f529c108770 2024-03-24T06:27:55.207 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.519+0000 7f52a1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:55.207 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.519+0000 7f52a1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47578/0 (socket says 172.21.15.22:47578) 2024-03-24T06:27:55.207 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.519+0000 7f52a1600700 1 -- 172.21.15.22:0/1469208183 learned_addr learned my addr 172.21.15.22:0/1469208183 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:55.211 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.520+0000 7f52a1600700 1 -- 172.21.15.22:0/1469208183 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c10b1e0 con 0x7f529c108770 2024-03-24T06:27:55.211 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.520+0000 7f52a1600700 1 --2- 172.21.15.22:0/1469208183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5290009e80 tx=0x7f529000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3b154ee4fe01a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:55.211 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.520+0000 7f529be00700 1 -- 172.21.15.22:0/1469208183 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f529000e070 con 0x7f529c108770 2024-03-24T06:27:55.211 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.520+0000 7f529be00700 1 -- 172.21.15.22:0/1469208183 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f529000b8c0 con 0x7f529c108770 2024-03-24T06:27:55.211 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.520+0000 7f529be00700 1 -- 172.21.15.22:0/1469208183 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5290004550 con 0x7f529c108770 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 -- 172.21.15.22:0/1469208183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 msgr2=0x7f529c10ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 --2- 172.21.15.22:0/1469208183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5290009e80 tx=0x7f529000b330 comp rx=0 tx=0).stop 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 -- 172.21.15.22:0/1469208183 shutdown_connections 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 --2- 172.21.15.22:0/1469208183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c10ab60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 -- 172.21.15.22:0/1469208183 >> 172.21.15.22:0/1469208183 conn(0x7f529c102180 msgr2=0x7f529c1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 -- 172.21.15.22:0/1469208183 shutdown_connections 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.521+0000 7f52a4000700 1 -- 172.21.15.22:0/1469208183 wait complete. 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.523+0000 7f52a4000700 1 Processor -- start 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.523+0000 7f52a4000700 1 -- start start 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a4000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a4000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f529c19bff0 con 0x7f529c108770 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47588/0 (socket says 172.21.15.22:47588) 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a1600700 1 -- 172.21.15.22:0/2096123776 learned_addr learned my addr 172.21.15.22:0/2096123776 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:55.212 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.524+0000 7f52a1600700 1 -- 172.21.15.22:0/2096123776 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5290009b30 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f52a1600700 1 --2- 172.21.15.22:0/2096123776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5290009fd0 tx=0x7f5290003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f529000e040 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52900128f0 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f529001c890 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c19c1f0 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.525+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c19c670 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.526+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f529001cb00 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.526+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f529001b850 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.527+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529c063830 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.530+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f529002a060 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.607+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f529c10c1c0 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.611+0000 7f529a000700 1 -- 172.21.15.22:0/2096123776 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f5290020070 con 0x7f529c108770 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 msgr2=0x7f529c19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 --2- 172.21.15.22:0/2096123776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5290009fd0 tx=0x7f5290003d70 comp rx=0 tx=0).stop 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 shutdown_connections 2024-03-24T06:27:55.213 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 --2- 172.21.15.22:0/2096123776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f529c108770 0x7f529c19bab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:55.214 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 >> 172.21.15.22:0/2096123776 conn(0x7f529c102180 msgr2=0x7f529c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:55.214 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 shutdown_connections 2024-03-24T06:27:55.214 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:54.612+0000 7f52a4000700 1 -- 172.21.15.22:0/2096123776 wait complete. 2024-03-24T06:27:55.214 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T06:27:55.238 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:27:55.238 INFO:teuthology.orchestra.run.smithi022.stdout:Creating mgr... 2024-03-24T06:27:55.240 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T06:27:55.241 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T06:27:55.659 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x 2024-03-24T06:27:55.659 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T06:27:55.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:55 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2096123776' entity='client.admin' 2024-03-24T06:27:55.921 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715.target.wants/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@.service. 2024-03-24T06:27:56.308 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:55 smithi022 systemd[1]: Starting Ceph mgr.x for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:27:56.610 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 podman[32217]: 2024-03-24 06:27:56.213729847 +0000 UTC m=+0.032534745 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T06:27:56.610 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 podman[32217]: 2024-03-24 06:27:56.308626961 +0000 UTC m=+0.127431858 container create c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T06:27:56.610 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 podman[32217]: 2024-03-24 06:27:56.529796806 +0000 UTC m=+0.348601777 container init c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T06:27:56.610 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 podman[32217]: 2024-03-24 06:27:56.54422149 +0000 UTC m=+0.363026391 container start c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T06:27:56.610 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 bash[32217]: c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 2024-03-24T06:27:56.639 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T06:27:56.639 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T06:27:56.654 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T06:27:56.654 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:27:56.654 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T06:27:56.654 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T06:27:56.673 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T06:27:56.674 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T06:27:56.693 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T06:27:56.693 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:27:56.693 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T06:27:56.694 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T06:27:56.694 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr to start... 2024-03-24T06:27:56.694 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr... 2024-03-24T06:27:56.894 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:56 smithi022 systemd[1]: Started Ceph mgr.x for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:27:57.435 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:57.356+0000 7fa9774fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:27:57.795 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:27:57 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1898358983' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T06:27:57.796 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:57.486+0000 7fa9774fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:27:58.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:27:58.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:27:58.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:27:58.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-24T06:27:58.122 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T06:27:58.123 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:27:58.123 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T06:27:58.123 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T06:27:58.123 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.123 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T06:27:58.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:27:58.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T06:27:58.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T06:27:58.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T06:27:58.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T06:27:47.344229+0000", 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.342+0000 7f7106400700 1 Processor -- start 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.342+0000 7f7106400700 1 -- start start 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.342+0000 7f7106400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.342+0000 7f7106400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f710007df80 con 0x7f710007c2a0 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.343+0000 7f7105000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:58.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.343+0000 7f7105000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43048/0 (socket says 172.21.15.22:43048) 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.343+0000 7f7105000700 1 -- 172.21.15.22:0/2300053413 learned_addr learned my addr 172.21.15.22:0/2300053413 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.343+0000 7f7105000700 1 -- 172.21.15.22:0/2300053413 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710007e0c0 con 0x7f710007c2a0 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.344+0000 7f7105000700 1 --2- 172.21.15.22:0/2300053413 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f70f40098a0 tx=0x7f70f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfffaeab564d1235 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.344+0000 7f70ff400700 1 -- 172.21.15.22:0/2300053413 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70f4004160 con 0x7f710007c2a0 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.344+0000 7f70ff400700 1 -- 172.21.15.22:0/2300053413 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f70f40042c0 con 0x7f710007c2a0 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.344+0000 7f70ff400700 1 -- 172.21.15.22:0/2300053413 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70f4011ac0 con 0x7f710007c2a0 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 -- 172.21.15.22:0/2300053413 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 msgr2=0x7f710007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 --2- 172.21.15.22:0/2300053413 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f70f40098a0 tx=0x7f70f4009bb0 comp rx=0 tx=0).stop 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 -- 172.21.15.22:0/2300053413 shutdown_connections 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 --2- 172.21.15.22:0/2300053413 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710007c6c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 -- 172.21.15.22:0/2300053413 >> 172.21.15.22:0/2300053413 conn(0x7f7100103090 msgr2=0x7f71001054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 -- 172.21.15.22:0/2300053413 shutdown_connections 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.345+0000 7f7106400700 1 -- 172.21.15.22:0/2300053413 wait complete. 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7106400700 1 Processor -- start 2024-03-24T06:27:58.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7106400700 1 -- start start 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7106400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7106400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f710019e100 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7105000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7105000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43052/0 (socket says 172.21.15.22:43052) 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.347+0000 7f7105000700 1 -- 172.21.15.22:0/1898358983 learned_addr learned my addr 172.21.15.22:0/1898358983 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f7105000700 1 -- 172.21.15.22:0/1898358983 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f4009550 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f7105000700 1 --2- 172.21.15.22:0/1898358983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009130 tx=0x7f70f4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70f4003aa0 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f70f4003c00 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70f4024800 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.348+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710019e300 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.349+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710019e7a0 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.349+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f70f402d3f0 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.349+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f70f4019760 con 0x7f710007c2a0 2024-03-24T06:27:58.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.350+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710010dfc0 con 0x7f710007c2a0 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.354+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f70f4024e00 con 0x7f710007c2a0 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.435+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f71001a1220 con 0x7f710007c2a0 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.436+0000 7f70fd600700 1 -- 172.21.15.22:0/1898358983 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f70f401abd0 con 0x7f710007c2a0 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.437+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 msgr2=0x7f710019dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.437+0000 7f7106400700 1 --2- 172.21.15.22:0/1898358983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009130 tx=0x7f70f4011b40 comp rx=0 tx=0).stop 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.438+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 shutdown_connections 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.438+0000 7f7106400700 1 --2- 172.21.15.22:0/1898358983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f710007c2a0 0x7f710019dbc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.438+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 >> 172.21.15.22:0/1898358983 conn(0x7f7100103090 msgr2=0x7f7100103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.438+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 shutdown_connections 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:27:57.438+0000 7f7106400700 1 -- 172.21.15.22:0/1898358983 wait complete. 2024-03-24T06:27:58.138 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (1/15)... 2024-03-24T06:27:58.398 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:58.337+0000 7fa9774fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:27:58.773 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:58.464+0000 7fa9774fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:27:58.773 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:58.644+0000 7fa9774fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:27:58.773 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:58.773+0000 7fa9774fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:27:59.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:58.887+0000 7fa9774fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:27:59.495 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:59.231+0000 7fa9774fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:27:59.495 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:59.377+0000 7fa9774fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:27:59.844 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:59.494+0000 7fa9774fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:27:59.845 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:59.622+0000 7fa9774fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:28:00.544 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:27:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:27:59.843+0000 7fa9774fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:28:01.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:00 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3294062667' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T06:28:01.432 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:01 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:01.253+0000 7fa9774fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:28:01.433 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:01 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:01.432+0000 7fa9774fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T06:28:01.508 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T06:28:01.509 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T06:28:01.510 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T06:28:01.521 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T06:28:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T06:28:01.523 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T06:27:47.344229+0000", 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.701+0000 7ff43d000700 1 Processor -- start 2024-03-24T06:28:01.524 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.701+0000 7ff43d000700 1 -- start start 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.702+0000 7ff43d000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.702+0000 7ff43d000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff438109dd0 con 0x7ff4381093e0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.702+0000 7ff435e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.702+0000 7ff435e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43058/0 (socket says 172.21.15.22:43058) 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.702+0000 7ff435e00700 1 -- 172.21.15.22:0/3413133395 learned_addr learned my addr 172.21.15.22:0/3413133395 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.703+0000 7ff435e00700 1 -- 172.21.15.22:0/3413133395 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff438107920 con 0x7ff4381093e0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.703+0000 7ff435e00700 1 --2- 172.21.15.22:0/3413133395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff4280098a0 tx=0x7ff428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27750457490c71eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.703+0000 7ff434a00700 1 -- 172.21.15.22:0/3413133395 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428004160 con 0x7ff4381093e0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.703+0000 7ff434a00700 1 -- 172.21.15.22:0/3413133395 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff4280042c0 con 0x7ff4381093e0 2024-03-24T06:28:01.525 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.703+0000 7ff434a00700 1 -- 172.21.15.22:0/3413133395 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428011ac0 con 0x7ff4381093e0 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.704+0000 7ff43d000700 1 -- 172.21.15.22:0/3413133395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 msgr2=0x7ff438109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.704+0000 7ff43d000700 1 --2- 172.21.15.22:0/3413133395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff4280098a0 tx=0x7ff428009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.704+0000 7ff43d000700 1 -- 172.21.15.22:0/3413133395 shutdown_connections 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.704+0000 7ff43d000700 1 --2- 172.21.15.22:0/3413133395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff438109800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.704+0000 7ff43d000700 1 -- 172.21.15.22:0/3413133395 >> 172.21.15.22:0/3413133395 conn(0x7ff4381032a0 msgr2=0x7ff4381056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.705+0000 7ff43d000700 1 -- 172.21.15.22:0/3413133395 shutdown_connections 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.705+0000 7ff43d000700 1 -- 172.21.15.22:0/3413133395 wait complete. 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff43d000700 1 Processor -- start 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff43d000700 1 -- start start 2024-03-24T06:28:01.526 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff43d000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff43d000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff43819c100 con 0x7ff4381093e0 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff435e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff435e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43068/0 (socket says 172.21.15.22:43068) 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.706+0000 7ff435e00700 1 -- 172.21.15.22:0/3294062667 learned_addr learned my addr 172.21.15.22:0/3294062667 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff435e00700 1 -- 172.21.15.22:0/3294062667 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff428009550 con 0x7ff4381093e0 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff435e00700 1 --2- 172.21.15.22:0/3294062667 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff428000c00 tx=0x7ff428011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff428003aa0 con 0x7ff4381093e0 2024-03-24T06:28:01.527 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43819c300 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff428003c00 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4280248a0 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.707+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43819c7a0 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.708+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff42802d3f0 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.708+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff4280197c0 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.709+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff43810c130 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.711+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff42801ac60 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.790+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff43819f210 con 0x7ff4381093e0 2024-03-24T06:28:01.528 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.791+0000 7ff42ea00700 1 -- 172.21.15.22:0/3294062667 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff428035090 con 0x7ff4381093e0 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 msgr2=0x7ff43819bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 --2- 172.21.15.22:0/3294062667 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff428000c00 tx=0x7ff428011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 shutdown_connections 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 --2- 172.21.15.22:0/3294062667 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4381093e0 0x7ff43819bbc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 >> 172.21.15.22:0/3294062667 conn(0x7ff4381032a0 msgr2=0x7ff438103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 shutdown_connections 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:00.792+0000 7ff43d000700 1 -- 172.21.15.22:0/3294062667 wait complete. 2024-03-24T06:28:01.529 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (2/15)... 2024-03-24T06:28:02.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:01 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:01.856+0000 7fa9774fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:28:02.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:01 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:01.978+0000 7fa9774fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:28:02.496 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:02 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:02.244+0000 7fa9774fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:02.880 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:02 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:02.496+0000 7fa9774fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:28:03.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:02 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:02.879+0000 7fa9774fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:03.513 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:03 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:03.200+0000 7fa9774fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:28:03.833 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:03 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:03.513+0000 7fa9774fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:28:04.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:04 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1317888622' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T06:28:04.898 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:04 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:04.626+0000 7fa9774fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:28:04.933 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:04.933 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:04.933 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:28:04.933 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T06:28:04.933 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T06:28:04.934 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T06:28:04.935 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T06:28:04.943 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T06:28:04.943 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T06:28:04.943 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T06:28:04.943 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T06:28:04.943 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:04.944 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:04.945 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T06:27:47.344229+0000", 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.131+0000 7f9d41400700 1 Processor -- start 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.131+0000 7f9d41400700 1 -- start start 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.132+0000 7f9d41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.132+0000 7f9d41400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3c10bf20 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.132+0000 7f9d3a800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.132+0000 7f9d3a800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43072/0 (socket says 172.21.15.22:43072) 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.132+0000 7f9d3a800700 1 -- 172.21.15.22:0/1706864073 learned_addr learned my addr 172.21.15.22:0/1706864073 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.133+0000 7f9d3a800700 1 -- 172.21.15.22:0/1706864073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3c109d30 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.946 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.133+0000 7f9d3a800700 1 --2- 172.21.15.22:0/1706864073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0098a0 tx=0x7f9d2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85967d829ffaf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.133+0000 7f9d39400700 1 -- 172.21.15.22:0/1706864073 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d2c004160 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.133+0000 7f9d39400700 1 -- 172.21.15.22:0/1706864073 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9d2c0042c0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.133+0000 7f9d39400700 1 -- 172.21.15.22:0/1706864073 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d2c011ac0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 -- 172.21.15.22:0/1706864073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 msgr2=0x7f9d3c10b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 --2- 172.21.15.22:0/1706864073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c0098a0 tx=0x7f9d2c009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 -- 172.21.15.22:0/1706864073 shutdown_connections 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 --2- 172.21.15.22:0/1706864073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c10b9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 -- 172.21.15.22:0/1706864073 >> 172.21.15.22:0/1706864073 conn(0x7f9d3c07d360 msgr2=0x7f9d3c07d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 -- 172.21.15.22:0/1706864073 shutdown_connections 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.134+0000 7f9d41400700 1 -- 172.21.15.22:0/1706864073 wait complete. 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.135+0000 7f9d41400700 1 Processor -- start 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d41400700 1 -- start start 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d41400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d41400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3c19e290 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d3a800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:04.947 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d3a800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43082/0 (socket says 172.21.15.22:43082) 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.136+0000 7f9d3a800700 1 -- 172.21.15.22:0/1317888622 learned_addr learned my addr 172.21.15.22:0/1317888622 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d3a800700 1 -- 172.21.15.22:0/1317888622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2c009550 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d3a800700 1 --2- 172.21.15.22:0/1317888622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009130 tx=0x7f9d2c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d2c003aa0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9d2c003c00 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d2c024800 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d3c19e490 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.137+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d3c19e930 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.138+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9d2c02d3f0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.138+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9d2c019760 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.138+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d3c063830 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.143+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9d2c024e00 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.948 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.222+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9d3c1a13f0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.223+0000 7f9d33400700 1 -- 172.21.15.22:0/1317888622 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f9d2c01abd0 con 0x7f9d3c10b5c0 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 msgr2=0x7f9d3c19dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 --2- 172.21.15.22:0/1317888622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009130 tx=0x7f9d2c011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 shutdown_connections 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 --2- 172.21.15.22:0/1317888622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9d3c10b5c0 0x7f9d3c19dd50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 >> 172.21.15.22:0/1317888622 conn(0x7f9d3c07d360 msgr2=0x7f9d3c107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 shutdown_connections 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:04.224+0000 7f9d41400700 1 -- 172.21.15.22:0/1317888622 wait complete. 2024-03-24T06:28:04.949 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (3/15)... 2024-03-24T06:28:05.649 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:05.346+0000 7fa9774fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:28:05.649 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:05.478+0000 7fa9774fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:28:06.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:05.768+0000 7fa9774fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:28:06.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: Activating manager daemon x 2024-03-24T06:28:06.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: mgrmap e2: x(active, starting, since 0.0127314s) 2024-03-24T06:28:06.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:28:06.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:28:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:28:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:28:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:28:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: Manager daemon x is now available 2024-03-24T06:28:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:05 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:28:07.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:06 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' 2024-03-24T06:28:07.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:06 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:28:07.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:06 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' 2024-03-24T06:28:07.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:06 smithi022 ceph-mon[32003]: from='mgr.14100 172.21.15.22:0/26331659' entity='mgr.x' 2024-03-24T06:28:07.095 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:06 smithi022 ceph-mon[32003]: mgrmap e3: x(active, since 1.02179s) 2024-03-24T06:28:08.850 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:08.850 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:08.850 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T06:28:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T06:28:08.852 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T06:28:08.862 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T06:28:08.863 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T06:27:47.344229+0000", 2024-03-24T06:28:08.864 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.554+0000 7f345ac00700 1 Processor -- start 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.554+0000 7f345ac00700 1 -- start start 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.555+0000 7f345ac00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.555+0000 7f345ac00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f345410bf00 con 0x7f345410b510 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.555+0000 7f3453e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.555+0000 7f3453e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40570/0 (socket says 172.21.15.22:40570) 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.555+0000 7f3453e00700 1 -- 172.21.15.22:0/1862425674 learned_addr learned my addr 172.21.15.22:0/1862425674 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.556+0000 7f3453e00700 1 -- 172.21.15.22:0/1862425674 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3454109d30 con 0x7f345410b510 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.556+0000 7f3453e00700 1 --2- 172.21.15.22:0/1862425674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34440098a0 tx=0x7f3444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4585dda91920dbc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.556+0000 7f3452a00700 1 -- 172.21.15.22:0/1862425674 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3444004160 con 0x7f345410b510 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.556+0000 7f3452a00700 1 -- 172.21.15.22:0/1862425674 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f34440042c0 con 0x7f345410b510 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.556+0000 7f3452a00700 1 -- 172.21.15.22:0/1862425674 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3444011ac0 con 0x7f345410b510 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 -- 172.21.15.22:0/1862425674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 msgr2=0x7f345410b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 --2- 172.21.15.22:0/1862425674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34440098a0 tx=0x7f3444009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 -- 172.21.15.22:0/1862425674 shutdown_connections 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 --2- 172.21.15.22:0/1862425674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345410b930 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 -- 172.21.15.22:0/1862425674 >> 172.21.15.22:0/1862425674 conn(0x7f345407d2b0 msgr2=0x7f345407d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 -- 172.21.15.22:0/1862425674 shutdown_connections 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.557+0000 7f345ac00700 1 -- 172.21.15.22:0/1862425674 wait complete. 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.558+0000 7f345ac00700 1 Processor -- start 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f345ac00700 1 -- start start 2024-03-24T06:28:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f345ac00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f345ac00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f345419e330 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f3453e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f3453e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40582/0 (socket says 172.21.15.22:40582) 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f3453e00700 1 -- 172.21.15.22:0/3628624681 learned_addr learned my addr 172.21.15.22:0/3628624681 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.559+0000 7f3453e00700 1 -- 172.21.15.22:0/3628624681 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3444009550 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f3453e00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3444000c00 tx=0x7f3444011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3444003aa0 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3444003c00 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345419e530 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34440248a0 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.560+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f345419e9d0 con 0x7f345410b510 2024-03-24T06:28:08.867 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.561+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3454063830 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.561+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 3) ==== 45811+0+0 (secure 0 0 0) 0x7f344402d3f0 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.562+0000 7f3450c00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 0x7f343803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.562+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f344404cbf0 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.562+0000 7f3453400700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 0x7f343803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.563+0000 7f3453400700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 0x7f343803b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3440006fd0 tx=0x7f3440006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.568+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3444018d40 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.836+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f3444019410 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.889+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f345410e340 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.890+0000 7f3450c00700 1 -- 172.21.15.22:0/3628624681 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f34440186c0 con 0x7f345410b510 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 msgr2=0x7f343803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:08.868 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 0x7f343803b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3440006fd0 tx=0x7f3440006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 msgr2=0x7f345419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3444000c00 tx=0x7f3444011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 shutdown_connections 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3438038dc0 0x7f343803b280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 --2- 172.21.15.22:0/3628624681 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f345410b510 0x7f345419ddf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 >> 172.21.15.22:0/3628624681 conn(0x7f345407d2b0 msgr2=0x7f3454107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 shutdown_connections 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:07.893+0000 7f345ac00700 1 -- 172.21.15.22:0/3628624681 wait complete. 2024-03-24T06:28:08.869 INFO:teuthology.orchestra.run.smithi022.stdout:mgr is available 2024-03-24T06:28:09.120 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:08 smithi022 ceph-mon[32003]: mgrmap e4: x(active, since 2s) 2024-03-24T06:28:09.121 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3628624681' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T06:28:09.985 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:09 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3231994274' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T06:28:10.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:10.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T06:28:10.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout fsid = 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T06:28:10.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T06:28:10.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T06:28:10.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-24T06:28:10.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T06:28:10.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T06:28:10.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T06:28:10.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T06:28:10.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T06:28:10.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.391+0000 7f3dd4000700 1 Processor -- start 2024-03-24T06:28:10.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.391+0000 7f3dd4000700 1 -- start start 2024-03-24T06:28:10.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.392+0000 7f3dd4000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:10.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.392+0000 7f3dd4000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc108cd0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.392+0000 7f3dd1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.392+0000 7f3dd1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40588/0 (socket says 172.21.15.22:40588) 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.392+0000 7f3dd1600700 1 -- 172.21.15.22:0/1527118382 learned_addr learned my addr 172.21.15.22:0/1527118382 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.393+0000 7f3dd1600700 1 -- 172.21.15.22:0/1527118382 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc106820 con 0x7f3dcc1082e0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.393+0000 7f3dd1600700 1 --2- 172.21.15.22:0/1527118382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00098a0 tx=0x7f3dc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda17c5c3354bc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dcbe00700 1 -- 172.21.15.22:0/1527118382 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc0004160 con 0x7f3dcc1082e0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dcbe00700 1 -- 172.21.15.22:0/1527118382 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3dc00042c0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dcbe00700 1 -- 172.21.15.22:0/1527118382 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc0011ac0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dd4000700 1 -- 172.21.15.22:0/1527118382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 msgr2=0x7f3dcc108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dd4000700 1 --2- 172.21.15.22:0/1527118382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00098a0 tx=0x7f3dc0009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:10.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dd4000700 1 -- 172.21.15.22:0/1527118382 shutdown_connections 2024-03-24T06:28:10.340 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dd4000700 1 --2- 172.21.15.22:0/1527118382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc108700 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.394+0000 7f3dd4000700 1 -- 172.21.15.22:0/1527118382 >> 172.21.15.22:0/1527118382 conn(0x7f3dcc102180 msgr2=0x7f3dcc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.395+0000 7f3dd4000700 1 -- 172.21.15.22:0/1527118382 shutdown_connections 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.395+0000 7f3dd4000700 1 -- 172.21.15.22:0/1527118382 wait complete. 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.396+0000 7f3dd4000700 1 Processor -- start 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.396+0000 7f3dd4000700 1 -- start start 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.396+0000 7f3dd4000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.396+0000 7f3dd4000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3dcc1923b0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.341 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.397+0000 7f3dd1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.397+0000 7f3dd1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40592/0 (socket says 172.21.15.22:40592) 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.397+0000 7f3dd1600700 1 -- 172.21.15.22:0/3231994274 learned_addr learned my addr 172.21.15.22:0/3231994274 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.397+0000 7f3dd1600700 1 -- 172.21.15.22:0/3231994274 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc0009550 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.397+0000 7f3dd1600700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.398+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc0003aa0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.398+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3dc0003c00 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.398+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc1905a0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.398+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc00248a0 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.398+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dcc190a40 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.399+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f3dc001ac60 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.399+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dcc063830 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.399+0000 7f3dca000700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 0x7f3db403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.399+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3dc004cc90 con 0x7f3dcc1082e0 2024-03-24T06:28:10.342 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.400+0000 7f3dd0c00700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 0x7f3db403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.400+0000 7f3dd0c00700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 0x7f3db403b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006fd0 tx=0x7f3db8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.405+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3dc0018530 con 0x7f3dcc1082e0 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.625+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f3dcc10b000 con 0x7f3dcc1082e0 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.627+0000 7f3dca000700 1 -- 172.21.15.22:0/3231994274 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f3dc0018350 con 0x7f3dcc1082e0 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 msgr2=0x7f3db403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 0x7f3db403b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3db8006fd0 tx=0x7f3db8006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 msgr2=0x7f3dcc191e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc0011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 shutdown_connections 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f3db4038ee0 0x7f3db403b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 --2- 172.21.15.22:0/3231994274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3dcc1082e0 0x7f3dcc191e70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.630+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 >> 172.21.15.22:0/3231994274 conn(0x7f3dcc102180 msgr2=0x7f3dcc103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.631+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 shutdown_connections 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:09.631+0000 7f3dd4000700 1 -- 172.21.15.22:0/3231994274 wait complete. 2024-03-24T06:28:10.343 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling cephadm module... 2024-03-24T06:28:11.623 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:11 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/371377818' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T06:28:11.623 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:11 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setuser ceph since I am not root 2024-03-24T06:28:11.623 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:11 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setgroup ceph since I am not root 2024-03-24T06:28:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.965+0000 7f1582800700 1 Processor -- start 2024-03-24T06:28:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.965+0000 7f1582800700 1 -- start start 2024-03-24T06:28:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.965+0000 7f1582800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:11.955 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.966+0000 7f1582800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f157c108cc0 con 0x7f157c1082d0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.966+0000 7f157b400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.966+0000 7f157b400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40602/0 (socket says 172.21.15.22:40602) 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.966+0000 7f157b400700 1 -- 172.21.15.22:0/448000600 learned_addr learned my addr 172.21.15.22:0/448000600 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.967+0000 7f157b400700 1 -- 172.21.15.22:0/448000600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c106810 con 0x7f157c1082d0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.967+0000 7f157b400700 1 --2- 172.21.15.22:0/448000600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f156c0098a0 tx=0x7f156c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94804cca949b9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.967+0000 7f157aa00700 1 -- 172.21.15.22:0/448000600 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f156c004160 con 0x7f157c1082d0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.967+0000 7f157aa00700 1 -- 172.21.15.22:0/448000600 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f156c0042c0 con 0x7f157c1082d0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.967+0000 7f157aa00700 1 -- 172.21.15.22:0/448000600 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f156c011ac0 con 0x7f157c1082d0 2024-03-24T06:28:11.956 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 -- 172.21.15.22:0/448000600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 msgr2=0x7f157c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 --2- 172.21.15.22:0/448000600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f156c0098a0 tx=0x7f156c009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 -- 172.21.15.22:0/448000600 shutdown_connections 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 --2- 172.21.15.22:0/448000600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1086f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 -- 172.21.15.22:0/448000600 >> 172.21.15.22:0/448000600 conn(0x7f157c1021b0 msgr2=0x7f157c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 -- 172.21.15.22:0/448000600 shutdown_connections 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.968+0000 7f1582800700 1 -- 172.21.15.22:0/448000600 wait complete. 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.969+0000 7f1582800700 1 Processor -- start 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f1582800700 1 -- start start 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f1582800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f1582800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f157c199ee0 con 0x7f157c1082d0 2024-03-24T06:28:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f157b400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f157b400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40610/0 (socket says 172.21.15.22:40610) 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.970+0000 7f157b400700 1 -- 172.21.15.22:0/371377818 learned_addr learned my addr 172.21.15.22:0/371377818 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.971+0000 7f157b400700 1 -- 172.21.15.22:0/371377818 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156c009550 con 0x7f157c1082d0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.971+0000 7f157b400700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f156c000c00 tx=0x7f156c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.971+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f156c003aa0 con 0x7f157c1082d0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.971+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f156c003c00 con 0x7f157c1082d0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.972+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f156c0248a0 con 0x7f157c1082d0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.972+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157c19a0e0 con 0x7f157c1082d0 2024-03-24T06:28:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.972+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157c19a580 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.973+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157c07b820 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.973+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f156c01ac60 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.973+0000 7f1578c00700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 0x7f156003d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.973+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f156c04cc60 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.974+0000 7f1572a00700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 0x7f156003d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.974+0000 7f1572a00700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 0x7f156003d600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1564006fd0 tx=0x7f1564006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:10.979+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f156c018530 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.267+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f157c193070 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.321+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f156c018350 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.325+0000 7f1578c00700 1 -- 172.21.15.22:0/371377818 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f156c02d770 con 0x7f157c1082d0 2024-03-24T06:28:11.959 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.325+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 msgr2=0x7f156003d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.325+0000 7f1582800700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 0x7f156003d600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1564006fd0 tx=0x7f1564006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.325+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 msgr2=0x7f157c1999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.325+0000 7f1582800700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f156c000c00 tx=0x7f156c011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.326+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 shutdown_connections 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.326+0000 7f1582800700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f156003b140 0x7f156003d600 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.326+0000 7f1582800700 1 --2- 172.21.15.22:0/371377818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f157c1082d0 0x7f157c1999a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.326+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 >> 172.21.15.22:0/371377818 conn(0x7f157c1021b0 msgr2=0x7f157c190130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.327+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 shutdown_connections 2024-03-24T06:28:11.960 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:11.327+0000 7f1582800700 1 -- 172.21.15.22:0/371377818 wait complete. 2024-03-24T06:28:12.279 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:12 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:12.146+0000 7f4d83b56200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:28:12.279 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:12 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:12.279+0000 7f4d83b56200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:28:12.548 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/371377818' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T06:28:12.548 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:12 smithi022 ceph-mon[32003]: mgrmap e5: x(active, since 5s) 2024-03-24T06:28:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/268316851' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T06:28:13.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:13.098+0000 7f4d83b56200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:28:13.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:13.228+0000 7f4d83b56200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:28:13.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:13.615 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.603+0000 7fea07400700 1 Processor -- start 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea07400700 1 -- start start 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea07400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea07400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fea00109dc0 con 0x7fea001093d0 2024-03-24T06:28:13.616 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea04a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea04a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40630/0 (socket says 172.21.15.22:40630) 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.604+0000 7fea04a00700 1 -- 172.21.15.22:0/4189734665 learned_addr learned my addr 172.21.15.22:0/4189734665 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.605+0000 7fea04a00700 1 -- 172.21.15.22:0/4189734665 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea00107910 con 0x7fea001093d0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.605+0000 7fea04a00700 1 --2- 172.21.15.22:0/4189734665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40098a0 tx=0x7fe9f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2820b26c54c183ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.605+0000 7fe9ff400700 1 -- 172.21.15.22:0/4189734665 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9f4004160 con 0x7fea001093d0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fe9ff400700 1 -- 172.21.15.22:0/4189734665 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe9f40042c0 con 0x7fea001093d0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fe9ff400700 1 -- 172.21.15.22:0/4189734665 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9f4011ac0 con 0x7fea001093d0 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fea07400700 1 -- 172.21.15.22:0/4189734665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 msgr2=0x7fea001097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fea07400700 1 --2- 172.21.15.22:0/4189734665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40098a0 tx=0x7fe9f4009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fea07400700 1 -- 172.21.15.22:0/4189734665 shutdown_connections 2024-03-24T06:28:13.617 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fea07400700 1 --2- 172.21.15.22:0/4189734665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea001097f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.606+0000 7fea07400700 1 -- 172.21.15.22:0/4189734665 >> 172.21.15.22:0/4189734665 conn(0x7fea001032b0 msgr2=0x7fea001056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.607+0000 7fea07400700 1 -- 172.21.15.22:0/4189734665 shutdown_connections 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.607+0000 7fea07400700 1 -- 172.21.15.22:0/4189734665 wait complete. 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea07400700 1 Processor -- start 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea07400700 1 -- start start 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea07400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea07400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fea0007e900 con 0x7fea001093d0 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea04a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea04a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40632/0 (socket says 172.21.15.22:40632) 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.608+0000 7fea04a00700 1 -- 172.21.15.22:0/268316851 learned_addr learned my addr 172.21.15.22:0/268316851 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:13.618 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.609+0000 7fea04a00700 1 -- 172.21.15.22:0/268316851 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f4009550 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.609+0000 7fea04a00700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4000c00 tx=0x7fe9f4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.609+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9f4003aa0 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.610+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe9f4003c00 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.610+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0007caf0 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.610+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9f40248a0 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.610+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea0007cf90 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7fe9f401ac60 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea0010c120 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fe9fd600700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 0x7fe9e803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe9f404cd70 con 0x7fea001093d0 2024-03-24T06:28:13.619 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fe9ffe00700 1 -- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 msgr2=0x7fe9e803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.611+0000 7fe9ffe00700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 0x7fe9e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.616+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe9f4018530 con 0x7fea001093d0 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.812+0000 7fe9ffe00700 1 -- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 msgr2=0x7fe9e803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.812+0000 7fe9ffe00700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 0x7fe9e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.896+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fea0007d4c0 con 0x7fea001093d0 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.897+0000 7fe9fd600700 1 -- 172.21.15.22:0/268316851 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fe9f4018350 con 0x7fea001093d0 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 msgr2=0x7fe9e803b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 0x7fe9e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 msgr2=0x7fea0007e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4000c00 tx=0x7fe9f4011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:13.620 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 shutdown_connections 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7fe9e8038ee0 0x7fe9e803b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 --2- 172.21.15.22:0/268316851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fea001093d0 0x7fea0007e3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 >> 172.21.15.22:0/268316851 conn(0x7fea001032b0 msgr2=0x7fea00103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 shutdown_connections 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:12.900+0000 7fea07400700 1 -- 172.21.15.22:0/268316851 wait complete. 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for the mgr to restart... 2024-03-24T06:28:13.621 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr epoch 5... 2024-03-24T06:28:13.765 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:13.411+0000 7f4d83b56200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:28:13.765 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:13.541+0000 7f4d83b56200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:28:13.765 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:13.656+0000 7f4d83b56200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:28:14.030 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:14.011+0000 7f4d83b56200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:28:14.398 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:14.159+0000 7f4d83b56200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:28:14.398 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:14.279+0000 7f4d83b56200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:28:14.648 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:14.408+0000 7f4d83b56200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:28:14.649 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:14.632+0000 7f4d83b56200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:28:16.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:15 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:15.939+0000 7f4d83b56200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:28:16.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:16 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:16.116+0000 7f4d83b56200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:28:16.898 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:16 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:16.542+0000 7f4d83b56200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:28:16.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:16 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:16.663+0000 7f4d83b56200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:28:17.201 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:16 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:16.928+0000 7f4d83b56200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:17.201 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:17 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:17.177+0000 7f4d83b56200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:28:17.885 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:17 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:17.561+0000 7f4d83b56200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:17.885 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:17 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:17.884+0000 7f4d83b56200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:28:18.648 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:18.197+0000 7f4d83b56200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:28:19.649 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:19 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:19.256+0000 7f4d83b56200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:28:20.392 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:19 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:19.969+0000 7f4d83b56200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:28:20.392 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:20.101+0000 7f4d83b56200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:20.391+0000 7f4d83b56200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: Active manager daemon x restarted 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: Activating manager daemon x 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: mgrmap e6: x(active, starting, since 0.00513746s) 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:28:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:28:20.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:28:20.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:28:20.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:28:20.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: Manager daemon x is now available 2024-03-24T06:28:20.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:20 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: Found migration_current of "None". Setting to last migration. 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: mgrmap e7: x(active, since 1.01061s) 2024-03-24T06:28:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T06:28:21.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:21 smithi022 ceph-mon[32003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T06:28:22.240 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:22.240 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T06:28:22.240 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T06:28:22.240 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.188+0000 7f51f1400700 1 Processor -- start 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.188+0000 7f51f1400700 1 -- start start 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.188+0000 7f51f1400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.188+0000 7f51f1400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec109dd0 con 0x7f51ec1093e0 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.188+0000 7f51ea800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.189+0000 7f51ea800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40642/0 (socket says 172.21.15.22:40642) 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.189+0000 7f51ea800700 1 -- 172.21.15.22:0/3658588708 learned_addr learned my addr 172.21.15.22:0/3658588708 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.189+0000 7f51ea800700 1 -- 172.21.15.22:0/3658588708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec107920 con 0x7f51ec1093e0 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.189+0000 7f51ea800700 1 --2- 172.21.15.22:0/3658588708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098a0 tx=0x7f51dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605609f15ff6ddb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.190+0000 7f51e9400700 1 -- 172.21.15.22:0/3658588708 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc004160 con 0x7f51ec1093e0 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.190+0000 7f51e9400700 1 -- 172.21.15.22:0/3658588708 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f51dc0042c0 con 0x7f51ec1093e0 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.190+0000 7f51e9400700 1 -- 172.21.15.22:0/3658588708 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc011ac0 con 0x7f51ec1093e0 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 -- 172.21.15.22:0/3658588708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 msgr2=0x7f51ec109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 --2- 172.21.15.22:0/3658588708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0098a0 tx=0x7f51dc009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 -- 172.21.15.22:0/3658588708 shutdown_connections 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 --2- 172.21.15.22:0/3658588708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec109800 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 -- 172.21.15.22:0/3658588708 >> 172.21.15.22:0/3658588708 conn(0x7f51ec1032a0 msgr2=0x7f51ec1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 -- 172.21.15.22:0/3658588708 shutdown_connections 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.191+0000 7f51f1400700 1 -- 172.21.15.22:0/3658588708 wait complete. 2024-03-24T06:28:22.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.192+0000 7f51f1400700 1 Processor -- start 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.192+0000 7f51f1400700 1 -- start start 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51f1400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51f1400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec07e070 con 0x7f51ec1093e0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51ea800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51ea800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40644/0 (socket says 172.21.15.22:40644) 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51ea800700 1 -- 172.21.15.22:0/989657270 learned_addr learned my addr 172.21.15.22:0/989657270 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.193+0000 7f51ea800700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc009550 con 0x7f51ec1093e0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51ea800700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51dc003f70 tx=0x7f51dc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc0039f0 con 0x7f51ec1093e0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f51dc003b50 con 0x7f51ec1093e0 2024-03-24T06:28:22.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51dc01a690 con 0x7f51ec1093e0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec07c260 con 0x7f51ec1093e0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.194+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec07c700 con 0x7f51ec1093e0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.195+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f51dc024430 con 0x7f51ec1093e0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.196+0000 7f51e3400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.196+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f51d003bab0 con 0x7f51d0038ee0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.196+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f51dc04d180 con 0x7f51ec1093e0 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.196+0000 7f51e9e00700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.196+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.397+0000 7f51e9e00700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.397+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:28:22.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.798+0000 7f51e9e00700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:14.798+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:15.599+0000 7f51e9e00700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:15.599+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:17.201+0000 7f51e9e00700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/37818756 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:17.201+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:20.399+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mgrmap(e 6) ==== 45695+0+0 (secure 0 0 0) 0x7f51dc02d910 con 0x7f51ec1093e0 2024-03-24T06:28:22.245 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:20.400+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 msgr2=0x7f51d003b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:20.400+0000 7f51e3400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.404+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7f51dc04e2e0 con 0x7f51ec1093e0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.405+0000 7f51e3400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 0x7f51d003ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.405+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f51d003bab0 con 0x7f51d003c7d0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.407+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 0x7f51d003ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.409+0000 7f51e9e00700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 0x7f51d003ebc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003a10 tx=0x7f51d80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.410+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f51d003bab0 con 0x7f51d003c7d0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f51ec063830 con 0x7f51d003c7d0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51e3400700 1 -- 172.21.15.22:0/989657270 <== mgr.14120 v2:172.21.15.22:6800/2009113619 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f51ec063830 con 0x7f51d003c7d0 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 msgr2=0x7f51d003ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:22.246 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51f1400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 0x7f51d003ebc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003a10 tx=0x7f51d80092b0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 msgr2=0x7f51ec07db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.418+0000 7f51f1400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f51dc003f70 tx=0x7f51dc011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 shutdown_connections 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f51d003c7d0 0x7f51d003ebc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f51ec1093e0 0x7f51ec07db30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 --2- 172.21.15.22:0/989657270 >> [v2:172.21.15.22:6800/37818756,v1:172.21.15.22:6801/37818756] conn(0x7f51d0038ee0 0x7f51d003b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 >> 172.21.15.22:0/989657270 conn(0x7f51ec1032a0 msgr2=0x7f51ec103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 shutdown_connections 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:21.419+0000 7f51f1400700 1 -- 172.21.15.22:0/989657270 wait complete. 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:mgr epoch 5 is available 2024-03-24T06:28:22.247 INFO:teuthology.orchestra.run.smithi022.stdout:Setting orchestrator backend to cephadm... 2024-03-24T06:28:22.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:22 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:22.793 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:22 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:23.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: mgrmap e8: x(active, since 2s) 2024-03-24T06:28:23.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:22] ENGINE Bus STARTING 2024-03-24T06:28:23.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:22] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T06:28:23.790 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:22] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T06:28:23.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:22] ENGINE Bus STARTED 2024-03-24T06:28:23.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:23.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:23.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:23.791 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:23 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:23.804 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.828+0000 7f6f67000700 1 Processor -- start 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.829+0000 7f6f67000700 1 -- start start 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.829+0000 7f6f67000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.829+0000 7f6f67000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6010be60 con 0x7f6f6010b500 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.830+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.830+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:49978/0 (socket says 172.21.15.22:49978) 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.830+0000 7f6f5fe00700 1 -- 172.21.15.22:0/3782082843 learned_addr learned my addr 172.21.15.22:0/3782082843 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.830+0000 7f6f5fe00700 1 -- 172.21.15.22:0/3782082843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f60109d30 con 0x7f6f6010b500 2024-03-24T06:28:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.830+0000 7f6f5fe00700 1 --2- 172.21.15.22:0/3782082843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f500098a0 tx=0x7f6f50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c66e7cd7e5ddabf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f5f400700 1 -- 172.21.15.22:0/3782082843 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f50004160 con 0x7f6f6010b500 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f5f400700 1 -- 172.21.15.22:0/3782082843 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f500042c0 con 0x7f6f6010b500 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f5f400700 1 -- 172.21.15.22:0/3782082843 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f50011ac0 con 0x7f6f6010b500 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f67000700 1 -- 172.21.15.22:0/3782082843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 msgr2=0x7f6f6010b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f67000700 1 --2- 172.21.15.22:0/3782082843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f500098a0 tx=0x7f6f50009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f67000700 1 -- 172.21.15.22:0/3782082843 shutdown_connections 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.831+0000 7f6f67000700 1 --2- 172.21.15.22:0/3782082843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6010b920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.832+0000 7f6f67000700 1 -- 172.21.15.22:0/3782082843 >> 172.21.15.22:0/3782082843 conn(0x7f6f6007d2e0 msgr2=0x7f6f6007d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:23.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.832+0000 7f6f67000700 1 -- 172.21.15.22:0/3782082843 shutdown_connections 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.832+0000 7f6f67000700 1 -- 172.21.15.22:0/3782082843 wait complete. 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.833+0000 7f6f67000700 1 Processor -- start 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.833+0000 7f6f67000700 1 -- start start 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.833+0000 7f6f67000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.833+0000 7f6f67000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6019e310 con 0x7f6f6010b500 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.834+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.834+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:49990/0 (socket says 172.21.15.22:49990) 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.834+0000 7f6f5fe00700 1 -- 172.21.15.22:0/3954942742 learned_addr learned my addr 172.21.15.22:0/3954942742 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.834+0000 7f6f5fe00700 1 -- 172.21.15.22:0/3954942742 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f50009550 con 0x7f6f6010b500 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.834+0000 7f6f5fe00700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6f50000c00 tx=0x7f6f50011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.835+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f50003aa0 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.835+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6f50003c00 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.835+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6019e510 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.835+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f500248a0 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.835+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6019e9b0 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.837+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f6f5001ac60 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.837+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f6010e380 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.837+0000 7f6f5d600700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 0x7f6f4003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.837+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6f5004d120 con 0x7f6f6010b500 2024-03-24T06:28:23.808 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.837+0000 7f6f57400700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 0x7f6f4003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.838+0000 7f6f57400700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 0x7f6f4003b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f48006fd0 tx=0x7f6f48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:22.843+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6f50018530 con 0x7f6f6010b500 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.088+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6f60108490 con 0x7f6f40038f30 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.099+0000 7f6f5d600700 1 -- 172.21.15.22:0/3954942742 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6f60108490 con 0x7f6f40038f30 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.102+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 msgr2=0x7f6f4003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.102+0000 7f6f67000700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 0x7f6f4003b3f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f48006fd0 tx=0x7f6f48006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.102+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 msgr2=0x7f6f6019ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.102+0000 7f6f67000700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6f50000c00 tx=0x7f6f50011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 shutdown_connections 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f6f40038f30 0x7f6f4003b3f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:23.809 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 --2- 172.21.15.22:0/3954942742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f6010b500 0x7f6f6019ddd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:23.810 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 >> 172.21.15.22:0/3954942742 conn(0x7f6f6007d2e0 msgr2=0x7f6f60107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:23.810 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 shutdown_connections 2024-03-24T06:28:23.810 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:23.103+0000 7f6f67000700 1 -- 172.21.15.22:0/3954942742 wait complete. 2024-03-24T06:28:25.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.396+0000 7fbdfc800700 1 Processor -- start 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.397+0000 7fbdfc800700 1 -- start start 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.397+0000 7fbdfc800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.397+0000 7fbdfc800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf410b090 con 0x7fbdf4108760 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.397+0000 7fbdf9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.398+0000 7fbdf9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50006/0 (socket says 172.21.15.22:50006) 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.398+0000 7fbdf9e00700 1 -- 172.21.15.22:0/7311806 learned_addr learned my addr 172.21.15.22:0/7311806 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.398+0000 7fbdf9e00700 1 -- 172.21.15.22:0/7311806 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf410b1d0 con 0x7fbdf4108760 2024-03-24T06:28:25.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.398+0000 7fbdf9e00700 1 --2- 172.21.15.22:0/7311806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ab0 tx=0x7fbde8009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272d544c853e2f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.399+0000 7fbdf8a00700 1 -- 172.21.15.22:0/7311806 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde8004160 con 0x7fbdf4108760 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.399+0000 7fbdf8a00700 1 -- 172.21.15.22:0/7311806 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbde800bca0 con 0x7fbdf4108760 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.399+0000 7fbdf8a00700 1 -- 172.21.15.22:0/7311806 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde8003c90 con 0x7fbdf4108760 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 -- 172.21.15.22:0/7311806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 msgr2=0x7fbdf410ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 --2- 172.21.15.22:0/7311806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009ab0 tx=0x7fbde8009dc0 comp rx=0 tx=0).stop 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 -- 172.21.15.22:0/7311806 shutdown_connections 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 --2- 172.21.15.22:0/7311806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf410ab50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 -- 172.21.15.22:0/7311806 >> 172.21.15.22:0/7311806 conn(0x7fbdf41021b0 msgr2=0x7fbdf41045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 -- 172.21.15.22:0/7311806 shutdown_connections 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.400+0000 7fbdfc800700 1 -- 172.21.15.22:0/7311806 wait complete. 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.401+0000 7fbdfc800700 1 Processor -- start 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.401+0000 7fbdfc800700 1 -- start start 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdfc800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:25.287 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdfc800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf419c090 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdf9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdf9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50010/0 (socket says 172.21.15.22:50010) 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdf9e00700 1 -- 172.21.15.22:0/4283855461 learned_addr learned my addr 172.21.15.22:0/4283855461 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.402+0000 7fbdf9e00700 1 -- 172.21.15.22:0/4283855461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde8009550 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdf9e00700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbde8006b20 tx=0x7fbde8003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde8003f00 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbde8004520 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde801a430 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf419c290 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.403+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf419c730 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.404+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf410bfd0 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.405+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fbde8024440 con 0x7fbdf4108760 2024-03-24T06:28:25.288 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.406+0000 7fbdeea00700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 0x7fbddc03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.406+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbde804d1c0 con 0x7fbdf4108760 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.406+0000 7fbdf9400700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 0x7fbddc03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.407+0000 7fbdf9400700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 0x7fbddc03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbde4006fd0 tx=0x7fbde4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.411+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbde8018350 con 0x7fbdf4108760 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.642+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fbdf4103430 con 0x7fbddc038f30 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.644+0000 7fbdeea00700 1 -- 172.21.15.22:0/4283855461 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fbdf4103430 con 0x7fbddc038f30 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.647+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 msgr2=0x7fbddc03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.647+0000 7fbdfc800700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 0x7fbddc03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbde4006fd0 tx=0x7fbde4006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.647+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 msgr2=0x7fbdf419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.647+0000 7fbdfc800700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbde8006b20 tx=0x7fbde8003e00 comp rx=0 tx=0).stop 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 shutdown_connections 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbddc038f30 0x7fbddc03b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:25.289 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 --2- 172.21.15.22:0/4283855461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdf4108760 0x7fbdf419bb50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:25.290 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 >> 172.21.15.22:0/4283855461 conn(0x7fbdf41021b0 msgr2=0x7fbdf4102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:25.290 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 shutdown_connections 2024-03-24T06:28:25.290 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:24.648+0000 7fbdfc800700 1 -- 172.21.15.22:0/4283855461 wait complete. 2024-03-24T06:28:25.290 INFO:teuthology.orchestra.run.smithi022.stdout:Generating ssh key... 2024-03-24T06:28:25.551 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:25 smithi022 ceph-mon[32003]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:26.650 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: Generating public/private rsa key pair. 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: Your identification has been saved in /tmp/tmp30x8yolx/key. 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: Your public key has been saved in /tmp/tmp30x8yolx/key.pub. 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: The key fingerprint is: 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: SHA256:+ifdn0G37Hyill4wtf6pxExItG5MEsS6fWGzwOr7cUY ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: The key's randomart image is: 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: +---[RSA 3072]----+ 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | oo . | 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | .o . | 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | o. + . | 2024-03-24T06:28:26.651 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | . o*+.. . | 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | S oE*.o .| 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | + .oo+=...| 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | o o.+ =+o | 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | o. * =.++o| 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: | .++ oo++=o| 2024-03-24T06:28:26.652 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:26 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: +----[SHA256]-----+ 2024-03-24T06:28:26.971 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.840+0000 7f2f16a00700 1 Processor -- start 2024-03-24T06:28:26.971 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.840+0000 7f2f16a00700 1 -- start start 2024-03-24T06:28:26.972 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.841+0000 7f2f16a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:26.972 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.841+0000 7f2f16a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1010bee0 con 0x7f2f1010b560 2024-03-24T06:28:26.972 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.841+0000 7f2f0fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:26.972 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.842+0000 7f2f0fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40714/0 (socket says 172.21.15.22:40714) 2024-03-24T06:28:26.972 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.842+0000 7f2f0fe00700 1 -- 172.21.15.22:0/2962205631 learned_addr learned my addr 172.21.15.22:0/2962205631 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.842+0000 7f2f0fe00700 1 -- 172.21.15.22:0/2962205631 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f10109b60 con 0x7f2f1010b560 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.842+0000 7f2f0fe00700 1 --2- 172.21.15.22:0/2962205631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7489d3b5b72f1465 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.843+0000 7f2f0ea00700 1 -- 172.21.15.22:0/2962205631 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00004160 con 0x7f2f1010b560 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.843+0000 7f2f0ea00700 1 -- 172.21.15.22:0/2962205631 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f000042c0 con 0x7f2f1010b560 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.843+0000 7f2f0ea00700 1 -- 172.21.15.22:0/2962205631 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00011ac0 con 0x7f2f1010b560 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.843+0000 7f2f16a00700 1 -- 172.21.15.22:0/2962205631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 msgr2=0x7f2f1010b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.843+0000 7f2f16a00700 1 --2- 172.21.15.22:0/2962205631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.844+0000 7f2f16a00700 1 -- 172.21.15.22:0/2962205631 shutdown_connections 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.844+0000 7f2f16a00700 1 --2- 172.21.15.22:0/2962205631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1010b9a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:26.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.844+0000 7f2f16a00700 1 -- 172.21.15.22:0/2962205631 >> 172.21.15.22:0/2962205631 conn(0x7f2f1007d300 msgr2=0x7f2f1007d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.844+0000 7f2f16a00700 1 -- 172.21.15.22:0/2962205631 shutdown_connections 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.844+0000 7f2f16a00700 1 -- 172.21.15.22:0/2962205631 wait complete. 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.845+0000 7f2f16a00700 1 Processor -- start 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.845+0000 7f2f16a00700 1 -- start start 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.845+0000 7f2f16a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.845+0000 7f2f16a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1019e340 con 0x7f2f1010b560 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.846+0000 7f2f0fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.846+0000 7f2f0fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40730/0 (socket says 172.21.15.22:40730) 2024-03-24T06:28:26.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.846+0000 7f2f0fe00700 1 -- 172.21.15.22:0/463642170 learned_addr learned my addr 172.21.15.22:0/463642170 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:26.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.846+0000 7f2f0fe00700 1 -- 172.21.15.22:0/463642170 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f00009550 con 0x7f2f1010b560 2024-03-24T06:28:26.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f0fe00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00003aa0 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2f00003c00 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1019e540 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f000248a0 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.847+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1019e9e0 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.848+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1010e390 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.848+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f2f0001ac60 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.849+0000 7f2f0cc00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 0x7f2ef403b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.849+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2f0004d120 con 0x7f2f1010b560 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.849+0000 7f2f0f400700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 0x7f2ef403b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:26.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.850+0000 7f2f0f400700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 0x7f2ef403b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2efc006fd0 tx=0x7f2efc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:25.854+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f00018530 con 0x7f2f1010b560 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.079+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2f10107bc0 con 0x7f2ef4038f30 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.284+0000 7f2f0cc00700 1 -- 172.21.15.22:0/463642170 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2f10107bc0 con 0x7f2ef4038f30 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 msgr2=0x7f2ef403b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 0x7f2ef403b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2efc006fd0 tx=0x7f2efc006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 msgr2=0x7f2f1019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 shutdown_connections 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2ef4038f30 0x7f2ef403b3f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 --2- 172.21.15.22:0/463642170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2f1010b560 0x7f2f1019de00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:26.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 >> 172.21.15.22:0/463642170 conn(0x7f2f1007d300 msgr2=0x7f2f101074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:26.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 shutdown_connections 2024-03-24T06:28:26.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:26.287+0000 7f2f16a00700 1 -- 172.21.15.22:0/463642170 wait complete. 2024-03-24T06:28:27.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:27 smithi022 ceph-mon[32003]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:27.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:27 smithi022 ceph-mon[32003]: Generating ssh key... 2024-03-24T06:28:27.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:27 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:27.487 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:27 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:28.436 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:28:28.436 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.571+0000 7fbe78200700 1 Processor -- start 2024-03-24T06:28:28.436 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.571+0000 7fbe78200700 1 -- start start 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.572+0000 7fbe78200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.572+0000 7fbe78200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7007e030 con 0x7fbe7007c300 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.572+0000 7fbe75800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.572+0000 7fbe75800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40738/0 (socket says 172.21.15.22:40738) 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.572+0000 7fbe75800700 1 -- 172.21.15.22:0/261832239 learned_addr learned my addr 172.21.15.22:0/261832239 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.573+0000 7fbe75800700 1 -- 172.21.15.22:0/261832239 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7007cc60 con 0x7fbe7007c300 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.573+0000 7fbe75800700 1 --2- 172.21.15.22:0/261832239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009ab0 tx=0x7fbe64009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddda82c6f2191694 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.573+0000 7fbe6fe00700 1 -- 172.21.15.22:0/261832239 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe64004160 con 0x7fbe7007c300 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.573+0000 7fbe6fe00700 1 -- 172.21.15.22:0/261832239 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe6400bca0 con 0x7fbe7007c300 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.573+0000 7fbe6fe00700 1 -- 172.21.15.22:0/261832239 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe64003c90 con 0x7fbe7007c300 2024-03-24T06:28:28.437 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 -- 172.21.15.22:0/261832239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 msgr2=0x7fbe7007c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 --2- 172.21.15.22:0/261832239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009ab0 tx=0x7fbe64009dc0 comp rx=0 tx=0).stop 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 -- 172.21.15.22:0/261832239 shutdown_connections 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 --2- 172.21.15.22:0/261832239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7007c720 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 -- 172.21.15.22:0/261832239 >> 172.21.15.22:0/261832239 conn(0x7fbe701032c0 msgr2=0x7fbe701056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 -- 172.21.15.22:0/261832239 shutdown_connections 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.574+0000 7fbe78200700 1 -- 172.21.15.22:0/261832239 wait complete. 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.575+0000 7fbe78200700 1 Processor -- start 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe78200700 1 -- start start 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe78200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:28.438 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe78200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7019e2e0 con 0x7fbe7007c300 2024-03-24T06:28:28.442 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe75800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:28.442 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe75800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40746/0 (socket says 172.21.15.22:40746) 2024-03-24T06:28:28.442 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe75800700 1 -- 172.21.15.22:0/1668616299 learned_addr learned my addr 172.21.15.22:0/1668616299 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:28.442 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.576+0000 7fbe75800700 1 -- 172.21.15.22:0/1668616299 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe64009550 con 0x7fbe7007c300 2024-03-24T06:28:28.442 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe75800700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe64006b20 tx=0x7fbe64003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe64003f00 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe64004520 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe6401a430 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7019e4e0 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.577+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7019e980 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.578+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fbe64024440 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.578+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe7010e210 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.579+0000 7fbe6e000700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 0x7fbe5803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.579+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbe6404cd80 con 0x7fbe7007c300 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.579+0000 7fbe74e00700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 0x7fbe5803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.580+0000 7fbe74e00700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 0x7fbe5803b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:28.443 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.584+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbe64018a40 con 0x7fbe7007c300 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.812+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fbe70104500 con 0x7fbe58038ee0 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.813+0000 7fbe6e000700 1 -- 172.21.15.22:0/1668616299 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fbe70104500 con 0x7fbe58038ee0 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 msgr2=0x7fbe5803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 0x7fbe5803b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 msgr2=0x7fbe7019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe64006b20 tx=0x7fbe64003e00 comp rx=0 tx=0).stop 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 shutdown_connections 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe58038ee0 0x7fbe5803b3a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 --2- 172.21.15.22:0/1668616299 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe7007c300 0x7fbe7019dda0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.816+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 >> 172.21.15.22:0/1668616299 conn(0x7fbe701032c0 msgr2=0x7fbe70103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.817+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 shutdown_connections 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:27.817+0000 7fbe78200700 1 -- 172.21.15.22:0/1668616299 wait complete. 2024-03-24T06:28:28.444 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T06:28:28.445 INFO:teuthology.orchestra.run.smithi022.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T06:28:28.445 INFO:teuthology.orchestra.run.smithi022.stdout:Adding host smithi022... 2024-03-24T06:28:28.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:28 smithi022 ceph-mon[32003]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:29.548 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:29 smithi022 ceph-mon[32003]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "addr": "172.21.15.22", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:31.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:31 smithi022 ceph-mon[32003]: Deploying cephadm binary to smithi022 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Added host 'smithi022' with addr '172.21.15.22' 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.039+0000 7fe3cc800700 1 Processor -- start 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.039+0000 7fe3cc800700 1 -- start start 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.040+0000 7fe3cc800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.040+0000 7fe3cc800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c410bf00 con 0x7fe3c410b510 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.040+0000 7fe3c9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.041+0000 7fe3c9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40762/0 (socket says 172.21.15.22:40762) 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.041+0000 7fe3c9e00700 1 -- 172.21.15.22:0/3606712683 learned_addr learned my addr 172.21.15.22:0/3606712683 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:36.386 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.041+0000 7fe3c9e00700 1 -- 172.21.15.22:0/3606712683 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c4109d30 con 0x7fe3c410b510 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.042+0000 7fe3c9e00700 1 --2- 172.21.15.22:0/3606712683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80098a0 tx=0x7fe3b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3de1a6da5bc71d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.042+0000 7fe3c8a00700 1 -- 172.21.15.22:0/3606712683 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3b8004160 con 0x7fe3c410b510 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.042+0000 7fe3c8a00700 1 -- 172.21.15.22:0/3606712683 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3b80042c0 con 0x7fe3c410b510 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.042+0000 7fe3c8a00700 1 -- 172.21.15.22:0/3606712683 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3b8011ac0 con 0x7fe3c410b510 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.043+0000 7fe3cc800700 1 -- 172.21.15.22:0/3606712683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 msgr2=0x7fe3c410b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.043+0000 7fe3cc800700 1 --2- 172.21.15.22:0/3606712683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80098a0 tx=0x7fe3b8009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.043+0000 7fe3cc800700 1 -- 172.21.15.22:0/3606712683 shutdown_connections 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.043+0000 7fe3cc800700 1 --2- 172.21.15.22:0/3606712683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c410b930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.043+0000 7fe3cc800700 1 -- 172.21.15.22:0/3606712683 >> 172.21.15.22:0/3606712683 conn(0x7fe3c407d2b0 msgr2=0x7fe3c407d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.044+0000 7fe3cc800700 1 -- 172.21.15.22:0/3606712683 shutdown_connections 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.044+0000 7fe3cc800700 1 -- 172.21.15.22:0/3606712683 wait complete. 2024-03-24T06:28:36.387 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.045+0000 7fe3cc800700 1 Processor -- start 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.045+0000 7fe3cc800700 1 -- start start 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.045+0000 7fe3cc800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.045+0000 7fe3cc800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c419e330 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.046+0000 7fe3c9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.046+0000 7fe3c9e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40774/0 (socket says 172.21.15.22:40774) 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.046+0000 7fe3c9e00700 1 -- 172.21.15.22:0/4038786365 learned_addr learned my addr 172.21.15.22:0/4038786365 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.046+0000 7fe3c9e00700 1 -- 172.21.15.22:0/4038786365 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b8009550 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.046+0000 7fe3c9e00700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8003f70 tx=0x7fe3b8011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.047+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3b80039f0 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.047+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3b8003b50 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.047+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3b801a690 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.047+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c419e530 con 0x7fe3c410b510 2024-03-24T06:28:36.388 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.047+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c419e9d0 con 0x7fe3c410b510 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.048+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c4063830 con 0x7fe3c410b510 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.048+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fe3b8024430 con 0x7fe3c410b510 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.049+0000 7fe3bea00700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 0x7fe3ac03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.049+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe3b804d2a0 con 0x7fe3c410b510 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.049+0000 7fe3c9400700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 0x7fe3ac03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.050+0000 7fe3c9400700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 0x7fe3ac03b350 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4006fd0 tx=0x7fe3b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.055+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe3b8018a40 con 0x7fe3c410b510 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:29.283+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi022", "addr": "172.21.15.22", "target": ["mon-mgr", ""]}) -- 0x7fe3c41a12b0 con 0x7fe3ac038e90 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.720+0000 7fe3bea00700 1 -- 172.21.15.22:0/4038786365 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fe3c41a12b0 con 0x7fe3ac038e90 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.723+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 msgr2=0x7fe3ac03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.723+0000 7fe3cc800700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 0x7fe3ac03b350 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4006fd0 tx=0x7fe3b4006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:36.389 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.723+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 msgr2=0x7fe3c419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.723+0000 7fe3cc800700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8003f70 tx=0x7fe3b8011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.723+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 shutdown_connections 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.724+0000 7fe3cc800700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fe3ac038e90 0x7fe3ac03b350 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.724+0000 7fe3cc800700 1 --2- 172.21.15.22:0/4038786365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c410b510 0x7fe3c419ddf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.724+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 >> 172.21.15.22:0/4038786365 conn(0x7fe3c407d2b0 msgr2=0x7fe3c4107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.724+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 shutdown_connections 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:35.724+0000 7fe3cc800700 1 -- 172.21.15.22:0/4038786365 wait complete. 2024-03-24T06:28:36.390 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying unmanaged mon service... 2024-03-24T06:28:37.061 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:36 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:37.061 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:36 smithi022 ceph-mon[32003]: Added host smithi022 2024-03-24T06:28:37.062 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:36 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:38 smithi022 ceph-mon[32003]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:38 smithi022 ceph-mon[32003]: Saving service mon spec with placement count:5 2024-03-24T06:28:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:38 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:39.128 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T06:28:39.128 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.074+0000 7fbe66400700 1 Processor -- start 2024-03-24T06:28:39.128 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.075+0000 7fbe66400700 1 -- start start 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.075+0000 7fbe66400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.075+0000 7fbe66400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6010bea0 con 0x7fbe6010b540 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.076+0000 7fbe5f400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.076+0000 7fbe5f400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36882/0 (socket says 172.21.15.22:36882) 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.076+0000 7fbe5f400700 1 -- 172.21.15.22:0/2012908654 learned_addr learned my addr 172.21.15.22:0/2012908654 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.077+0000 7fbe5f400700 1 -- 172.21.15.22:0/2012908654 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe60109d30 con 0x7fbe6010b540 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.077+0000 7fbe5f400700 1 --2- 172.21.15.22:0/2012908654 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098a0 tx=0x7fbe50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7908abca6cf4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.078+0000 7fbe5e000700 1 -- 172.21.15.22:0/2012908654 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe50004160 con 0x7fbe6010b540 2024-03-24T06:28:39.129 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.078+0000 7fbe5e000700 1 -- 172.21.15.22:0/2012908654 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe500042c0 con 0x7fbe6010b540 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.078+0000 7fbe5e000700 1 -- 172.21.15.22:0/2012908654 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe50011ac0 con 0x7fbe6010b540 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.078+0000 7fbe66400700 1 -- 172.21.15.22:0/2012908654 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 msgr2=0x7fbe6010b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.078+0000 7fbe66400700 1 --2- 172.21.15.22:0/2012908654 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098a0 tx=0x7fbe50009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.079+0000 7fbe66400700 1 -- 172.21.15.22:0/2012908654 shutdown_connections 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.079+0000 7fbe66400700 1 --2- 172.21.15.22:0/2012908654 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6010b960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.079+0000 7fbe66400700 1 -- 172.21.15.22:0/2012908654 >> 172.21.15.22:0/2012908654 conn(0x7fbe6007d370 msgr2=0x7fbe6007d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.079+0000 7fbe66400700 1 -- 172.21.15.22:0/2012908654 shutdown_connections 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.079+0000 7fbe66400700 1 -- 172.21.15.22:0/2012908654 wait complete. 2024-03-24T06:28:39.130 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.080+0000 7fbe66400700 1 Processor -- start 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.080+0000 7fbe66400700 1 -- start start 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe66400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe66400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6019e300 con 0x7fbe6010b540 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe5f400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe5f400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36896/0 (socket says 172.21.15.22:36896) 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe5f400700 1 -- 172.21.15.22:0/1644120791 learned_addr learned my addr 172.21.15.22:0/1644120791 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.081+0000 7fbe5f400700 1 -- 172.21.15.22:0/1644120791 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe50009550 con 0x7fbe6010b540 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe5f400700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe50003f70 tx=0x7fbe50011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe500039f0 con 0x7fbe6010b540 2024-03-24T06:28:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe50003b50 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6019e500 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe5001a760 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.082+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6019e9a0 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.083+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fbe50024430 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.083+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6010e320 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.084+0000 7fbe57e00700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 0x7fbe4403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.084+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbe5004d200 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.084+0000 7fbe5ea00700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 0x7fbe4403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.085+0000 7fbe5ea00700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 0x7fbe4403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c006fd0 tx=0x7fbe4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.090+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbe50018a40 con 0x7fbe6010b540 2024-03-24T06:28:39.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.345+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fbe60108420 con 0x7fbe44038ee0 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.353+0000 7fbe57e00700 1 -- 172.21.15.22:0/1644120791 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbe60108420 con 0x7fbe44038ee0 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 msgr2=0x7fbe4403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 0x7fbe4403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c006fd0 tx=0x7fbe4c006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 msgr2=0x7fbe6019ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbe50003f70 tx=0x7fbe50011b40 comp rx=0 tx=0).stop 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 shutdown_connections 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fbe44038ee0 0x7fbe4403b3a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 --2- 172.21.15.22:0/1644120791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe6010b540 0x7fbe6019ddc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.356+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 >> 172.21.15.22:0/1644120791 conn(0x7fbe6007d370 msgr2=0x7fbe60107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.357+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 shutdown_connections 2024-03-24T06:28:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:37.357+0000 7fbe66400700 1 -- 172.21.15.22:0/1644120791 wait complete. 2024-03-24T06:28:39.134 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying unmanaged mgr service... 2024-03-24T06:28:40.006 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:39 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.736+0000 7f4117200700 1 Processor -- start 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.736+0000 7f4117200700 1 -- start start 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.737+0000 7f4117200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.737+0000 7f4117200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f411807d160 con 0x7f411807e730 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.737+0000 7f4115e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:40.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.737+0000 7f4115e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36904/0 (socket says 172.21.15.22:36904) 2024-03-24T06:28:40.640 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.737+0000 7f4115e00700 1 -- 172.21.15.22:0/42993026 learned_addr learned my addr 172.21.15.22:0/42993026 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:40.640 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.738+0000 7f4115e00700 1 -- 172.21.15.22:0/42993026 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411807d2a0 con 0x7f411807e730 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.738+0000 7f4115e00700 1 --2- 172.21.15.22:0/42993026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41080098a0 tx=0x7f4108009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39df4054f874d63b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.738+0000 7f4114a00700 1 -- 172.21.15.22:0/42993026 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4108004160 con 0x7f411807e730 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.738+0000 7f4114a00700 1 -- 172.21.15.22:0/42993026 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41080042c0 con 0x7f411807e730 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.738+0000 7f4114a00700 1 -- 172.21.15.22:0/42993026 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4108011a80 con 0x7f411807e730 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.739+0000 7f4117200700 1 -- 172.21.15.22:0/42993026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 msgr2=0x7f411807cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.739+0000 7f4117200700 1 --2- 172.21.15.22:0/42993026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41080098a0 tx=0x7f4108009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.740+0000 7f4117200700 1 -- 172.21.15.22:0/42993026 shutdown_connections 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.740+0000 7f4117200700 1 --2- 172.21.15.22:0/42993026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f411807cb90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.740+0000 7f4117200700 1 -- 172.21.15.22:0/42993026 >> 172.21.15.22:0/42993026 conn(0x7f41181030a0 msgr2=0x7f41181054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.740+0000 7f4117200700 1 -- 172.21.15.22:0/42993026 shutdown_connections 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.740+0000 7f4117200700 1 -- 172.21.15.22:0/42993026 wait complete. 2024-03-24T06:28:40.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.741+0000 7f4117200700 1 Processor -- start 2024-03-24T06:28:40.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.741+0000 7f4117200700 1 -- start start 2024-03-24T06:28:40.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.741+0000 7f4117200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.741+0000 7f4117200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f41181a2430 con 0x7f411807e730 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.742+0000 7f4115e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.742+0000 7f4115e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36910/0 (socket says 172.21.15.22:36910) 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.742+0000 7f4115e00700 1 -- 172.21.15.22:0/1614174976 learned_addr learned my addr 172.21.15.22:0/1614174976 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.742+0000 7f4115e00700 1 -- 172.21.15.22:0/1614174976 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4108009550 con 0x7f411807e730 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.742+0000 7f4115e00700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4108000c00 tx=0x7f4108011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.743+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4108003aa0 con 0x7f411807e730 2024-03-24T06:28:40.644 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.743+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4108003c00 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.743+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4108024930 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.743+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41181a2630 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.743+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41181a2ad0 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.744+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f410801aca0 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.744+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4118109b30 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.745+0000 7f410ea00700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 0x7f40fc03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.745+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f410804d100 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.745+0000 7f4115400700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 0x7f40fc03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.746+0000 7f4115400700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 0x7f40fc03b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4104006fd0 tx=0x7f4104006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:39.751+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f410801a430 con 0x7f411807e730 2024-03-24T06:28:40.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.006+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4118062640 con 0x7f40fc038ee0 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.013+0000 7f410ea00700 1 -- 172.21.15.22:0/1614174976 <== mgr.14120 v2:172.21.15.22:6800/2009113619 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4118062640 con 0x7f40fc038ee0 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.015+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 msgr2=0x7f40fc03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.015+0000 7f4117200700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 0x7f40fc03b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4104006fd0 tx=0x7f4104006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 msgr2=0x7f41181a1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4108000c00 tx=0x7f4108011b00 comp rx=0 tx=0).stop 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 shutdown_connections 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f40fc038ee0 0x7f40fc03b3a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 --2- 172.21.15.22:0/1614174976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f411807e730 0x7f41181a1ef0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 >> 172.21.15.22:0/1614174976 conn(0x7f41181030a0 msgr2=0x7f41181054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 shutdown_connections 2024-03-24T06:28:40.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:40.016+0000 7f4117200700 1 -- 172.21.15.22:0/1614174976 wait complete. 2024-03-24T06:28:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:40 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:40 smithi022 ceph-mon[32003]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:40.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:40 smithi022 ceph-mon[32003]: Saving service mgr spec with placement count:2 2024-03-24T06:28:40.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:40 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:40.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:40 smithi022 ceph-mon[32003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:28:42.463 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.210+0000 7f9cf2a00700 1 Processor -- start 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.210+0000 7f9cf2a00700 1 -- start start 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.211+0000 7f9cf2a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.211+0000 7f9cf2a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec108cd0 con 0x7f9cec1082e0 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.211+0000 7f9cebe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.211+0000 7f9cebe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36918/0 (socket says 172.21.15.22:36918) 2024-03-24T06:28:42.465 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.211+0000 7f9cebe00700 1 -- 172.21.15.22:0/504725414 learned_addr learned my addr 172.21.15.22:0/504725414 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:42.466 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.212+0000 7f9cebe00700 1 -- 172.21.15.22:0/504725414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec106820 con 0x7f9cec1082e0 2024-03-24T06:28:42.466 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.212+0000 7f9cebe00700 1 --2- 172.21.15.22:0/504725414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0098a0 tx=0x7f9cdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5bf6bfefd9c82ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:42.472 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.213+0000 7f9ceaa00700 1 -- 172.21.15.22:0/504725414 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc004160 con 0x7f9cec1082e0 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.213+0000 7f9ceaa00700 1 -- 172.21.15.22:0/504725414 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc0042c0 con 0x7f9cec1082e0 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.213+0000 7f9ceaa00700 1 -- 172.21.15.22:0/504725414 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc011a80 con 0x7f9cec1082e0 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 -- 172.21.15.22:0/504725414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 msgr2=0x7f9cec108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 --2- 172.21.15.22:0/504725414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0098a0 tx=0x7f9cdc009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 -- 172.21.15.22:0/504725414 shutdown_connections 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 --2- 172.21.15.22:0/504725414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec108700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 -- 172.21.15.22:0/504725414 >> 172.21.15.22:0/504725414 conn(0x7f9cec102180 msgr2=0x7f9cec1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 -- 172.21.15.22:0/504725414 shutdown_connections 2024-03-24T06:28:42.473 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.214+0000 7f9cf2a00700 1 -- 172.21.15.22:0/504725414 wait complete. 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.215+0000 7f9cf2a00700 1 Processor -- start 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cf2a00700 1 -- start start 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cf2a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cf2a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec19e2e0 con 0x7f9cec1082e0 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cebe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cebe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36924/0 (socket says 172.21.15.22:36924) 2024-03-24T06:28:42.474 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.216+0000 7f9cebe00700 1 -- 172.21.15.22:0/521027533 learned_addr learned my addr 172.21.15.22:0/521027533 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9cebe00700 1 -- 172.21.15.22:0/521027533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc009550 con 0x7f9cec1082e0 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9cebe00700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc011740 tx=0x7f9cdc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc003b20 con 0x7f9cec1082e0 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc01a910 con 0x7f9cec1082e0 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cdc024850 con 0x7f9cec1082e0 2024-03-24T06:28:42.475 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9cf2a00700 1 -- 172.21.15.22:0/521027533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec19e4e0 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.217+0000 7f9cf2a00700 1 -- 172.21.15.22:0/521027533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec19e980 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.219+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc4005330 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.219+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f9cdc01aa80 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.219+0000 7f9ce8c00700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 0x7f9cd003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.219+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9cdc04d2e0 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.220+0000 7f9ceb400700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 0x7f9cd003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.220+0000 7f9ceb400700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 0x7f9cd003b3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8006fd0 tx=0x7f9cd8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.225+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9cdc01ae10 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.458+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9cc40055e0 con 0x7f9cec1082e0 2024-03-24T06:28:42.476 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.463+0000 7f9ce8c00700 1 -- 172.21.15.22:0/521027533 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f9cdc018be0 con 0x7f9cec1082e0 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 msgr2=0x7f9cd003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 0x7f9cd003b3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8006fd0 tx=0x7f9cd8006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 msgr2=0x7f9cec19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc011740 tx=0x7f9cdc011b00 comp rx=0 tx=0).stop 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 shutdown_connections 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f9cd0038f30 0x7f9cd003b3f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 --2- 172.21.15.22:0/521027533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cec1082e0 0x7f9cec19dda0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.466+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 >> 172.21.15.22:0/521027533 conn(0x7f9cec102180 msgr2=0x7f9cec102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:42.477 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.467+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 shutdown_connections 2024-03-24T06:28:42.478 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:41.467+0000 7f9ce2000700 1 -- 172.21.15.22:0/521027533 wait complete. 2024-03-24T06:28:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/521027533' entity='client.admin' 2024-03-24T06:28:42.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:42 smithi022 ceph-mon[32003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:28:43.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:43 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:43.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2461321495' entity='client.admin' 2024-03-24T06:28:44.131 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.124+0000 7f537ea00700 1 Processor -- start 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.124+0000 7f537ea00700 1 -- start start 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.125+0000 7f537ea00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.125+0000 7f537ea00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f537807df80 con 0x7f537807c2a0 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.125+0000 7f537d600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.125+0000 7f537d600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36940/0 (socket says 172.21.15.22:36940) 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.125+0000 7f537d600700 1 -- 172.21.15.22:0/939766796 learned_addr learned my addr 172.21.15.22:0/939766796 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.126+0000 7f537d600700 1 -- 172.21.15.22:0/939766796 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537807e0c0 con 0x7f537807c2a0 2024-03-24T06:28:44.132 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.126+0000 7f537d600700 1 --2- 172.21.15.22:0/939766796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f536c0098a0 tx=0x7f536c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee047db999a60a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.127+0000 7f5377e00700 1 -- 172.21.15.22:0/939766796 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f536c004160 con 0x7f537807c2a0 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.127+0000 7f5377e00700 1 -- 172.21.15.22:0/939766796 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f536c0042c0 con 0x7f537807c2a0 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.127+0000 7f5377e00700 1 -- 172.21.15.22:0/939766796 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f536c011a80 con 0x7f537807c2a0 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 -- 172.21.15.22:0/939766796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 msgr2=0x7f537807c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 --2- 172.21.15.22:0/939766796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f536c0098a0 tx=0x7f536c009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 -- 172.21.15.22:0/939766796 shutdown_connections 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 --2- 172.21.15.22:0/939766796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537807c6c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 -- 172.21.15.22:0/939766796 >> 172.21.15.22:0/939766796 conn(0x7f5378103090 msgr2=0x7f53781054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:44.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 -- 172.21.15.22:0/939766796 shutdown_connections 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.128+0000 7f537ea00700 1 -- 172.21.15.22:0/939766796 wait complete. 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.129+0000 7f537ea00700 1 Processor -- start 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537ea00700 1 -- start start 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537ea00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537ea00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f537819e0a0 con 0x7f537807c2a0 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537d600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537d600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36946/0 (socket says 172.21.15.22:36946) 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.130+0000 7f537d600700 1 -- 172.21.15.22:0/2461321495 learned_addr learned my addr 172.21.15.22:0/2461321495 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:44.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f537d600700 1 -- 172.21.15.22:0/2461321495 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c009550 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f537d600700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f536c011740 tx=0x7f536c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f536c003b20 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f536c01a910 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f536c0248c0 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.131+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537819e2a0 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.132+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537819e740 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.133+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537810e010 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.133+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f536c01aa80 con 0x7f537807c2a0 2024-03-24T06:28:44.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.134+0000 7f5376000700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 0x7f536003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.134+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f536c04d7d0 con 0x7f537807c2a0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.134+0000 7f537cc00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 0x7f536003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.135+0000 7f537cc00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 0x7f536003b3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fd0 tx=0x7f5364006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.139+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f536c024a80 con 0x7f537807c2a0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.383+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f537810e220 con 0x7f537807c2a0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.389+0000 7f5376000700 1 -- 172.21.15.22:0/2461321495 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f536c04b0c0 con 0x7f537807c2a0 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 msgr2=0x7f536003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:44.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 0x7f536003b3f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fd0 tx=0x7f5364006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 msgr2=0x7f537819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f536c011740 tx=0x7f536c011b00 comp rx=0 tx=0).stop 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 shutdown_connections 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f5360038f30 0x7f536003b3f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 --2- 172.21.15.22:0/2461321495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f537807c2a0 0x7f537819db60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.393+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 >> 172.21.15.22:0/2461321495 conn(0x7f5378103090 msgr2=0x7f5378103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.394+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 shutdown_connections 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:43.394+0000 7f537ea00700 1 -- 172.21.15.22:0/2461321495 wait complete. 2024-03-24T06:28:44.137 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling the dashboard module... 2024-03-24T06:28:44.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:44 smithi022 ceph-mon[32003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:28:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:45 smithi022 ceph-mon[32003]: from='mgr.14120 172.21.15.22:0/3401801180' entity='mgr.x' 2024-03-24T06:28:46.033 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:45 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1594166132' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T06:28:46.034 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:45 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setuser ceph since I am not root 2024-03-24T06:28:46.034 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:45 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setgroup ceph since I am not root 2024-03-24T06:28:46.709 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:46 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:46.593+0000 7f026e3bf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:28:46.960 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:46 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:46.720+0000 7f026e3bf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:28:46.961 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:46 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1594166132' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T06:28:46.961 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:46 smithi022 ceph-mon[32003]: mgrmap e9: x(active, since 25s) 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.691+0000 7f25a8c00700 1 Processor -- start 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.691+0000 7f25a8c00700 1 -- start start 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.692+0000 7f25a8c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.692+0000 7f25a8c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410b0a0 con 0x7f25a4108770 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.692+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.692+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36954/0 (socket says 172.21.15.22:36954) 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.692+0000 7f25a1e00700 1 -- 172.21.15.22:0/1283217236 learned_addr learned my addr 172.21.15.22:0/1283217236 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.693+0000 7f25a1e00700 1 -- 172.21.15.22:0/1283217236 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a410b1e0 con 0x7f25a4108770 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.693+0000 7f25a1e00700 1 --2- 172.21.15.22:0/1283217236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25940098a0 tx=0x7f2594009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c8b14c6ac0cf57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.693+0000 7f25a0a00700 1 -- 172.21.15.22:0/1283217236 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2594004160 con 0x7f25a4108770 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.693+0000 7f25a0a00700 1 -- 172.21.15.22:0/1283217236 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25940042c0 con 0x7f25a4108770 2024-03-24T06:28:47.241 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.693+0000 7f25a0a00700 1 -- 172.21.15.22:0/1283217236 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2594011a80 con 0x7f25a4108770 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.694+0000 7f25a8c00700 1 -- 172.21.15.22:0/1283217236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 msgr2=0x7f25a410ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.694+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1283217236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25940098a0 tx=0x7f2594009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.695+0000 7f25a8c00700 1 -- 172.21.15.22:0/1283217236 shutdown_connections 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.695+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1283217236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a410ab60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.695+0000 7f25a8c00700 1 -- 172.21.15.22:0/1283217236 >> 172.21.15.22:0/1283217236 conn(0x7f25a4102180 msgr2=0x7f25a41045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.695+0000 7f25a8c00700 1 -- 172.21.15.22:0/1283217236 shutdown_connections 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.695+0000 7f25a8c00700 1 -- 172.21.15.22:0/1283217236 wait complete. 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.696+0000 7f25a8c00700 1 Processor -- start 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.696+0000 7f25a8c00700 1 -- start start 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.696+0000 7f25a8c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.696+0000 7f25a8c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f25a419c070 con 0x7f25a4108770 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.697+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.697+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36968/0 (socket says 172.21.15.22:36968) 2024-03-24T06:28:47.242 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.697+0000 7f25a1e00700 1 -- 172.21.15.22:0/1594166132 learned_addr learned my addr 172.21.15.22:0/1594166132 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.697+0000 7f25a1e00700 1 -- 172.21.15.22:0/1594166132 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2594009550 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.697+0000 7f25a1e00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2594011740 tx=0x7f2594011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.698+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2594003b20 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.698+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f259401a910 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.698+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a419c270 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.698+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2594024900 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.698+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a419c710 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.699+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a410c020 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.699+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f2594024b70 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.700+0000 7f259aa00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 0x7f258803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.700+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f259404d240 con 0x7f25a4108770 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.700+0000 7f25a1400700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 0x7f258803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.701+0000 7f25a1400700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 0x7f258803b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:47.243 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:44.706+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2594018530 con 0x7f25a4108770 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.012+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f25a419f340 con 0x7f25a4108770 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.749+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2594018350 con 0x7f25a4108770 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.752+0000 7f259aa00700 1 -- 172.21.15.22:0/1594166132 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f259402ad40 con 0x7f25a4108770 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 msgr2=0x7f258803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 0x7f258803b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 msgr2=0x7f25a419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2594011740 tx=0x7f2594011a50 comp rx=0 tx=0).stop 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 shutdown_connections 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2588038e90 0x7f258803b350 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 --2- 172.21.15.22:0/1594166132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25a4108770 0x7f25a419bb30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.755+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 >> 172.21.15.22:0/1594166132 conn(0x7f25a4102180 msgr2=0x7f25a4102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.756+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 shutdown_connections 2024-03-24T06:28:47.244 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:45.756+0000 7f25a8c00700 1 -- 172.21.15.22:0/1594166132 wait complete. 2024-03-24T06:28:47.823 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:47 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:47.559+0000 7f026e3bf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:28:47.824 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:47 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:47.687+0000 7f026e3bf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:28:48.119 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:47 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:47.875+0000 7f026e3bf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:28:48.119 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.006+0000 7f026e3bf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:28:48.379 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1886502421' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T06:28:48.379 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.122+0000 7f026e3bf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.823+0000 7f256f600700 1 Processor -- start 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.823+0000 7f256f600700 1 -- start start 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.824+0000 7f256f600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:48.635 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.824+0000 7f256f600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2568109dd0 con 0x7f25681093e0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.824+0000 7f256cc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.824+0000 7f256cc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52350/0 (socket says 172.21.15.22:52350) 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.824+0000 7f256cc00700 1 -- 172.21.15.22:0/884954508 learned_addr learned my addr 172.21.15.22:0/884954508 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.825+0000 7f256cc00700 1 -- 172.21.15.22:0/884954508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2568107920 con 0x7f25681093e0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.825+0000 7f256cc00700 1 --2- 172.21.15.22:0/884954508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f255c009ab0 tx=0x7f255c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46eaf4b13b229d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.826+0000 7f2567400700 1 -- 172.21.15.22:0/884954508 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f255c004160 con 0x7f25681093e0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.826+0000 7f2567400700 1 -- 172.21.15.22:0/884954508 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f255c00bcc0 con 0x7f25681093e0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.826+0000 7f2567400700 1 -- 172.21.15.22:0/884954508 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f255c003c50 con 0x7f25681093e0 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.826+0000 7f256f600700 1 -- 172.21.15.22:0/884954508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 msgr2=0x7f2568109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.826+0000 7f256f600700 1 --2- 172.21.15.22:0/884954508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f255c009ab0 tx=0x7f255c009dc0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.827+0000 7f256f600700 1 -- 172.21.15.22:0/884954508 shutdown_connections 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.827+0000 7f256f600700 1 --2- 172.21.15.22:0/884954508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f2568109800 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.827+0000 7f256f600700 1 -- 172.21.15.22:0/884954508 >> 172.21.15.22:0/884954508 conn(0x7f25681032c0 msgr2=0x7f25681056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.827+0000 7f256f600700 1 -- 172.21.15.22:0/884954508 shutdown_connections 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.827+0000 7f256f600700 1 -- 172.21.15.22:0/884954508 wait complete. 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.828+0000 7f256f600700 1 Processor -- start 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.828+0000 7f256f600700 1 -- start start 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.828+0000 7f256f600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.828+0000 7f256f600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f256819e280 con 0x7f25681093e0 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.829+0000 7f256cc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:48.637 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.829+0000 7f256cc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52366/0 (socket says 172.21.15.22:52366) 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.829+0000 7f256cc00700 1 -- 172.21.15.22:0/1886502421 learned_addr learned my addr 172.21.15.22:0/1886502421 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.829+0000 7f256cc00700 1 -- 172.21.15.22:0/1886502421 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255c009550 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.829+0000 7f256cc00700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f255c00ba10 tx=0x7f255c00baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.830+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f255c0040d0 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.830+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f255c004230 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.830+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256819e480 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.830+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f255c011420 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.830+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256819e8a0 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.831+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f255c0116e0 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.831+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256810c130 con 0x7f25681093e0 2024-03-24T06:28:48.638 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.832+0000 7f2565600700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 0x7f255003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:48.639 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.832+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f255c04d0c0 con 0x7f25681093e0 2024-03-24T06:28:48.639 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.472+0000 7f026e3bf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:28:48.639 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.616+0000 7f026e3bf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:28:48.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.832+0000 7f2567e00700 1 -- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 msgr2=0x7f255003b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:48.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.832+0000 7f2567e00700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 0x7f255003b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:28:48.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:47.837+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f255c018dc0 con 0x7f25681093e0 2024-03-24T06:28:48.641 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.033+0000 7f2567e00700 1 -- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 msgr2=0x7f255003b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.033+0000 7f2567e00700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 0x7f255003b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.118+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f25681a13d0 con 0x7f25681093e0 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.119+0000 7f2565600700 1 -- 172.21.15.22:0/1886502421 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f255c018be0 con 0x7f25681093e0 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.122+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 msgr2=0x7f255003b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.122+0000 7f256f600700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 0x7f255003b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.122+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 msgr2=0x7f256819dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.122+0000 7f256f600700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f255c00ba10 tx=0x7f255c00baf0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 shutdown_connections 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7f2550038f30 0x7f255003b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 --2- 172.21.15.22:0/1886502421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25681093e0 0x7f256819dd40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:48.642 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 >> 172.21.15.22:0/1886502421 conn(0x7f25681032c0 msgr2=0x7f2568103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:48.643 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 shutdown_connections 2024-03-24T06:28:48.643 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:48.123+0000 7f256f600700 1 -- 172.21.15.22:0/1886502421 wait complete. 2024-03-24T06:28:48.643 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for the mgr to restart... 2024-03-24T06:28:48.643 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr epoch 9... 2024-03-24T06:28:48.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.749+0000 7f026e3bf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:28:48.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:48 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:48.877+0000 7f026e3bf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:28:49.224 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:49 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:49.106+0000 7f026e3bf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:28:50.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:50 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:50.434+0000 7f026e3bf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:28:50.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:50 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:50.611+0000 7f026e3bf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:28:51.398 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:51.040+0000 7f026e3bf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:28:51.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:51.162+0000 7f026e3bf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:28:51.687 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:51.431+0000 7f026e3bf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:52.077 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:51.686+0000 7f026e3bf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:28:52.077 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:52.076+0000 7f026e3bf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:28:52.717 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:52.399+0000 7f026e3bf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:28:53.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:52.717+0000 7f026e3bf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:28:54.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:53 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:53.788+0000 7f026e3bf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:28:54.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:54.507+0000 7f026e3bf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:28:54.899 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:54.642+0000 7f026e3bf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:28:55.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:28:54.937+0000 7f026e3bf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:28:55.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: Active manager daemon x restarted 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: Activating manager daemon x 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: mgrmap e10: x(active, starting, since 0.00905904s) 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:28:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:54 smithi022 ceph-mon[32003]: Manager daemon x is now available 2024-03-24T06:28:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:28:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:28:56.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:28:56.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:55 smithi022 ceph-mon[32003]: mgrmap e11: x(active, since 1.00997s) 2024-03-24T06:28:56.777 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T06:28:56.778 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T06:28:56.778 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T06:28:56.778 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T06:28:56.779 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.224+0000 7fd12e200700 1 Processor -- start 2024-03-24T06:28:56.779 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.224+0000 7fd12e200700 1 -- start start 2024-03-24T06:28:56.780 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.224+0000 7fd12e200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:56.780 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.225+0000 7fd12e200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd12807dfd0 con 0x7fd12807c2a0 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.225+0000 7fd127400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.225+0000 7fd127400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52380/0 (socket says 172.21.15.22:52380) 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.225+0000 7fd127400700 1 -- 172.21.15.22:0/3663136593 learned_addr learned my addr 172.21.15.22:0/3663136593 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.225+0000 7fd127400700 1 -- 172.21.15.22:0/3663136593 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12807e110 con 0x7fd12807c2a0 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.226+0000 7fd127400700 1 --2- 172.21.15.22:0/3663136593 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1180098a0 tx=0x7fd118009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eddf6918159b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.226+0000 7fd126000700 1 -- 172.21.15.22:0/3663136593 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd118004160 con 0x7fd12807c2a0 2024-03-24T06:28:56.781 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.226+0000 7fd126000700 1 -- 172.21.15.22:0/3663136593 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1180042c0 con 0x7fd12807c2a0 2024-03-24T06:28:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.226+0000 7fd126000700 1 -- 172.21.15.22:0/3663136593 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd118011a80 con 0x7fd12807c2a0 2024-03-24T06:28:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 -- 172.21.15.22:0/3663136593 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 msgr2=0x7fd12807c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 --2- 172.21.15.22:0/3663136593 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1180098a0 tx=0x7fd118009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 -- 172.21.15.22:0/3663136593 shutdown_connections 2024-03-24T06:28:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 --2- 172.21.15.22:0/3663136593 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12807c6c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.784 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 -- 172.21.15.22:0/3663136593 >> 172.21.15.22:0/3663136593 conn(0x7fd1281032a0 msgr2=0x7fd1281056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:56.784 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 -- 172.21.15.22:0/3663136593 shutdown_connections 2024-03-24T06:28:56.784 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.227+0000 7fd12e200700 1 -- 172.21.15.22:0/3663136593 wait complete. 2024-03-24T06:28:56.784 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.228+0000 7fd12e200700 1 Processor -- start 2024-03-24T06:28:56.784 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.228+0000 7fd12e200700 1 -- start start 2024-03-24T06:28:56.785 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd12e200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd12e200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd12819e250 con 0x7fd12807c2a0 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd127400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd127400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52396/0 (socket says 172.21.15.22:52396) 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd127400700 1 -- 172.21.15.22:0/138618714 learned_addr learned my addr 172.21.15.22:0/138618714 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.229+0000 7fd127400700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd118009550 con 0x7fd12807c2a0 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd127400700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd118011740 tx=0x7fd118011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:56.786 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd118003b20 con 0x7fd12807c2a0 2024-03-24T06:28:56.788 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd11801a910 con 0x7fd12807c2a0 2024-03-24T06:28:56.788 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd1180248c0 con 0x7fd12807c2a0 2024-03-24T06:28:56.788 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12819e450 con 0x7fd12807c2a0 2024-03-24T06:28:56.788 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.230+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12819e8f0 con 0x7fd12807c2a0 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.231+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fd11801aa80 con 0x7fd12807c2a0 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.232+0000 7fd11fe00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.232+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd10c03bbb0 con 0x7fd10c038fe0 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.232+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd11804d0d0 con 0x7fd12807c2a0 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.232+0000 7fd126a00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:56.789 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.232+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:28:56.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.433+0000 7fd126a00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:56.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.433+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:28:56.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.834+0000 7fd126a00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:56.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:49.834+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:50.635+0000 7fd126a00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:50.635+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:52.237+0000 7fd126a00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2009113619 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:52.237+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:54.947+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mgrmap(e 10) ==== 45708+0+0 (secure 0 0 0) 0x7fd118024a80 con 0x7fd12807c2a0 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:54.947+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 msgr2=0x7fd10c03b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:54.947+0000 7fd11fe00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.948+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7fd11804e490 con 0x7fd12807c2a0 2024-03-24T06:28:56.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.948+0000 7fd11fe00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 0x7fd10c03eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.948+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd10c03bbb0 con 0x7fd10c03cab0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.949+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 0x7fd10c03eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.951+0000 7fd126a00700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 0x7fd10c03eea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd114003a10 tx=0x7fd1140092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.952+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd10c03bbb0 con 0x7fd10c03cab0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.960+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd12807e110 con 0x7fd10c03cab0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.960+0000 7fd11fe00700 1 -- 172.21.15.22:0/138618714 <== mgr.14152 v2:172.21.15.22:6800/503368228 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fd12807e110 con 0x7fd10c03cab0 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 msgr2=0x7fd10c03eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 0x7fd10c03eea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd114003a10 tx=0x7fd1140092b0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 msgr2=0x7fd12819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:56.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd118011740 tx=0x7fd118011b00 comp rx=0 tx=0).stop 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 shutdown_connections 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd10c03cab0 0x7fd10c03eea0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd12807c2a0 0x7fd12819dd10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 --2- 172.21.15.22:0/138618714 >> [v2:172.21.15.22:6800/2009113619,v1:172.21.15.22:6801/2009113619] conn(0x7fd10c038fe0 0x7fd10c03b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 >> 172.21.15.22:0/138618714 conn(0x7fd1281032a0 msgr2=0x7fd128103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.961+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 shutdown_connections 2024-03-24T06:28:56.796 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:55.962+0000 7fd12e200700 1 -- 172.21.15.22:0/138618714 wait complete. 2024-03-24T06:28:56.797 INFO:teuthology.orchestra.run.smithi022.stdout:mgr epoch 9 is available 2024-03-24T06:28:56.797 INFO:teuthology.orchestra.run.smithi022.stdout:Generating a dashboard self-signed certificate... 2024-03-24T06:28:57.082 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:57 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:55] ENGINE Bus STARTING 2024-03-24T06:28:57.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:57 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:56] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T06:28:57.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:57 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:56] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T06:28:57.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:57 smithi022 ceph-mon[32003]: [24/Mar/2024:06:28:56] ENGINE Bus STARTED 2024-03-24T06:28:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:58 smithi022 ceph-mon[32003]: mgrmap e12: x(active, since 2s) 2024-03-24T06:28:58.279 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.902+0000 7fd3f2000700 1 Processor -- start 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.902+0000 7fd3f2000700 1 -- start start 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.902+0000 7fd3f2000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.902+0000 7fd3f2000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec07dfd0 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.903+0000 7fd3eb400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.903+0000 7fd3eb400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55062/0 (socket says 172.21.15.22:55062) 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.903+0000 7fd3eb400700 1 -- 172.21.15.22:0/55368260 learned_addr learned my addr 172.21.15.22:0/55368260 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.904+0000 7fd3eb400700 1 -- 172.21.15.22:0/55368260 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec07e110 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.904+0000 7fd3eb400700 1 --2- 172.21.15.22:0/55368260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0098a0 tx=0x7fd3dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc453a8a791c424 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.904+0000 7fd3ea000700 1 -- 172.21.15.22:0/55368260 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3dc004160 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.904+0000 7fd3ea000700 1 -- 172.21.15.22:0/55368260 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc0042c0 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.904+0000 7fd3ea000700 1 -- 172.21.15.22:0/55368260 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3dc011a80 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.905+0000 7fd3f2000700 1 -- 172.21.15.22:0/55368260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 msgr2=0x7fd3ec07c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.905+0000 7fd3f2000700 1 --2- 172.21.15.22:0/55368260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0098a0 tx=0x7fd3dc009bb0 comp rx=0 tx=0).stop 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.905+0000 7fd3f2000700 1 -- 172.21.15.22:0/55368260 shutdown_connections 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.905+0000 7fd3f2000700 1 --2- 172.21.15.22:0/55368260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec07c6c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.905+0000 7fd3f2000700 1 -- 172.21.15.22:0/55368260 >> 172.21.15.22:0/55368260 conn(0x7fd3ec1032a0 msgr2=0x7fd3ec1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.906+0000 7fd3f2000700 1 -- 172.21.15.22:0/55368260 shutdown_connections 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.906+0000 7fd3f2000700 1 -- 172.21.15.22:0/55368260 wait complete. 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.907+0000 7fd3f2000700 1 Processor -- start 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.907+0000 7fd3f2000700 1 -- start start 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.908+0000 7fd3f2000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.908+0000 7fd3f2000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec19e250 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.119 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.908+0000 7fd3eb400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.908+0000 7fd3eb400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55078/0 (socket says 172.21.15.22:55078) 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.908+0000 7fd3eb400700 1 -- 172.21.15.22:0/2916720582 learned_addr learned my addr 172.21.15.22:0/2916720582 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3eb400700 1 -- 172.21.15.22:0/2916720582 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc009550 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3eb400700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc011740 tx=0x7fd3dc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3dc003b20 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc01a910 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3dc0248c0 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.909+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec19e450 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.910+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec19e8f0 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.911+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7fd3dc01aa80 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.911+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ec10e260 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.912+0000 7fd3e3e00700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 0x7fd3d003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.912+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd3dc04d340 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.912+0000 7fd3eaa00700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 0x7fd3d003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:28:59.120 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.913+0000 7fd3eaa00700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 0x7fd3d003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006fd0 tx=0x7fd3d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:57.922+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd3dc018530 con 0x7fd3ec07c2a0 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.157+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd3ec07e110 con 0x7fd3d0038f90 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.279+0000 7fd3e3e00700 1 -- 172.21.15.22:0/2916720582 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd3ec07e110 con 0x7fd3d0038f90 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 msgr2=0x7fd3d003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 0x7fd3d003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8006fd0 tx=0x7fd3d8006e40 comp rx=0 tx=0).stop 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 msgr2=0x7fd3ec19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc011740 tx=0x7fd3dc011b00 comp rx=0 tx=0).stop 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 shutdown_connections 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd3d0038f90 0x7fd3d003b450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 --2- 172.21.15.22:0/2916720582 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3ec07c2a0 0x7fd3ec19dd10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 >> 172.21.15.22:0/2916720582 conn(0x7fd3ec1032a0 msgr2=0x7fd3ec103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 shutdown_connections 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:58.282+0000 7fd3f2000700 1 -- 172.21.15.22:0/2916720582 wait complete. 2024-03-24T06:28:59.121 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial admin user... 2024-03-24T06:28:59.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:59 smithi022 ceph-mon[32003]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:28:59.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:28:59.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:28:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:00.918 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:00.919 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:01.253 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$RrtOzKbkBZrpCSV.c2GdgOqTNcfKxPQziA3l6a.Myn5c9jbJgSKXq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711261740, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T06:29:01.253 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.704+0000 7f3eedc00700 1 Processor -- start 2024-03-24T06:29:01.253 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.705+0000 7f3eedc00700 1 -- start start 2024-03-24T06:29:01.253 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.705+0000 7f3eedc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:01.253 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.705+0000 7f3eedc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee810bf20 con 0x7f3ee810b5c0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.705+0000 7f3ee6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.706+0000 7f3ee6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55086/0 (socket says 172.21.15.22:55086) 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.706+0000 7f3ee6a00700 1 -- 172.21.15.22:0/4121133827 learned_addr learned my addr 172.21.15.22:0/4121133827 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.706+0000 7f3ee6a00700 1 -- 172.21.15.22:0/4121133827 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee8109d30 con 0x7f3ee810b5c0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.706+0000 7f3ee6a00700 1 --2- 172.21.15.22:0/4121133827 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80098a0 tx=0x7f3ed8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6334612e7b1c92b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.707+0000 7f3ee5600700 1 -- 172.21.15.22:0/4121133827 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8004160 con 0x7f3ee810b5c0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.707+0000 7f3ee5600700 1 -- 172.21.15.22:0/4121133827 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed80042c0 con 0x7f3ee810b5c0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.707+0000 7f3ee5600700 1 -- 172.21.15.22:0/4121133827 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8011a80 con 0x7f3ee810b5c0 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.707+0000 7f3eedc00700 1 -- 172.21.15.22:0/4121133827 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 msgr2=0x7f3ee810b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.707+0000 7f3eedc00700 1 --2- 172.21.15.22:0/4121133827 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80098a0 tx=0x7f3ed8009bb0 comp rx=0 tx=0).stop 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.708+0000 7f3eedc00700 1 -- 172.21.15.22:0/4121133827 shutdown_connections 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.708+0000 7f3eedc00700 1 --2- 172.21.15.22:0/4121133827 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee810b9e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.708+0000 7f3eedc00700 1 -- 172.21.15.22:0/4121133827 >> 172.21.15.22:0/4121133827 conn(0x7f3ee807d360 msgr2=0x7f3ee807d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.708+0000 7f3eedc00700 1 -- 172.21.15.22:0/4121133827 shutdown_connections 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.708+0000 7f3eedc00700 1 -- 172.21.15.22:0/4121133827 wait complete. 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.709+0000 7f3eedc00700 1 Processor -- start 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.709+0000 7f3eedc00700 1 -- start start 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3eedc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3eedc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee819e260 con 0x7f3ee810b5c0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3ee6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3ee6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55100/0 (socket says 172.21.15.22:55100) 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3ee6a00700 1 -- 172.21.15.22:0/293698534 learned_addr learned my addr 172.21.15.22:0/293698534 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.710+0000 7f3ee6a00700 1 -- 172.21.15.22:0/293698534 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed8009550 con 0x7f3ee810b5c0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3ee6a00700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8011740 tx=0x7f3ed8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8003ba0 con 0x7f3ee810b5c0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed801a910 con 0x7f3ee810b5c0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee819e460 con 0x7f3ee810b5c0 2024-03-24T06:29:01.255 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ed8024940 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.711+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee819e900 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.712+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f3ed801a430 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.712+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee810e350 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.713+0000 7f3edf400700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 0x7f3ecc03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.713+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ed804d930 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.713+0000 7f3ee6000700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 0x7f3ecc03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.714+0000 7f3ee6000700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 0x7f3ecc03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006fd0 tx=0x7f3ed4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.722+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ed8010380 con 0x7f3ee810b5c0 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:28:59.970+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f3ee8063650 con 0x7f3ecc038f30 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.241+0000 7f3edf400700 1 -- 172.21.15.22:0/293698534 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f3ee8063650 con 0x7f3ecc038f30 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 msgr2=0x7f3ecc03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 0x7f3ecc03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006fd0 tx=0x7f3ed4006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:01.256 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 msgr2=0x7f3ee819dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8011740 tx=0x7f3ed8011b00 comp rx=0 tx=0).stop 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 shutdown_connections 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3ecc038f30 0x7f3ecc03b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 --2- 172.21.15.22:0/293698534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ee810b5c0 0x7f3ee819dd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.245+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 >> 172.21.15.22:0/293698534 conn(0x7f3ee807d360 msgr2=0x7f3ee8107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.246+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 shutdown_connections 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:00.246+0000 7f3eedc00700 1 -- 172.21.15.22:0/293698534 wait complete. 2024-03-24T06:29:01.257 INFO:teuthology.orchestra.run.smithi022.stdout:Fetching dashboard port number... 2024-03-24T06:29:02.067 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:01 smithi022 ceph-mon[32003]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:02.966 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:02 smithi022 ceph-mon[32003]: mgrmap e13: x(active, since 6s) 2024-03-24T06:29:02.966 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:02 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1220197594' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T06:29:02.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T06:29:02.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.067+0000 7fde26a00700 1 Processor -- start 2024-03-24T06:29:02.974 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.068+0000 7fde26a00700 1 -- start start 2024-03-24T06:29:02.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.068+0000 7fde26a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:02.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.068+0000 7fde26a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fde20109dd0 con 0x7fde201093e0 2024-03-24T06:29:02.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.069+0000 7fde1fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:02.975 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.069+0000 7fde1fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55116/0 (socket says 172.21.15.22:55116) 2024-03-24T06:29:02.976 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.069+0000 7fde1fe00700 1 -- 172.21.15.22:0/3939946150 learned_addr learned my addr 172.21.15.22:0/3939946150 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:02.976 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.070+0000 7fde1fe00700 1 -- 172.21.15.22:0/3939946150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde20107920 con 0x7fde201093e0 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.070+0000 7fde1fe00700 1 --2- 172.21.15.22:0/3939946150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde100098a0 tx=0x7fde10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bdd072368ae9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.070+0000 7fde1ea00700 1 -- 172.21.15.22:0/3939946150 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde10004160 con 0x7fde201093e0 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.071+0000 7fde1ea00700 1 -- 172.21.15.22:0/3939946150 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde100042c0 con 0x7fde201093e0 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.071+0000 7fde1ea00700 1 -- 172.21.15.22:0/3939946150 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde10011a80 con 0x7fde201093e0 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 -- 172.21.15.22:0/3939946150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 msgr2=0x7fde20109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 --2- 172.21.15.22:0/3939946150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde100098a0 tx=0x7fde10009bb0 comp rx=0 tx=0).stop 2024-03-24T06:29:02.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 -- 172.21.15.22:0/3939946150 shutdown_connections 2024-03-24T06:29:02.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 --2- 172.21.15.22:0/3939946150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde20109800 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:02.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 -- 172.21.15.22:0/3939946150 >> 172.21.15.22:0/3939946150 conn(0x7fde201032a0 msgr2=0x7fde201056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:02.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 -- 172.21.15.22:0/3939946150 shutdown_connections 2024-03-24T06:29:02.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.072+0000 7fde26a00700 1 -- 172.21.15.22:0/3939946150 wait complete. 2024-03-24T06:29:02.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.073+0000 7fde26a00700 1 Processor -- start 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.073+0000 7fde26a00700 1 -- start start 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde26a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde26a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fde2019e290 con 0x7fde201093e0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde1fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde1fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55122/0 (socket says 172.21.15.22:55122) 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde1fe00700 1 -- 172.21.15.22:0/1220197594 learned_addr learned my addr 172.21.15.22:0/1220197594 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.074+0000 7fde1fe00700 1 -- 172.21.15.22:0/1220197594 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde10009550 con 0x7fde201093e0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde1fe00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde10011740 tx=0x7fde10011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde10003b20 con 0x7fde201093e0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde1001a910 con 0x7fde201093e0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde10024850 con 0x7fde201093e0 2024-03-24T06:29:02.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2019e490 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.075+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2019e930 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.077+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fde1001a430 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.077+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2010c130 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.077+0000 7fde1cc00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 0x7fde0403b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.077+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fde1004d540 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.078+0000 7fde1f400700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 0x7fde0403b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.079+0000 7fde1f400700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 0x7fde0403b4f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde0c006fd0 tx=0x7fde0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.083+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fde1001ae10 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.313+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fde201a13e0 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.313+0000 7fde1cc00700 1 -- 172.21.15.22:0/1220197594 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fde10018be0 con 0x7fde201093e0 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 msgr2=0x7fde0403b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 0x7fde0403b4f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde0c006fd0 tx=0x7fde0c006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 msgr2=0x7fde2019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde10011740 tx=0x7fde10011b00 comp rx=0 tx=0).stop 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 shutdown_connections 2024-03-24T06:29:02.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde04039030 0x7fde0403b4f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:02.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 --2- 172.21.15.22:0/1220197594 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde201093e0 0x7fde2019dd50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:02.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 >> 172.21.15.22:0/1220197594 conn(0x7fde201032a0 msgr2=0x7fde20103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:02.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 shutdown_connections 2024-03-24T06:29:02.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:02.317+0000 7fde26a00700 1 -- 172.21.15.22:0/1220197594 wait complete. 2024-03-24T06:29:03.015 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T06:29:03.015 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T06:29:03.040 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T06:29:03.040 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T06:29:03.041 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T06:29:03.041 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout:Ceph Dashboard is now available at: 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout: URL: https://smithi022.front.sepia.ceph.com:8443/ 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout: User: admin 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout: Password: rxktepnrq6 2024-03-24T06:29:03.049 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:03.050 INFO:teuthology.orchestra.run.smithi022.stdout:Saving cluster configuration to /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config directory 2024-03-24T06:29:03.052 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling autotune for osd_memory_target 2024-03-24T06:29:05.133 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.280+0000 7fcccdc00700 1 Processor -- start 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.281+0000 7fcccdc00700 1 -- start start 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.281+0000 7fcccdc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.281+0000 7fcccdc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fccc81084e0 con 0x7fccc8107af0 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.281+0000 7fccc7e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.282+0000 7fccc7e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55130/0 (socket says 172.21.15.22:55130) 2024-03-24T06:29:05.134 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.282+0000 7fccc7e00700 1 -- 172.21.15.22:0/2165769797 learned_addr learned my addr 172.21.15.22:0/2165769797 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.282+0000 7fccc7e00700 1 -- 172.21.15.22:0/2165769797 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc8108cf0 con 0x7fccc8107af0 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.282+0000 7fccc7e00700 1 --2- 172.21.15.22:0/2165769797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fccb80098a0 tx=0x7fccb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bfdc9c955b9ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.283+0000 7fccc6a00700 1 -- 172.21.15.22:0/2165769797 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb8004160 con 0x7fccc8107af0 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.283+0000 7fccc6a00700 1 -- 172.21.15.22:0/2165769797 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb80042c0 con 0x7fccc8107af0 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.283+0000 7fccc6a00700 1 -- 172.21.15.22:0/2165769797 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb8011a80 con 0x7fccc8107af0 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.283+0000 7fcccdc00700 1 -- 172.21.15.22:0/2165769797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 msgr2=0x7fccc8107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.283+0000 7fcccdc00700 1 --2- 172.21.15.22:0/2165769797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fccb80098a0 tx=0x7fccb8009bb0 comp rx=0 tx=0).stop 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.284+0000 7fcccdc00700 1 -- 172.21.15.22:0/2165769797 shutdown_connections 2024-03-24T06:29:05.135 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.284+0000 7fcccdc00700 1 --2- 172.21.15.22:0/2165769797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc8107f10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.284+0000 7fcccdc00700 1 -- 172.21.15.22:0/2165769797 >> 172.21.15.22:0/2165769797 conn(0x7fccc8103070 msgr2=0x7fccc81054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.284+0000 7fcccdc00700 1 -- 172.21.15.22:0/2165769797 shutdown_connections 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.284+0000 7fcccdc00700 1 -- 172.21.15.22:0/2165769797 wait complete. 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.285+0000 7fcccdc00700 1 Processor -- start 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.285+0000 7fcccdc00700 1 -- start start 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.285+0000 7fcccdc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.285+0000 7fcccdc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fccc807e900 con 0x7fccc8107af0 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.286+0000 7fccc7e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.286+0000 7fccc7e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55138/0 (socket says 172.21.15.22:55138) 2024-03-24T06:29:05.136 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.286+0000 7fccc7e00700 1 -- 172.21.15.22:0/3053294652 learned_addr learned my addr 172.21.15.22:0/3053294652 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.286+0000 7fccc7e00700 1 -- 172.21.15.22:0/3053294652 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb8009550 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.286+0000 7fccc7e00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000c00 tx=0x7fccb8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.287+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb80039e0 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.287+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb8003b40 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.287+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc807caf0 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.287+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb8024820 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.287+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc807cf90 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.288+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc810bf20 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.288+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fccb801ab80 con 0x7fccc8107af0 2024-03-24T06:29:05.137 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.289+0000 7fccc4c00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 0x7fccac03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.289+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fccb804d230 con 0x7fccc8107af0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.289+0000 7fccc7400700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 0x7fccac03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.290+0000 7fccc7400700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 0x7fccac03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccb4006fd0 tx=0x7fccb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.294+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fccb801a950 con 0x7fccc8107af0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.522+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fccc810c130 con 0x7fccc8107af0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.523+0000 7fccc4c00700 1 -- 172.21.15.22:0/3053294652 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fccb804cac0 con 0x7fccc8107af0 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.526+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 msgr2=0x7fccac03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.526+0000 7fcccdc00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 0x7fccac03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fccb4006fd0 tx=0x7fccb4006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.526+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 msgr2=0x7fccc807e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.526+0000 7fcccdc00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fccb8000c00 tx=0x7fccb8011b00 comp rx=0 tx=0).stop 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 shutdown_connections 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fccac038fd0 0x7fccac03b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 --2- 172.21.15.22:0/3053294652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fccc8107af0 0x7fccc807e3c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 >> 172.21.15.22:0/3053294652 conn(0x7fccc8103070 msgr2=0x7fccc8103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 shutdown_connections 2024-03-24T06:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:04.527+0000 7fcccdc00700 1 -- 172.21.15.22:0/3053294652 wait complete. 2024-03-24T06:29:06.801 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.736+0000 7fcc39a00700 1 Processor -- start 2024-03-24T06:29:06.802 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.736+0000 7fcc39a00700 1 -- start start 2024-03-24T06:29:06.802 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.737+0000 7fcc39a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:06.802 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.737+0000 7fcc39a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34108500 con 0x7fcc34107b10 2024-03-24T06:29:06.802 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.737+0000 7fcc33e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.738+0000 7fcc33e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40028/0 (socket says 172.21.15.22:40028) 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.738+0000 7fcc33e00700 1 -- 172.21.15.22:0/2689738431 learned_addr learned my addr 172.21.15.22:0/2689738431 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.738+0000 7fcc33e00700 1 -- 172.21.15.22:0/2689738431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc34108d10 con 0x7fcc34107b10 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.738+0000 7fcc33e00700 1 --2- 172.21.15.22:0/2689738431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc240098a0 tx=0x7fcc24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6087cf72666af836 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc32a00700 1 -- 172.21.15.22:0/2689738431 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc24004160 con 0x7fcc34107b10 2024-03-24T06:29:06.805 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc32a00700 1 -- 172.21.15.22:0/2689738431 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc240042c0 con 0x7fcc34107b10 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc32a00700 1 -- 172.21.15.22:0/2689738431 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc24011a80 con 0x7fcc34107b10 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc39a00700 1 -- 172.21.15.22:0/2689738431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 msgr2=0x7fcc34107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc39a00700 1 --2- 172.21.15.22:0/2689738431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc240098a0 tx=0x7fcc24009bb0 comp rx=0 tx=0).stop 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc39a00700 1 -- 172.21.15.22:0/2689738431 shutdown_connections 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc39a00700 1 --2- 172.21.15.22:0/2689738431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc34107f30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.739+0000 7fcc39a00700 1 -- 172.21.15.22:0/2689738431 >> 172.21.15.22:0/2689738431 conn(0x7fcc34103150 msgr2=0x7fcc34105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.740+0000 7fcc39a00700 1 -- 172.21.15.22:0/2689738431 shutdown_connections 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.740+0000 7fcc39a00700 1 -- 172.21.15.22:0/2689738431 wait complete. 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.741+0000 7fcc39a00700 1 Processor -- start 2024-03-24T06:29:06.806 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.741+0000 7fcc39a00700 1 -- start start 2024-03-24T06:29:06.807 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.742+0000 7fcc39a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:06.811 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.742+0000 7fcc39a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3419be00 con 0x7fcc34107b10 2024-03-24T06:29:06.811 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.742+0000 7fcc33e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:06.811 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.742+0000 7fcc33e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40042/0 (socket says 172.21.15.22:40042) 2024-03-24T06:29:06.811 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.742+0000 7fcc33e00700 1 -- 172.21.15.22:0/424436481 learned_addr learned my addr 172.21.15.22:0/424436481 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc33e00700 1 -- 172.21.15.22:0/424436481 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc24009550 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc33e00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcc24011740 tx=0x7fcc24011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc240039e0 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc24003b40 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc24024740 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3419c000 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.743+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc3419c4a0 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.745+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fcc2401ab30 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.745+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3410bf40 con 0x7fcc34107b10 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.745+0000 7fcc30c00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 0x7fcc1803b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:06.812 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.745+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc2404d390 con 0x7fcc34107b10 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.746+0000 7fcc33400700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 0x7fcc1803b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.747+0000 7fcc33400700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 0x7fcc1803b4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc20006fd0 tx=0x7fcc20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:05.751+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcc24010380 con 0x7fcc34107b10 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.103+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fcc3419ed80 con 0x7fcc34107b10 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.107+0000 7fcc30c00700 1 -- 172.21.15.22:0/424436481 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fcc24018be0 con 0x7fcc34107b10 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 msgr2=0x7fcc1803b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 0x7fcc1803b4f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc20006fd0 tx=0x7fcc20006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 msgr2=0x7fcc3419b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcc24011740 tx=0x7fcc24011b00 comp rx=0 tx=0).stop 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 shutdown_connections 2024-03-24T06:29:06.813 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcc18039030 0x7fcc1803b4f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 --2- 172.21.15.22:0/424436481 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcc34107b10 0x7fcc3419b8c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 >> 172.21.15.22:0/424436481 conn(0x7fcc34103150 msgr2=0x7fcc34103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 shutdown_connections 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T06:29:06.110+0000 7fcc39a00700 1 -- 172.21.15.22:0/424436481 wait complete. 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T06:29:06.814 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: ceph telemetry on 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout:For more information see: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:06.815 INFO:teuthology.orchestra.run.smithi022.stdout:Bootstrap complete. 2024-03-24T06:29:06.927 INFO:tasks.cephadm:Fetching config... 2024-03-24T06:29:06.927 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:29:06.927 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T06:29:06.982 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T06:29:06.982 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:29:06.982 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T06:29:07.025 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T06:29:07.025 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:29:07.025 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T06:29:07.146 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T06:29:07.146 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:29:07.146 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T06:29:07.197 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T06:29:07.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T06:29:07.378 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:07 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/424436481' entity='client.admin' 2024-03-24T06:29:07.387 INFO:teuthology.orchestra.run.smithi022.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:07.453 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T06:29:07.498 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:07.513 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T06:29:07.558 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT3jfj9qXCGFR1amNwa4q+ZLyv/Cs9ssiiH8RY3w0uKz1sEMkzlT0vsHVMR7glwsXGCZp9alNrpbcFB1gD2JCxt3qSY4n5ir2b7PUbq91Q2FFgkiQWI46xnB8UQ+cU2twAf14UMHz+j6MKYfLI+Rmtkeoa+jYKgzav2Lg85hbNnBEfnPOUfZS1rtrYPQJDNZfai8A5Di2bV6xfz5BzEgnZvYvD4ath4IlkyVIaMC6uvV6lsPPVk1gDg1GvI/SSXSnlBWdT1MPAYyXG361rnlWQJzmZ2quqkrdrbaCVsar3/qm6JWT5Tpg9o5KI2f04YA6O116Kxp0VPMUJM6YA+IM9oqOv339J/SaUGf3MhQxr6bc4jzBxlYNOUHEMX42/ItxrfDn24xu0fkAx6th/WaPhZUUcgCCKrSd2w8ZCnOlh+kyE+AjkyPY0Zj0ZK/EuenFi72NcwVvZ9iRXDtJ3UCt7vFCPY9/uZqFbBAsNr67HW9OiSHlIrt2xv5kqfoFq4jE= ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:07.573 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T06:29:08.002 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:10.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 -- 172.21.15.22:0/1252701082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 msgr2=0x7f6648108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 --2- 172.21.15.22:0/1252701082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f6648108830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6638009910 tx=0x7f6638009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 -- 172.21.15.22:0/1252701082 shutdown_connections 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 --2- 172.21.15.22:0/1252701082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f6648108830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 -- 172.21.15.22:0/1252701082 >> 172.21.15.22:0/1252701082 conn(0x7f66480772f0 msgr2=0x7f6648079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 -- 172.21.15.22:0/1252701082 shutdown_connections 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.906+0000 7f664cc00700 1 -- 172.21.15.22:0/1252701082 wait complete. 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.908+0000 7f664cc00700 1 Processor -- start 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.908+0000 7f664cc00700 1 -- start start 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.908+0000 7f664cc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.908+0000 7f664cc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f664819dc60 con 0x7f6648108410 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.909+0000 7f6645e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.909+0000 7f6645e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40066/0 (socket says 172.21.15.22:40066) 2024-03-24T06:29:10.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.909+0000 7f6645e00700 1 -- 172.21.15.22:0/929222383 learned_addr learned my addr 172.21.15.22:0/929222383 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:10.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.909+0000 7f6645e00700 1 -- 172.21.15.22:0/929222383 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66380095f0 con 0x7f6648108410 2024-03-24T06:29:10.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.909+0000 7f6645e00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6638005950 tx=0x7f6638004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:10.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.910+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663801d070 con 0x7f6648108410 2024-03-24T06:29:10.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.910+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6638022470 con 0x7f6648108410 2024-03-24T06:29:10.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.910+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f663800f5f0 con 0x7f6648108410 2024-03-24T06:29:10.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.910+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664819de60 con 0x7f6648108410 2024-03-24T06:29:10.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.910+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664819e300 con 0x7f6648108410 2024-03-24T06:29:10.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.911+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6648197060 con 0x7f6648108410 2024-03-24T06:29:10.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.911+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f663800ba90 con 0x7f6648108410 2024-03-24T06:29:10.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.912+0000 7f663ea00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 0x7f662c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:10.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.912+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f663804d190 con 0x7f6648108410 2024-03-24T06:29:10.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.912+0000 7f6645400700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 0x7f662c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:10.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.913+0000 7f6645400700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 0x7f662c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6634006fd0 tx=0x7f6634006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:10.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:10.917+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66380225e0 con 0x7f6648108410 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:11.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:11.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.147+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f6648063830 con 0x7f6648108410 2024-03-24T06:29:11.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.152+0000 7f663ea00700 1 -- 172.21.15.22:0/929222383 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f6638027070 con 0x7f6648108410 2024-03-24T06:29:11.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.156+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 msgr2=0x7f662c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:11.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.156+0000 7f664cc00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 0x7f662c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6634006fd0 tx=0x7f6634006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:11.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.156+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 msgr2=0x7f664819d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:11.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.156+0000 7f664cc00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6638005950 tx=0x7f6638004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 shutdown_connections 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f662c038fe0 0x7f662c03b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 --2- 172.21.15.22:0/929222383 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6648108410 0x7f664819d720 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 >> 172.21.15.22:0/929222383 conn(0x7f66480772f0 msgr2=0x7f6648077d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 shutdown_connections 2024-03-24T06:29:11.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:11.157+0000 7f664cc00700 1 -- 172.21.15.22:0/929222383 wait complete. 2024-03-24T06:29:11.977 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T06:29:11.978 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T06:29:12.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/929222383' entity='client.admin' 2024-03-24T06:29:12.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:12.178 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:12.895 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:14.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.654+0000 7febdb200700 1 -- 172.21.15.22:0/334444386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 msgr2=0x7febd4074690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.654+0000 7febdb200700 1 --2- 172.21.15.22:0/334444386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4074690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7febc400b0a0 tx=0x7febc400b3b0 comp rx=0 tx=0).stop 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.655+0000 7febdb200700 1 -- 172.21.15.22:0/334444386 shutdown_connections 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.655+0000 7febdb200700 1 --2- 172.21.15.22:0/334444386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4074690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.655+0000 7febdb200700 1 -- 172.21.15.22:0/334444386 >> 172.21.15.22:0/334444386 conn(0x7febd40fd1d0 msgr2=0x7febd40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.655+0000 7febdb200700 1 -- 172.21.15.22:0/334444386 shutdown_connections 2024-03-24T06:29:14.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.655+0000 7febdb200700 1 -- 172.21.15.22:0/334444386 wait complete. 2024-03-24T06:29:14.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.656+0000 7febdb200700 1 Processor -- start 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febdb200700 1 -- start start 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febdb200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febdb200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7febd4199870 con 0x7febd4076230 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febd3e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febd3e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40086/0 (socket says 172.21.15.22:40086) 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.657+0000 7febd3e00700 1 -- 172.21.15.22:0/514112229 learned_addr learned my addr 172.21.15.22:0/514112229 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:14.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febd3e00700 1 -- 172.21.15.22:0/514112229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc4009d00 con 0x7febd4076230 2024-03-24T06:29:14.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febd3e00700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7febc40093e0 tx=0x7febc400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:14.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc4021070 con 0x7febd4076230 2024-03-24T06:29:14.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc4003c10 con 0x7febd4076230 2024-03-24T06:29:14.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd4199a70 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febc400e040 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.658+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd4199f10 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.659+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7febc4019070 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.660+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd4192ca0 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.660+0000 7febd1600700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 0x7febb403b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.660+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7febc404cb20 con 0x7febd4076230 2024-03-24T06:29:14.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.660+0000 7febc3400700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 0x7febb403b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:14.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.661+0000 7febc3400700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 0x7febb403b520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7febbc006fd0 tx=0x7febbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:14.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.666+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febc401c3d0 con 0x7febd4076230 2024-03-24T06:29:14.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.896+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7febd4062640 con 0x7febb4039060 2024-03-24T06:29:14.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.902+0000 7febd1600700 1 -- 172.21.15.22:0/514112229 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7febd4062640 con 0x7febb4039060 2024-03-24T06:29:14.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.905+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 msgr2=0x7febb403b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:14.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.905+0000 7febdb200700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 0x7febb403b520 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7febbc006fd0 tx=0x7febbc006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.905+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 msgr2=0x7febd4199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.905+0000 7febdb200700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7febc40093e0 tx=0x7febc400bd40 comp rx=0 tx=0).stop 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 shutdown_connections 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7febb4039060 0x7febb403b520 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 --2- 172.21.15.22:0/514112229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4076230 0x7febd4199330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 >> 172.21.15.22:0/514112229 conn(0x7febd40fd1d0 msgr2=0x7febd40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 shutdown_connections 2024-03-24T06:29:14.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:14.906+0000 7febdb200700 1 -- 172.21.15.22:0/514112229 wait complete. 2024-03-24T06:29:15.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:14 smithi022 ceph-mon[32003]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:15.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:15.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:15.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:15.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:15.673 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi143 2024-03-24T06:29:15.673 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:29:15.674 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.conf 2024-03-24T06:29:15.692 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:29:15.692 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:15.749 INFO:tasks.cephadm:Adding host smithi143 to orchestrator... 2024-03-24T06:29:15.749 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch host add smithi143 2024-03-24T06:29:16.179 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:16.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:15 smithi022 ceph-mon[32003]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:29:16.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:15 smithi022 ceph-mon[32003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:16.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:15 smithi022 ceph-mon[32003]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:17.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:16 smithi022 ceph-mon[32003]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 -- 172.21.15.22:0/985949268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 msgr2=0x7fedd8108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 --2- 172.21.15.22:0/985949268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd8108840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009910 tx=0x7fedc8009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 -- 172.21.15.22:0/985949268 shutdown_connections 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 --2- 172.21.15.22:0/985949268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd8108840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 -- 172.21.15.22:0/985949268 >> 172.21.15.22:0/985949268 conn(0x7fedd80779a0 msgr2=0x7fedd8079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 -- 172.21.15.22:0/985949268 shutdown_connections 2024-03-24T06:29:18.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.095+0000 7feddda00700 1 -- 172.21.15.22:0/985949268 wait complete. 2024-03-24T06:29:18.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.096+0000 7feddda00700 1 Processor -- start 2024-03-24T06:29:18.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.096+0000 7feddda00700 1 -- start start 2024-03-24T06:29:18.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7feddda00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:18.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7feddda00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81a3d20 con 0x7fedd8108460 2024-03-24T06:29:18.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7fedd6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:18.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7fedd6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35388/0 (socket says 172.21.15.22:35388) 2024-03-24T06:29:18.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7fedd6a00700 1 -- 172.21.15.22:0/3316374109 learned_addr learned my addr 172.21.15.22:0/3316374109 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:18.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.097+0000 7fedd6a00700 1 -- 172.21.15.22:0/3316374109 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc80095f0 con 0x7fedd8108460 2024-03-24T06:29:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7fedd6a00700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fedc8006010 tx=0x7fedc8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedc801d070 con 0x7fedd8108460 2024-03-24T06:29:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc800bc30 con 0x7fedd8108460 2024-03-24T06:29:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedc8003930 con 0x7fedd8108460 2024-03-24T06:29:18.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd81a3f20 con 0x7fedd8108460 2024-03-24T06:29:18.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.098+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81a4340 con 0x7fedd8108460 2024-03-24T06:29:18.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.099+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd8063830 con 0x7fedd8108460 2024-03-24T06:29:18.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.099+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fedc8003ae0 con 0x7fedd8108460 2024-03-24T06:29:18.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.100+0000 7fedcf400700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 0x7fedbc03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:18.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.100+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fedc804cda0 con 0x7fedd8108460 2024-03-24T06:29:18.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.100+0000 7fedd6000700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 0x7fedbc03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:18.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.101+0000 7fedd6000700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 0x7fedbc03b4a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006fd0 tx=0x7fedc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:18.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.105+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fedc8004390 con 0x7fedd8108460 2024-03-24T06:29:18.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:18.334+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}) -- 0x7fedd81127a0 con 0x7fedbc038fe0 2024-03-24T06:29:18.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:18 smithi022 ceph-mon[32003]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:18.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:18 smithi022 ceph-mon[32003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:18.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:18.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:18.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:19.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:19 smithi022 ceph-mon[32003]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:20.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:20 smithi022 ceph-mon[32003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:20.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:20 smithi022 ceph-mon[32003]: Deploying cephadm binary to smithi143 2024-03-24T06:29:22.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:22 smithi022 ceph-mon[32003]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:23.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.193+0000 7fedcf400700 1 -- 172.21.15.22:0/3316374109 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fedd81127a0 con 0x7fedbc038fe0 2024-03-24T06:29:23.194 INFO:teuthology.orchestra.run.smithi022.stdout:Added host 'smithi143' with addr '172.21.15.143' 2024-03-24T06:29:23.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.197+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 msgr2=0x7fedbc03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.197+0000 7feddda00700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 0x7fedbc03b4a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedc4006fd0 tx=0x7fedc4006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 msgr2=0x7fedd81a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fedc8006010 tx=0x7fedc8004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 shutdown_connections 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fedbc038fe0 0x7fedbc03b4a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 --2- 172.21.15.22:0/3316374109 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fedd8108460 0x7fedd81a37e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 >> 172.21.15.22:0/3316374109 conn(0x7fedd80779a0 msgr2=0x7fedd810b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 shutdown_connections 2024-03-24T06:29:23.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:23.198+0000 7feddda00700 1 -- 172.21.15.22:0/3316374109 wait complete. 2024-03-24T06:29:23.944 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T06:29:24.398 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:24.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:24 smithi022 ceph-mon[32003]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:24.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:24.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:24 smithi022 ceph-mon[32003]: Added host smithi143 2024-03-24T06:29:24.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:24.541 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:25.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:25 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 -- 172.21.15.22:0/3001777974 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 msgr2=0x7fbb50104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 --2- 172.21.15.22:0/3001777974 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb50104440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009910 tx=0x7fbb40009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 -- 172.21.15.22:0/3001777974 shutdown_connections 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 --2- 172.21.15.22:0/3001777974 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb50104440 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 -- 172.21.15.22:0/3001777974 >> 172.21.15.22:0/3001777974 conn(0x7fbb500ff8f0 msgr2=0x7fbb50101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 -- 172.21.15.22:0/3001777974 shutdown_connections 2024-03-24T06:29:26.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.280+0000 7fbb55400700 1 -- 172.21.15.22:0/3001777974 wait complete. 2024-03-24T06:29:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.282+0000 7fbb55400700 1 Processor -- start 2024-03-24T06:29:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.282+0000 7fbb55400700 1 -- start start 2024-03-24T06:29:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.282+0000 7fbb55400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.282+0000 7fbb55400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5019f910 con 0x7fbb50104060 2024-03-24T06:29:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.283+0000 7fbb4e800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.283+0000 7fbb4e800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47812/0 (socket says 172.21.15.22:47812) 2024-03-24T06:29:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.283+0000 7fbb4e800700 1 -- 172.21.15.22:0/3535124999 learned_addr learned my addr 172.21.15.22:0/3535124999 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.283+0000 7fbb4e800700 1 -- 172.21.15.22:0/3535124999 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb400095f0 con 0x7fbb50104060 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb4e800700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbb40004f40 tx=0x7fbb40005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb4001d070 con 0x7fbb50104060 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb4000bc30 con 0x7fbb50104060 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5019fb10 con 0x7fbb50104060 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb40003970 con 0x7fbb50104060 2024-03-24T06:29:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.284+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5019ff30 con 0x7fbb50104060 2024-03-24T06:29:26.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.285+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fbb40003c30 con 0x7fbb50104060 2024-03-24T06:29:26.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.285+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50063830 con 0x7fbb50104060 2024-03-24T06:29:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.286+0000 7fbb47400700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 0x7fbb3403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.286+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbb4004ce40 con 0x7fbb50104060 2024-03-24T06:29:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.286+0000 7fbb4de00700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 0x7fbb3403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.287+0000 7fbb4de00700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 0x7fbb3403b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c006fd0 tx=0x7fbb3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:26.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.292+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbb40005c00 con 0x7fbb50104060 2024-03-24T06:29:26.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:26 smithi022 ceph-mon[32003]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:26.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:26.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.523+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb5010e300 con 0x7fbb34038f80 2024-03-24T06:29:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.524+0000 7fbb47400700 1 -- 172.21.15.22:0/3535124999 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fbb5010e300 con 0x7fbb34038f80 2024-03-24T06:29:26.524 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:26.524 INFO:teuthology.orchestra.run.smithi022.stdout:[{"addr": "172.21.15.22", "hostname": "smithi022", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}] 2024-03-24T06:29:26.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 msgr2=0x7fbb3403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:26.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 0x7fbb3403b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c006fd0 tx=0x7fbb3c006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:26.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 msgr2=0x7fbb5019f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbb40004f40 tx=0x7fbb40005020 comp rx=0 tx=0).stop 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 shutdown_connections 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbb34038f80 0x7fbb3403b440 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 --2- 172.21.15.22:0/3535124999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb50104060 0x7fbb5019f3d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 >> 172.21.15.22:0/3535124999 conn(0x7fbb500ff8f0 msgr2=0x7fbb50105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.527+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 shutdown_connections 2024-03-24T06:29:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:26.528+0000 7fbb55400700 1 -- 172.21.15.22:0/3535124999 wait complete. 2024-03-24T06:29:27.227 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-03-24T06:29:27.227 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:29:27.227 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-03-24T06:29:27.245 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:29:27.245 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:27.303 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-03-24T06:29:27.303 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch host add smithi155 2024-03-24T06:29:27.403 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:27 smithi022 ceph-mon[32003]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:27.771 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:28.482 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:28 smithi022 ceph-mon[32003]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:29.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 -- 172.21.15.22:0/2104902358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 msgr2=0x7f3d64074a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 --2- 172.21.15.22:0/2104902358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64074a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009910 tx=0x7f3d54009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 -- 172.21.15.22:0/2104902358 shutdown_connections 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 --2- 172.21.15.22:0/2104902358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64074a10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 -- 172.21.15.22:0/2104902358 >> 172.21.15.22:0/2104902358 conn(0x7f3d640fd530 msgr2=0x7f3d640ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 -- 172.21.15.22:0/2104902358 shutdown_connections 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.582+0000 7f3d63e00700 1 -- 172.21.15.22:0/2104902358 wait complete. 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d63e00700 1 Processor -- start 2024-03-24T06:29:29.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d63e00700 1 -- start start 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d63e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d63e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64199b60 con 0x7f3d64074630 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d62a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d62a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47830/0 (socket says 172.21.15.22:47830) 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.584+0000 7f3d62a00700 1 -- 172.21.15.22:0/201899189 learned_addr learned my addr 172.21.15.22:0/201899189 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.585+0000 7f3d62a00700 1 -- 172.21.15.22:0/201899189 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d540095f0 con 0x7f3d64074630 2024-03-24T06:29:29.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.585+0000 7f3d62a00700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006010 tx=0x7f3d54004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.585+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5401d070 con 0x7f3d64074630 2024-03-24T06:29:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.586+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5400bc30 con 0x7f3d64074630 2024-03-24T06:29:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.586+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d540039a0 con 0x7f3d64074630 2024-03-24T06:29:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.586+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d64199d60 con 0x7f3d64074630 2024-03-24T06:29:29.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.586+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6419a180 con 0x7f3d64074630 2024-03-24T06:29:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.587+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f3d54004390 con 0x7f3d64074630 2024-03-24T06:29:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.587+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d64063830 con 0x7f3d64074630 2024-03-24T06:29:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.587+0000 7f3d5b400700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 0x7f3d4803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.587+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3d5404cec0 con 0x7f3d64074630 2024-03-24T06:29:29.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.588+0000 7f3d62000700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 0x7f3d4803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:29.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.588+0000 7f3d62000700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 0x7f3d4803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006fd0 tx=0x7f3d50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:29.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.593+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3d5402ac00 con 0x7f3d64074630 2024-03-24T06:29:29.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:29.825+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) -- 0x7f3d6410e1b0 con 0x7f3d48038f80 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:30.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:30.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:29:30.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:30.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:30 smithi022 ceph-mon[32003]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:31.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:31 smithi022 ceph-mon[32003]: Deploying cephadm binary to smithi155 2024-03-24T06:29:33.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:32 smithi022 ceph-mon[32003]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:34.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.802+0000 7f3d5b400700 1 -- 172.21.15.22:0/201899189 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3d6410e1b0 con 0x7f3d48038f80 2024-03-24T06:29:34.803 INFO:teuthology.orchestra.run.smithi022.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-03-24T06:29:34.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 msgr2=0x7f3d4803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 0x7f3d4803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006fd0 tx=0x7f3d50006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 msgr2=0x7f3d64199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006010 tx=0x7f3d54004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 shutdown_connections 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3d48038f80 0x7f3d4803b440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 --2- 172.21.15.22:0/201899189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3d64074630 0x7f3d64199620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 >> 172.21.15.22:0/201899189 conn(0x7f3d640fd530 msgr2=0x7f3d64105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 shutdown_connections 2024-03-24T06:29:34.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:34.806+0000 7f3d63e00700 1 -- 172.21.15.22:0/201899189 wait complete. 2024-03-24T06:29:35.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:34 smithi022 ceph-mon[32003]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:35.457 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T06:29:35.924 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:36.068 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:36.068 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:35 smithi022 ceph-mon[32003]: Added host smithi155 2024-03-24T06:29:36.068 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:36.068 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:37.103 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:36 smithi022 ceph-mon[32003]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:37.103 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:38.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 -- 172.21.15.22:0/2425986812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 msgr2=0x7fb82c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 --2- 172.21.15.22:0/2425986812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c108840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb814009910 tx=0x7fb814009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 -- 172.21.15.22:0/2425986812 shutdown_connections 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 --2- 172.21.15.22:0/2425986812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c108840 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 -- 172.21.15.22:0/2425986812 >> 172.21.15.22:0/2425986812 conn(0x7fb82c0779a0 msgr2=0x7fb82c079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 -- 172.21.15.22:0/2425986812 shutdown_connections 2024-03-24T06:29:38.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.016+0000 7fb830c00700 1 -- 172.21.15.22:0/2425986812 wait complete. 2024-03-24T06:29:38.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.017+0000 7fb830c00700 1 Processor -- start 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb830c00700 1 -- start start 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb830c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb830c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb82c1a3d20 con 0x7fb82c108460 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb829e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb829e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35044/0 (socket says 172.21.15.22:35044) 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.018+0000 7fb829e00700 1 -- 172.21.15.22:0/977847102 learned_addr learned my addr 172.21.15.22:0/977847102 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:38.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb829e00700 1 -- 172.21.15.22:0/977847102 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8140095f0 con 0x7fb82c108460 2024-03-24T06:29:38.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb829e00700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb814006010 tx=0x7fb814004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:38.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb81401d070 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb81400bc30 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb814003930 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.019+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82c1a3f20 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.020+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb82c1a4340 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.021+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82c063830 con 0x7fb82c108460 2024-03-24T06:29:38.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.021+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fb814003ae0 con 0x7fb82c108460 2024-03-24T06:29:38.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.021+0000 7fb823400700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 0x7fb81003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:38.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.021+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb81404cdc0 con 0x7fb82c108460 2024-03-24T06:29:38.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.022+0000 7fb821400700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 0x7fb81003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:38.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.022+0000 7fb821400700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 0x7fb81003b4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006fd0 tx=0x7fb81c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:38.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.027+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb81400b6f0 con 0x7fb82c108460 2024-03-24T06:29:38.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.258+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb82c112700 con 0x7fb810038fe0 2024-03-24T06:29:38.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.260+0000 7fb823400700 1 -- 172.21.15.22:0/977847102 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fb82c112700 con 0x7fb810038fe0 2024-03-24T06:29:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:29:38.261 INFO:teuthology.orchestra.run.smithi022.stdout:[{"addr": "172.21.15.22", "hostname": "smithi022", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-03-24T06:29:38.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 msgr2=0x7fb81003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:38.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 0x7fb81003b4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006fd0 tx=0x7fb81c006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:38.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 msgr2=0x7fb82c1a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:38.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb814006010 tx=0x7fb814004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:38.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 shutdown_connections 2024-03-24T06:29:38.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb810038fe0 0x7fb81003b4a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:38.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 --2- 172.21.15.22:0/977847102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb82c108460 0x7fb82c1a37e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:38.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 >> 172.21.15.22:0/977847102 conn(0x7fb82c0779a0 msgr2=0x7fb82c10b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:38.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 shutdown_connections 2024-03-24T06:29:38.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:38.264+0000 7fb830c00700 1 -- 172.21.15.22:0/977847102 wait complete. 2024-03-24T06:29:38.630 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:38 smithi022 ceph-mon[32003]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:38.630 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:38 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:39.030 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T06:29:39.031 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T06:29:39.489 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:29:39.760 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:39 smithi022 ceph-mon[32003]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:40.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:40 smithi022 ceph-mon[32003]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 -- 172.21.15.22:0/951753402 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 msgr2=0x7f605c104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 --2- 172.21.15.22:0/951753402 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c104440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f605000b0a0 tx=0x7f605000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 -- 172.21.15.22:0/951753402 shutdown_connections 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 --2- 172.21.15.22:0/951753402 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c104440 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 -- 172.21.15.22:0/951753402 >> 172.21.15.22:0/951753402 conn(0x7f605c0ff8f0 msgr2=0x7f605c101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 -- 172.21.15.22:0/951753402 shutdown_connections 2024-03-24T06:29:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.474+0000 7f6064200700 1 -- 172.21.15.22:0/951753402 wait complete. 2024-03-24T06:29:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.475+0000 7f6064200700 1 Processor -- start 2024-03-24T06:29:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6064200700 1 -- start start 2024-03-24T06:29:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6064200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6064200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f605c19f910 con 0x7f605c104060 2024-03-24T06:29:41.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6061800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:41.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6061800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35062/0 (socket says 172.21.15.22:35062) 2024-03-24T06:29:41.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.476+0000 7f6061800700 1 -- 172.21.15.22:0/2508220283 learned_addr learned my addr 172.21.15.22:0/2508220283 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:29:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.477+0000 7f6061800700 1 -- 172.21.15.22:0/2508220283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6050009d00 con 0x7f605c104060 2024-03-24T06:29:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.477+0000 7f6061800700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6050000f80 tx=0x7f605000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.477+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f605000e040 con 0x7f605c104060 2024-03-24T06:29:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.478+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6050003c10 con 0x7f605c104060 2024-03-24T06:29:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.478+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c19fb10 con 0x7f605c104060 2024-03-24T06:29:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.478+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f605001d600 con 0x7f605c104060 2024-03-24T06:29:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.478+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c19ff30 con 0x7f605c104060 2024-03-24T06:29:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.479+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c063830 con 0x7f605c104060 2024-03-24T06:29:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.479+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f6050019070 con 0x7f605c104060 2024-03-24T06:29:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.480+0000 7f605a000700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 0x7f604403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.480+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f605004d160 con 0x7f605c104060 2024-03-24T06:29:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.480+0000 7f6060e00700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 0x7f604403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.481+0000 7f6060e00700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 0x7f604403b490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6048006fd0 tx=0x7f6048006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.485+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f605001c3d0 con 0x7f605c104060 2024-03-24T06:29:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.722+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f605c10e290 con 0x7f605c104060 2024-03-24T06:29:41.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.789+0000 7f605a000700 1 -- 172.21.15.22:0/2508220283 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f6050017070 con 0x7f605c104060 2024-03-24T06:29:41.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.792+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 msgr2=0x7f604403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.792+0000 7f6064200700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 0x7f604403b490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6048006fd0 tx=0x7f6048006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.792+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 msgr2=0x7f605c19f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.792+0000 7f6064200700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6050000f80 tx=0x7f605000bd40 comp rx=0 tx=0).stop 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 shutdown_connections 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6044038fd0 0x7f604403b490 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 --2- 172.21.15.22:0/2508220283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f605c104060 0x7f605c19f3d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:41.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 >> 172.21.15.22:0/2508220283 conn(0x7f605c0ff8f0 msgr2=0x7f605c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:41.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 shutdown_connections 2024-03-24T06:29:41.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:29:41.793+0000 7f6064200700 1 -- 172.21.15.22:0/2508220283 wait complete. 2024-03-24T06:29:41.795 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:42.129 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:42.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:29:42.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:41 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T06:29:42.501 INFO:tasks.cephadm:Adding mon.a on smithi022 2024-03-24T06:29:42.501 INFO:tasks.cephadm:Adding mon.b on smithi143 2024-03-24T06:29:42.501 INFO:tasks.cephadm:Adding mon.c on smithi155 2024-03-24T06:29:42.502 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c' 2024-03-24T06:29:42.736 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:43.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:43.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 -- 172.21.15.155:0/4063828599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 msgr2=0x7fdc8c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/4063828599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c102210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b0a0 tx=0x7fdc7c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 -- 172.21.15.155:0/4063828599 shutdown_connections 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/4063828599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c102210 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 -- 172.21.15.155:0/4063828599 >> 172.21.15.155:0/4063828599 conn(0x7fdc8c0fd6e0 msgr2=0x7fdc8c0ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:44.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 -- 172.21.15.155:0/4063828599 shutdown_connections 2024-03-24T06:29:44.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.195+0000 7fdc93c5f700 1 -- 172.21.15.155:0/4063828599 wait complete. 2024-03-24T06:29:44.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.196+0000 7fdc93c5f700 1 Processor -- start 2024-03-24T06:29:44.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.196+0000 7fdc93c5f700 1 -- start start 2024-03-24T06:29:44.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.196+0000 7fdc93c5f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:44.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.196+0000 7fdc93c5f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c197ae0 con 0x7fdc8c101e30 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.197+0000 7fdc919fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.197+0000 7fdc919fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:42608/0 (socket says 172.21.15.155:42608) 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.197+0000 7fdc919fb700 1 -- 172.21.15.155:0/1247960614 learned_addr learned my addr 172.21.15.155:0/1247960614 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.197+0000 7fdc919fb700 1 -- 172.21.15.155:0/1247960614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c009d00 con 0x7fdc8c101e30 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc919fb700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b830 tx=0x7fdc7c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc7c00e040 con 0x7fdc8c101e30 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8c197ce0 con 0x7fdc8c101e30 2024-03-24T06:29:44.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8c198100 con 0x7fdc8c101e30 2024-03-24T06:29:44.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc7c003c10 con 0x7fdc8c101e30 2024-03-24T06:29:44.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.198+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc7c01d5e0 con 0x7fdc8c101e30 2024-03-24T06:29:44.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.199+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc8c103310 con 0x7fdc8c101e30 2024-03-24T06:29:44.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.200+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fdc7c019070 con 0x7fdc8c101e30 2024-03-24T06:29:44.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.200+0000 7fdc8affd700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 0x7fdc7403b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:44.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.201+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdc7c01cac0 con 0x7fdc8c101e30 2024-03-24T06:29:44.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.201+0000 7fdc911fa700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 0x7fdc7403b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:44.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.202+0000 7fdc911fa700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 0x7fdc7403b3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdc80006fd0 tx=0x7fdc80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:44.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.217+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdc7c004040 con 0x7fdc8c101e30 2024-03-24T06:29:44.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.423+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}) -- 0x7fdc8c10c190 con 0x7fdc74038f30 2024-03-24T06:29:44.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.439+0000 7fdc8affd700 1 -- 172.21.15.155:0/1247960614 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdc8c10c190 con 0x7fdc74038f30 2024-03-24T06:29:44.439 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 msgr2=0x7fdc7403b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 0x7fdc7403b3f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdc80006fd0 tx=0x7fdc80006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 msgr2=0x7fdc8c1975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b830 tx=0x7fdc7c00bd40 comp rx=0 tx=0).stop 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 shutdown_connections 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdc74038f30 0x7fdc7403b3f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 --2- 172.21.15.155:0/1247960614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc8c101e30 0x7fdc8c1975a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 >> 172.21.15.155:0/1247960614 conn(0x7fdc8c0fd6e0 msgr2=0x7fdc8c103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 shutdown_connections 2024-03-24T06:29:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:44.440+0000 7fdc93c5f700 1 -- 172.21.15.155:0/1247960614 wait complete. 2024-03-24T06:29:45.064 DEBUG:teuthology.orchestra.run.smithi143:mon.b> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T06:29:45.067 DEBUG:teuthology.orchestra.run.smithi155:mon.c> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T06:29:45.070 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T06:29:45.070 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T06:29:45.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:45.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:45.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:45.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:45.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:45.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:45.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:45.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:46.030 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:46.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:45 smithi022 ceph-mon[32003]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:46.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:45 smithi022 ceph-mon[32003]: Saving service mon spec with placement smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-03-24T06:29:46.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:45 smithi022 ceph-mon[32003]: Deploying daemon mon.c on smithi155 2024-03-24T06:29:47.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:46 smithi022 ceph-mon[32003]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:48.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- 172.21.15.155:0/3061171699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 msgr2=0x7fe768103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:48.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3061171699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe768103160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe764009910 tx=0x7fe764009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- 172.21.15.155:0/3061171699 shutdown_connections 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3061171699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe768103160 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- 172.21.15.155:0/3061171699 >> 172.21.15.155:0/3061171699 conn(0x7fe7680fc5b0 msgr2=0x7fe7680fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- 172.21.15.155:0/3061171699 shutdown_connections 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- 172.21.15.155:0/3061171699 wait complete. 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 Processor -- start 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.422+0000 7fe76fddb700 1 -- start start 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.423+0000 7fe76fddb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:48.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.423+0000 7fe76fddb700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe7681a3ce0 con 0x7fe768102d80 2024-03-24T06:29:48.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.423+0000 7fe76db77700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:48.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.423+0000 7fe76db77700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:42616/0 (socket says 172.21.15.155:42616) 2024-03-24T06:29:48.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.423+0000 7fe76db77700 1 -- 172.21.15.155:0/3019365516 learned_addr learned my addr 172.21.15.155:0/3019365516 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:29:48.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.424+0000 7fe76db77700 1 -- 172.21.15.155:0/3019365516 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7640095f0 con 0x7fe768102d80 2024-03-24T06:29:48.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.424+0000 7fe76db77700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe764005850 tx=0x7fe764004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:48.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.425+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe76401d070 con 0x7fe768102d80 2024-03-24T06:29:48.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.425+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7681a3ee0 con 0x7fe768102d80 2024-03-24T06:29:48.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.425+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7681a4300 con 0x7fe768102d80 2024-03-24T06:29:48.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.425+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76400bc30 con 0x7fe768102d80 2024-03-24T06:29:48.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.425+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe764003a20 con 0x7fe768102d80 2024-03-24T06:29:48.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.426+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c005330 con 0x7fe768102d80 2024-03-24T06:29:48.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.426+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fe764004390 con 0x7fe768102d80 2024-03-24T06:29:48.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.427+0000 7fe75effd700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 0x7fe75403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:48.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.427+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe76404d1e0 con 0x7fe768102d80 2024-03-24T06:29:48.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.427+0000 7fe76d376700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 0x7fe75403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:48.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.428+0000 7fe76d376700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 0x7fe75403b4a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe758006fd0 tx=0x7fe758006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:48.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.441+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe764003b80 con 0x7fe768102d80 2024-03-24T06:29:48.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.769+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe74c0055e0 con 0x7fe768102d80 2024-03-24T06:29:48.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.770+0000 7fe75effd700 1 -- 172.21.15.155:0/3019365516 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fe764026030 con 0x7fe768102d80 2024-03-24T06:29:48.770 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:29:48.770 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","modified":"2024-03-24T06:27:43.107757Z","created":"2024-03-24T06:27:43.107757Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T06:29:48.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 msgr2=0x7fe75403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:48.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 0x7fe75403b4a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe758006fd0 tx=0x7fe758006e40 comp rx=0 tx=0).stop 2024-03-24T06:29:48.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 msgr2=0x7fe7681a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe764005850 tx=0x7fe764004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 shutdown_connections 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe754038fe0 0x7fe75403b4a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 --2- 172.21.15.155:0/3019365516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe768102d80 0x7fe7681a37a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.772+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 >> 172.21.15.155:0/3019365516 conn(0x7fe7680fc5b0 msgr2=0x7fe7681023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.773+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 shutdown_connections 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:48.773+0000 7fe76fddb700 1 -- 172.21.15.155:0/3019365516 wait complete. 2024-03-24T06:29:48.773 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T06:29:49.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:48 smithi155 systemd[1]: Starting Ceph mon.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:29:49.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:48 smithi022 ceph-mon[32003]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:48 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/3019365516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:49.654 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 podman[36242]: 2024-03-24 06:29:49.519659887 +0000 UTC m=+0.017850566 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:29:49.654 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 podman[36242]: 2024-03-24 06:29:49.655345487 +0000 UTC m=+0.153536198 container create 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T06:29:49.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 podman[36242]: 2024-03-24 06:29:49.849175291 +0000 UTC m=+0.347365961 container init 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T06:29:49.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 podman[36242]: 2024-03-24 06:29:49.852348846 +0000 UTC m=+0.350539516 container start 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T06:29:49.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T06:29:49.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev), process ceph-mon, pid 2 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pidfile_write: ignore empty --pid-file 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: load: jerasure load: lrc 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: RocksDB version: 7.9.2 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Git sha 0 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Compile date 2024-03-23 03:28:38 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: DB SUMMARY 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: DB Session ID: 9T188MZZ68YTSC72E4L5 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: CURRENT file: CURRENT 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.error_if_exists: 0 2024-03-24T06:29:49.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.create_if_missing: 0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.paranoid_checks: 1 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.env: 0x561a4ecd3c20 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.info_log: 0x561a5001bac0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.statistics: (nil) 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.use_fsync: 0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_log_file_size: 0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T06:29:49.913 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_fallocate: 1 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.use_direct_reads: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.db_log_dir: 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.wal_dir: 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T06:29:49.914 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.write_buffer_manager: 0x561a4ffee1e0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.unordered_write: 0 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T06:29:49.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T06:29:49.916 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.row_cache: None 2024-03-24T06:29:49.916 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.wal_filter: None 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.two_write_queues: 0 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.wal_compression: 0 2024-03-24T06:29:49.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.atomic_flush: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.log_readahead_size: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_background_jobs: 2 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_background_compactions: -1 2024-03-24T06:29:49.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_subcompactions: 1 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_open_files: -1 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_background_flushes: -1 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Compression algorithms supported: 2024-03-24T06:29:49.919 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kZSTD supported: 0 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kXpressCompression supported: 0 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kZlibCompression supported: 1 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kSnappyCompression supported: 1 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kLZ4Compression supported: 1 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: kBZip2Compression supported: 0 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.merge_operator: 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_filter: None 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T06:29:49.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561a5001bbc0) 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks: 1 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_top_level_index_and_filter: 1 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_type: 0 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_index_type: 0 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_shortening: 1 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: checksum: 4 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: no_block_cache: 0 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache: 0x561a5001f090 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_name: BinnedLRUCache 2024-03-24T06:29:49.921 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_options: 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: capacity : 536870912 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_shard_bits : 4 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: strict_capacity_limit : 0 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: high_pri_pool_ratio: 0.000 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_compressed: (nil) 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: persistent_cache: (nil) 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size: 4096 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size_deviation: 10 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_restart_interval: 16 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_block_restart_interval: 1 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: metadata_block_size: 4096 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: partition_filters: 0 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: use_delta_encoding: 1 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: filter_policy: bloomfilter 2024-03-24T06:29:49.922 INFO:journalctl@ceph.mon.c.smithi155.stdout: whole_key_filtering: 1 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: verify_compression: 0 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: read_amp_bytes_per_bit: 0 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: format_version: 5 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_index_compression: 1 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_align: 0 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: max_auto_readahead_size: 262144 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: prepopulate_block_cache: 0 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: initial_auto_readahead_size: 8192 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression: NoCompression 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.num_levels: 7 2024-03-24T06:29:49.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T06:29:49.924 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T06:29:49.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T06:29:49.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.table_properties_collectors: 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.inplace_update_support: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.bloom_locality: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.max_successive_merges: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.ttl: 2592000 2024-03-24T06:29:49.927 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enable_blob_files: false 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.min_blob_size: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 032e8afe-860a-4480-8541-2245b31c16d8 2024-03-24T06:29:49.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261789882705, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261789883288, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711261789, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "032e8afe-860a-4480-8541-2245b31c16d8", "db_session_id": "9T188MZZ68YTSC72E4L5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261789883392, "job": 1, "event": "recovery_finished"} 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561a50110000 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: DB pointer 0x561a500fa000 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** DB Stats ** 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:29:49.929 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T06:29:49.930 INFO:journalctl@ceph.mon.c.smithi155.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache BinnedLRUCache@0x561a5001f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.931 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: using public_addr v2:172.21.15.155:0/0 -> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: starting mon.c rank -1 at public addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] at bind addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(???) e0 preinit fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: e1 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: legacy client fscid: -1 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout: No filesystems configured 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T06:29:49.932 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Deploying cephadm binary to smithi143 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Added host smithi143 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:49.933 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:49.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Deploying cephadm binary to smithi155 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Added host smithi155 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.935 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Saving service mon spec with placement smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: Deploying daemon mon.c on smithi155 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/3019365516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:49.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 ceph-mon[36262]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T06:29:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 bash[36242]: 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 2024-03-24T06:29:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:49 smithi155 systemd[1]: Started Ceph mon.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:29:50.423 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T06:29:50.423 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T06:29:50.656 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 -- 172.21.15.155:0/3662443298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 msgr2=0x7f36a4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3662443298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36a4005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f36b8069bd0 tx=0x7f36a800feb0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 -- 172.21.15.155:0/3662443298 shutdown_connections 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3662443298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36a4005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 -- 172.21.15.155:0/3662443298 >> 172.21.15.155:0/3662443298 conn(0x7f36b80fc6b0 msgr2=0x7f36b80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 -- 172.21.15.155:0/3662443298 shutdown_connections 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.019+0000 7f36be1c4700 1 -- 172.21.15.155:0/3662443298 wait complete. 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.020+0000 7f36be1c4700 1 Processor -- start 2024-03-24T06:29:55.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.020+0000 7f36be1c4700 1 -- start start 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36be1c4700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36b81976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36be1c4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36be1c4700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f36b81981b0 con 0x7f36a40035c0 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36be1c4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8198320 con 0x7f36b8197c20 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:45836/0 (socket says 172.21.15.155:45836) 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 -- 172.21.15.155:0/3495793244 learned_addr learned my addr 172.21.15.155:0/3495793244 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 -- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 msgr2=0x7f36b819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T06:29:55.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 -- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 msgr2=0x7f36b819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36aeffd700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.021+0000 7f36b77fe700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36b81976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.022+0000 7f36b77fe700 1 -- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 msgr2=0x7f36b819bfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.022+0000 7f36b77fe700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.022+0000 7f36b77fe700 1 -- 172.21.15.155:0/3495793244 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a800f900 con 0x7f36a40035c0 2024-03-24T06:29:55.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.022+0000 7f36b77fe700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36b81976e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f36a8010010 tx=0x7f36a80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:55.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.024+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f36a8004aa0 con 0x7f36a40035c0 2024-03-24T06:29:55.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.024+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36a8013e40 con 0x7f36a40035c0 2024-03-24T06:29:55.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.024+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b819c550 con 0x7f36a40035c0 2024-03-24T06:29:55.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.024+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f36a8011a80 con 0x7f36a40035c0 2024-03-24T06:29:55.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.025+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b819caa0 con 0x7f36a40035c0 2024-03-24T06:29:55.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.026+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b8191350 con 0x7f36a40035c0 2024-03-24T06:29:55.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.026+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f36a80041c0 con 0x7f36a40035c0 2024-03-24T06:29:55.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.027+0000 7f36b57fa700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 0x7f369803b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:55.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.027+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f36a804e8c0 con 0x7f36a40035c0 2024-03-24T06:29:55.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.027+0000 7f36aeffd700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 0x7f369803b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:55.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.028+0000 7f36aeffd700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 0x7f369803b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009180 tx=0x7f36a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:55.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.046+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36a8035ad0 con 0x7f36a40035c0 2024-03-24T06:29:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:29:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: mon.a calling monitor election 2024-03-24T06:29:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: mon.c calling monitor election 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: monmap epoch 2 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: last_changed 2024-03-24T06:29:49.919088+0000 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:29:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: min_mon_release 19 (squid) 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: election_strategy: 1 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: fsmap 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: mgrmap e13: x(active, since 60s) 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: overall HEALTH_OK 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:55.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:55.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.332+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f36b819cd70 con 0x7f36a40035c0 2024-03-24T06:29:55.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.333+0000 7f36b57fa700 1 -- 172.21.15.155:0/3495793244 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f36a8031020 con 0x7f36a40035c0 2024-03-24T06:29:55.333 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:29:55.333 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","modified":"2024-03-24T06:29:49.919088Z","created":"2024-03-24T06:27:43.107757Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T06:29:55.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.335+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 msgr2=0x7f369803b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:55.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 0x7f369803b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009180 tx=0x7f36a0008040 comp rx=0 tx=0).stop 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 msgr2=0x7f36b81976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36b81976e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f36a8010010 tx=0x7f36a80056d0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 shutdown_connections 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36a40035c0 0x7f36b81976e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3698038d70 0x7f369803b230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 --2- 172.21.15.155:0/3495793244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b8197c20 0x7f36b819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 >> 172.21.15.155:0/3495793244 conn(0x7f36b80fc6b0 msgr2=0x7f36b81074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 shutdown_connections 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:55.336+0000 7f36be1c4700 1 -- 172.21.15.155:0/3495793244 wait complete. 2024-03-24T06:29:55.336 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-03-24T06:29:55.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:29:55.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: mon.a calling monitor election 2024-03-24T06:29:55.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: mon.c calling monitor election 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: monmap epoch 2 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: last_changed 2024-03-24T06:29:49.919088+0000 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: min_mon_release 19 (squid) 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: election_strategy: 1 2024-03-24T06:29:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: fsmap 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: mgrmap e13: x(active, since 60s) 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: overall HEALTH_OK 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: Deploying daemon mon.b on smithi143 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/3495793244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: Deploying daemon mon.b on smithi143 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/3495793244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:56.929 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T06:29:56.930 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T06:29:57.173 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:29:58.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:29:58 smithi155 ceph-mon[36262]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:58.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:29:58 smithi022 ceph-mon[32003]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:58.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.519+0000 7fc43ab0f700 1 -- 172.21.15.155:0/4144962419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 msgr2=0x7fc434102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:58.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.519+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/4144962419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc434102320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc428009910 tx=0x7fc428009c20 comp rx=0 tx=0).stop 2024-03-24T06:29:58.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.519+0000 7fc43ab0f700 1 -- 172.21.15.155:0/4144962419 shutdown_connections 2024-03-24T06:29:58.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.519+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/4144962419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc434102320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- 172.21.15.155:0/4144962419 >> 172.21.15.155:0/4144962419 conn(0x7fc4340fd7d0 msgr2=0x7fc4340ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:58.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- 172.21.15.155:0/4144962419 shutdown_connections 2024-03-24T06:29:58.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- 172.21.15.155:0/4144962419 wait complete. 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 Processor -- start 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- start start 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc43419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc43419df20 con 0x7fc434101f40 2024-03-24T06:29:58.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.520+0000 7fc43ab0f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc43419e090 con 0x7fc43419d990 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:45850/0 (socket says 172.21.15.155:45850) 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 -- 172.21.15.155:0/3768158174 learned_addr learned my addr 172.21.15.155:0/3768158174 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 -- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 msgr2=0x7fc434197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 -- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 msgr2=0x7fc434197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc433fff700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.521+0000 7fc4388ab700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc43419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.522+0000 7fc4388ab700 1 -- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 msgr2=0x7fc434197610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.522+0000 7fc4388ab700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.522+0000 7fc4388ab700 1 -- 172.21.15.155:0/3768158174 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4280095f0 con 0x7fc434101f40 2024-03-24T06:29:58.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.522+0000 7fc4388ab700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc43419d450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc428014000 tx=0x7fc428004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:58.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.522+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc42801d070 con 0x7fc434101f40 2024-03-24T06:29:58.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.523+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc428005dc0 con 0x7fc434101f40 2024-03-24T06:29:58.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.523+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc434197bb0 con 0x7fc434101f40 2024-03-24T06:29:58.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.523+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc428003880 con 0x7fc434101f40 2024-03-24T06:29:58.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.523+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc434198100 con 0x7fc434101f40 2024-03-24T06:29:58.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.524+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434063830 con 0x7fc434101f40 2024-03-24T06:29:58.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.524+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fc42800b970 con 0x7fc434101f40 2024-03-24T06:29:58.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.525+0000 7fc431ffb700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 0x7fc42403b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:29:58.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.525+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc42804d2b0 con 0x7fc434101f40 2024-03-24T06:29:58.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.525+0000 7fc433fff700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 0x7fc42403b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:29:58.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.526+0000 7fc433fff700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 0x7fc42403b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc420009180 tx=0x7fc420008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:29:58.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.540+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc42800b6f0 con 0x7fc434101f40 2024-03-24T06:29:58.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.811+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc434198320 con 0x7fc434101f40 2024-03-24T06:29:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.812+0000 7fc431ffb700 1 -- 172.21.15.155:0/3768158174 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fc42800e3e0 con 0x7fc434101f40 2024-03-24T06:29:58.812 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:29:58.812 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","modified":"2024-03-24T06:29:49.919088Z","created":"2024-03-24T06:27:43.107757Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 msgr2=0x7fc42403b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 0x7fc42403b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc420009180 tx=0x7fc420008040 comp rx=0 tx=0).stop 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 msgr2=0x7fc43419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc43419d450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc428014000 tx=0x7fc428004ff0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 shutdown_connections 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc434101f40 0x7fc43419d450 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc424038d20 0x7fc42403b1e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 --2- 172.21.15.155:0/3768158174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43419d990 0x7fc434197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:29:58.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 >> 172.21.15.155:0/3768158174 conn(0x7fc4340fd7d0 msgr2=0x7fc434103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:29:58.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 shutdown_connections 2024-03-24T06:29:58.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:29:58.814+0000 7fc43ab0f700 1 -- 172.21.15.155:0/3768158174 wait complete. 2024-03-24T06:29:58.815 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-03-24T06:29:58.834 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Compile date 2024-03-23 03:28:38 2024-03-24T06:29:58.834 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: DB SUMMARY 2024-03-24T06:29:58.834 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: DB Session ID: NEW1Q0LX9O8QGPTSBMPO 2024-03-24T06:29:58.834 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: CURRENT file: CURRENT 2024-03-24T06:29:58.835 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T06:29:58.835 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T06:29:58.835 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T06:29:58.835 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-24T06:29:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.error_if_exists: 0 2024-03-24T06:29:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.create_if_missing: 0 2024-03-24T06:29:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.paranoid_checks: 1 2024-03-24T06:29:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.env: 0x55ca08e29c20 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.info_log: 0x55ca0ac67ac0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.statistics: (nil) 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.use_fsync: 0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_log_file_size: 0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T06:29:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_fallocate: 1 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.use_direct_reads: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.db_log_dir: 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.wal_dir: 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T06:29:59.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.write_buffer_manager: 0x55ca0ac3a1e0 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T06:29:59.181 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T06:29:59.182 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.unordered_write: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.row_cache: None 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.wal_filter: None 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.two_write_queues: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.wal_compression: 0 2024-03-24T06:29:59.183 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.atomic_flush: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.log_readahead_size: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_background_jobs: 2 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_background_compactions: -1 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_subcompactions: 1 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T06:29:59.184 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_open_files: -1 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_background_flushes: -1 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Compression algorithms supported: 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kZSTD supported: 0 2024-03-24T06:29:59.185 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kXpressCompression supported: 0 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kZlibCompression supported: 1 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kSnappyCompression supported: 1 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kLZ4Compression supported: 1 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: kBZip2Compression supported: 0 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.merge_operator: 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_filter: None 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T06:29:59.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ca0ac67bc0) 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: cache_index_and_filter_blocks: 1 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: pin_top_level_index_and_filter: 1 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: index_type: 0 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: data_block_index_type: 0 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: index_shortening: 1 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: checksum: 4 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: no_block_cache: 0 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_cache: 0x55ca0ac6b090 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_cache_name: BinnedLRUCache 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_cache_options: 2024-03-24T06:29:59.187 INFO:journalctl@ceph.mon.b.smithi143.stdout: capacity : 536870912 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: num_shard_bits : 4 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: strict_capacity_limit : 0 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: high_pri_pool_ratio: 0.000 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_cache_compressed: (nil) 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: persistent_cache: (nil) 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_size: 4096 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_size_deviation: 10 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_restart_interval: 16 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: index_block_restart_interval: 1 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: metadata_block_size: 4096 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: partition_filters: 0 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: use_delta_encoding: 1 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: filter_policy: bloomfilter 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: whole_key_filtering: 1 2024-03-24T06:29:59.188 INFO:journalctl@ceph.mon.b.smithi143.stdout: verify_compression: 0 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: read_amp_bytes_per_bit: 0 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: format_version: 5 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: enable_index_compression: 1 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: block_align: 0 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: max_auto_readahead_size: 262144 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: prepopulate_block_cache: 0 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: initial_auto_readahead_size: 8192 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression: NoCompression 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T06:29:59.189 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.num_levels: 7 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T06:29:59.190 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T06:29:59.191 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T06:29:59.192 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.table_properties_collectors: 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.inplace_update_support: 0 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T06:29:59.193 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.bloom_locality: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.max_successive_merges: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.ttl: 2592000 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enable_blob_files: false 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.min_blob_size: 0 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T06:29:59.194 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: adcde6fe-086e-4f25-baba-12924b2bcebd 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261798836568, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261798836995, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711261798, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "adcde6fe-086e-4f25-baba-12924b2bcebd", "db_session_id": "NEW1Q0LX9O8QGPTSBMPO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711261798837074, "job": 1, "event": "recovery_finished"} 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ca0ad5c000 2024-03-24T06:29:59.195 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: DB pointer 0x55ca0ad46000 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: ** DB Stats ** 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: ** Compaction Stats [default] ** 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T06:29:59.196 INFO:journalctl@ceph.mon.b.smithi143.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: ** Compaction Stats [default] ** 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T06:29:59.197 INFO:journalctl@ceph.mon.b.smithi143.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: Block cache BinnedLRUCache@0x55ca0ac6b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: using public_addr v2:172.21.15.143:0/0 -> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: starting mon.b rank -1 at public addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] at bind addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(???) e0 preinit fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T06:29:59.198 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: e1 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: legacy client fscid: -1 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout: No filesystems configured 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T06:29:59.199 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Deploying cephadm binary to smithi143 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Added host smithi143 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.200 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:59.201 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Deploying cephadm binary to smithi155 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Added host smithi155 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.202 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2508220283' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T06:29:59.203 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Saving service mon spec with placement smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Deploying daemon mon.c on smithi155 2024-03-24T06:29:59.204 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/3019365516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.a calling monitor election 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.c calling monitor election 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: monmap epoch 2 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:29:59.205 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: last_changed 2024-03-24T06:29:49.919088+0000 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: min_mon_release 19 (squid) 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: election_strategy: 1 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: fsmap 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mgrmap e13: x(active, since 60s) 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: overall HEALTH_OK 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:29:59.206 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: Deploying daemon mon.b on smithi143 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/3495793244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 ceph-mon[36520]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 bash[36501]: afc985d51a41c8a62779659b1615ee7ea1859dcaa2739318781069475ca51004 2024-03-24T06:29:59.207 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:29:58 smithi143 systemd[1]: Started Ceph mon.b for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:30:00.373 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T06:30:00.373 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T06:30:00.627 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:30:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: mon.c calling monitor election 2024-03-24T06:30:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:30:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: mon.a calling monitor election 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: monmap epoch 3 2024-03-24T06:30:04.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: min_mon_release 19 (squid) 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: election_strategy: 1 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: fsmap 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: mgrmap e13: x(active, since 68s) 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: overall HEALTH_OK 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:04.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: mon.c calling monitor election 2024-03-24T06:30:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:30:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: mon.a calling monitor election 2024-03-24T06:30:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: monmap epoch 3 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: min_mon_release 19 (squid) 2024-03-24T06:30:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: election_strategy: 1 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: fsmap 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: mgrmap e13: x(active, since 68s) 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: overall HEALTH_OK 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:04.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:04.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 -- 172.21.15.155:0/2367428528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8340035c0 msgr2=0x7fb834005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:04.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 --2- 172.21.15.155:0/2367428528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8340035c0 0x7fb834005a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb84c06a460 tx=0x7fb844009e80 comp rx=0 tx=0).stop 2024-03-24T06:30:04.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 -- 172.21.15.155:0/2367428528 shutdown_connections 2024-03-24T06:30:04.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 --2- 172.21.15.155:0/2367428528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8340035c0 0x7fb834005a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:04.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 -- 172.21.15.155:0/2367428528 >> 172.21.15.155:0/2367428528 conn(0x7fb84c0fc710 msgr2=0x7fb84c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:04.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 -- 172.21.15.155:0/2367428528 shutdown_connections 2024-03-24T06:30:04.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.990+0000 7fb850d8a700 1 -- 172.21.15.155:0/2367428528 wait complete. 2024-03-24T06:30:04.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 Processor -- start 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 -- start start 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb84c1010c0 0x7fb84c101540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb84c1a3940 0x7fb84c1a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c1a6330 con 0x7fb84c1010c0 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c1a64a0 con 0x7fb84c1a3940 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb850d8a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb84c1a6660 con 0x7fb8340035c0 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb84a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb84a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:57538/0 (socket says 172.21.15.155:57538) 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb84a59c700 1 -- 172.21.15.155:0/776303965 learned_addr learned my addr 172.21.15.155:0/776303965 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:30:04.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.991+0000 7fb849d9b700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb84c1010c0 0x7fb84c101540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 -- 172.21.15.155:0/776303965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb84c1a3940 msgr2=0x7fb84c1a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84ad9d700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb84c1a3940 0x7fb84c1a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb84c1a3940 0x7fb84c1a5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 -- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb84c1010c0 msgr2=0x7fb84c101540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb84c1010c0 0x7fb84c101540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 -- 172.21.15.155:0/776303965 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8440098d0 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb84a59c700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb844017720 tx=0x7fb844017800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844004d80 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb844021b40 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84c1a68e0 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844005870 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.992+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84c1a6e30 con 0x7fb8340035c0 2024-03-24T06:30:04.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.993+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fb8440203e0 con 0x7fb8340035c0 2024-03-24T06:30:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.993+0000 7fb83b7fe700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 0x7fb82c03b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.993+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb84c191b30 con 0x7fb8340035c0 2024-03-24T06:30:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.993+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb84404dfb0 con 0x7fb8340035c0 2024-03-24T06:30:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.993+0000 7fb849d9b700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 0x7fb82c03b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:04.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.994+0000 7fb849d9b700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 0x7fb82c03b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb840009850 tx=0x7fb840008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:04.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:04.998+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb844025080 con 0x7fb8340035c0 2024-03-24T06:30:05.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:05.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:05.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:05.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:05.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:05.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:05.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.281+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb84c063830 con 0x7fb8340035c0 2024-03-24T06:30:05.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.282+0000 7fb83b7fe700 1 -- 172.21.15.155:0/776303965 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7fb8440206e0 con 0x7fb8340035c0 2024-03-24T06:30:05.282 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T06:30:05.282 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","modified":"2024-03-24T06:29:58.859841Z","created":"2024-03-24T06:27:43.107757Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 msgr2=0x7fb82c03b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 0x7fb82c03b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb840009850 tx=0x7fb840008040 comp rx=0 tx=0).stop 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 msgr2=0x7fb84c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb844017720 tx=0x7fb844017800 comp rx=0 tx=0).stop 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 shutdown_connections 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb82c039410 0x7fb82c03b8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb84c1a3940 0x7fb84c1a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:05.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8340035c0 0x7fb84c100b80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:05.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 --2- 172.21.15.155:0/776303965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb84c1010c0 0x7fb84c101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:05.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 >> 172.21.15.155:0/776303965 conn(0x7fb84c0fc710 msgr2=0x7fb84c103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:05.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 shutdown_connections 2024-03-24T06:30:05.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:05.284+0000 7fb850d8a700 1 -- 172.21.15.155:0/776303965 wait complete. 2024-03-24T06:30:05.286 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 3 2024-03-24T06:30:06.164 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T06:30:06.165 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: mon.c calling monitor election 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: mon.a calling monitor election 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:30:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: monmap epoch 3 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: min_mon_release 19 (squid) 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: election_strategy: 1 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:06.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: fsmap 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: mgrmap e13: x(active, since 68s) 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: overall HEALTH_OK 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:06.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:06.570 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mon.b calling monitor election 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mon.b calling monitor election 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mon.c calling monitor election 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mon.a calling monitor election 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: monmap epoch 3 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:06.896 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: min_mon_release 19 (squid) 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: election_strategy: 1 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: fsmap 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: mgrmap e13: x(active, since 70s) 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: overall HEALTH_OK 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mon.b calling monitor election 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mon.b calling monitor election 2024-03-24T06:30:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mon.c calling monitor election 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mon.a calling monitor election 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: monmap epoch 3 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: min_mon_release 19 (squid) 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: election_strategy: 1 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: fsmap 2024-03-24T06:30:06.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: mgrmap e13: x(active, since 70s) 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: overall HEALTH_OK 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:06.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mon.b calling monitor election 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mon.b calling monitor election 2024-03-24T06:30:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mon.c calling monitor election 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mon.a calling monitor election 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: monmap epoch 3 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: last_changed 2024-03-24T06:29:58.859841+0000 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: created 2024-03-24T06:27:43.107757+0000 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: min_mon_release 19 (squid) 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: election_strategy: 1 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: 0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: 2: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: fsmap 2024-03-24T06:30:07.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: mgrmap e13: x(active, since 70s) 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: overall HEALTH_OK 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:07.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.024 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:30:07.860+0000 7f0223600700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T06:30:08.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:08.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.024 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:08.025 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:08.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:08.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:08.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:08.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:08.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:08 smithi022 ceph-mon[32003]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:08 smithi022 ceph-mon[32003]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T06:30:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:08 smithi022 ceph-mon[32003]: Reconfiguring daemon mon.a on smithi022 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:08 smithi143 ceph-mon[36520]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:08 smithi143 ceph-mon[36520]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:08 smithi143 ceph-mon[36520]: Reconfiguring daemon mon.a on smithi022 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:08 smithi155 ceph-mon[36262]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:08 smithi155 ceph-mon[36262]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T06:30:09.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:08 smithi155 ceph-mon[36262]: Reconfiguring daemon mon.a on smithi022 2024-03-24T06:30:09.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 -- 172.21.15.22:0/3605054701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 msgr2=0x7f9df0108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:09.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 --2- 172.21.15.22:0/3605054701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df0108840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009910 tx=0x7f9de4009c20 comp rx=0 tx=0).stop 2024-03-24T06:30:09.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 -- 172.21.15.22:0/3605054701 shutdown_connections 2024-03-24T06:30:09.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 --2- 172.21.15.22:0/3605054701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df0108840 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 -- 172.21.15.22:0/3605054701 >> 172.21.15.22:0/3605054701 conn(0x7f9df00779a0 msgr2=0x7f9df0079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 -- 172.21.15.22:0/3605054701 shutdown_connections 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.203+0000 7f9df8400700 1 -- 172.21.15.22:0/3605054701 wait complete. 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.204+0000 7f9df8400700 1 Processor -- start 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 -- start start 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df019e2d0 0x7f9df01a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df01a2ba0 0x7f9df01a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9df019e9c0 con 0x7f9df0108460 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01aa910 con 0x7f9df01a2ba0 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df8400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01aaa50 con 0x7f9df019e2d0 2024-03-24T06:30:09.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df5a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df5a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:55578/0 (socket says 172.21.15.22:55578) 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df5a00700 1 -- 172.21.15.22:0/1375366158 learned_addr learned my addr 172.21.15.22:0/1375366158 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.205+0000 7f9df5000700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df019e2d0 0x7f9df01a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df6400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df01a2ba0 0x7f9df01a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 -- 172.21.15.22:0/1375366158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df019e2d0 msgr2=0x7f9df01a2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df019e2d0 0x7f9df01a2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 -- 172.21.15.22:0/1375366158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df01a2ba0 msgr2=0x7f9df01a3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df01a2ba0 0x7f9df01a3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 -- 172.21.15.22:0/1375366158 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de40095f0 con 0x7f9df0108460 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df5a00700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9de400bcf0 tx=0x7f9de400bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:09.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de401d070 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01aacd0 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.206+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de400fb50 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.207+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4022d70 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.207+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01ab170 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.208+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9de4017440 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.208+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df0063830 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.208+0000 7f9dee000700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 0x7f9dd803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.208+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9de404d170 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.208+0000 7f9df5000700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 0x7f9dd803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.209+0000 7f9df5000700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 0x7f9dd803b8d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009ad0 tx=0x7f9ddc009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.214+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9de400ee50 con 0x7f9df0108460 2024-03-24T06:30:09.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.447+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9df0112690 con 0x7f9df0108460 2024-03-24T06:30:09.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.447+0000 7f9dee000700 1 -- 172.21.15.22:0/1375366158 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f9de4027020 con 0x7f9df0108460 2024-03-24T06:30:09.490 INFO:teuthology.orchestra.run.smithi022.stdout:# minimal ceph.conf for 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:09.490 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-24T06:30:09.490 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:30:09.490 INFO:teuthology.orchestra.run.smithi022.stdout: mon_host = [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 msgr2=0x7f9dd803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 0x7f9dd803b8d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009ad0 tx=0x7f9ddc009210 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 msgr2=0x7f9df019dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9de400bcf0 tx=0x7f9de400bd20 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 shutdown_connections 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.450+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9df0108460 0x7f9df019dd90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9dd8039410 0x7f9dd803b8d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9df01a2ba0 0x7f9df01a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 --2- 172.21.15.22:0/1375366158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9df019e2d0 0x7f9df01a2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:09.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 >> 172.21.15.22:0/1375366158 conn(0x7f9df00779a0 msgr2=0x7f9df0109e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:09.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 shutdown_connections 2024-03-24T06:30:09.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:09.451+0000 7f9df8400700 1 -- 172.21.15.22:0/1375366158 wait complete. 2024-03-24T06:30:10.212 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T06:30:10.213 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:30:10.213 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T06:30:10.317 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:30:10.317 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:30:10.460 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:30:10.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T06:30:10.489 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:30:10.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1375366158' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: Reconfiguring mon.b (monmap changed)... 2024-03-24T06:30:10.554 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:10.555 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:10.555 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.555 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:10 smithi143 ceph-mon[36520]: Reconfiguring daemon mon.b on smithi143 2024-03-24T06:30:10.558 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:30:10.558 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T06:30:10.588 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:30:10.588 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1375366158' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: Reconfiguring mon.b (monmap changed)... 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:10.652 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.653 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:10 smithi155 ceph-mon[36262]: Reconfiguring daemon mon.b on smithi143 2024-03-24T06:30:10.655 INFO:tasks.cephadm:Adding mgr.x on smithi022 2024-03-24T06:30:10.655 INFO:tasks.cephadm:Adding mgr.y on smithi143 2024-03-24T06:30:10.655 INFO:tasks.cephadm:Adding mgr.z on smithi155 2024-03-24T06:30:10.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi022=x;smithi143=y;smithi155=z' 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1375366158' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: Reconfiguring mon.b (monmap changed)... 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:10 smithi022 ceph-mon[32003]: Reconfiguring daemon mon.b on smithi143 2024-03-24T06:30:10.923 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:30:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: Reconfiguring mon.c (monmap changed)... 2024-03-24T06:30:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:12.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:12.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:12.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:12 smithi155 ceph-mon[36262]: Reconfiguring daemon mon.c on smithi155 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: Reconfiguring mon.c (monmap changed)... 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:12 smithi022 ceph-mon[32003]: Reconfiguring daemon mon.c on smithi155 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: Reconfiguring mon.c (monmap changed)... 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:12 smithi143 ceph-mon[36520]: Reconfiguring daemon mon.c on smithi155 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 -- 172.21.15.155:0/1360105482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 msgr2=0x7fd520101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 --2- 172.21.15.155:0/1360105482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520101540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd510009990 tx=0x7fd510009ca0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 -- 172.21.15.155:0/1360105482 shutdown_connections 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 --2- 172.21.15.155:0/1360105482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 0x7fd520109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 --2- 172.21.15.155:0/1360105482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520101540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 --2- 172.21.15.155:0/1360105482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd520102320 0x7fd520102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 -- 172.21.15.155:0/1360105482 >> 172.21.15.155:0/1360105482 conn(0x7fd5200fc6c0 msgr2=0x7fd5200feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:13.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 -- 172.21.15.155:0/1360105482 shutdown_connections 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.605+0000 7fd525461700 1 -- 172.21.15.155:0/1360105482 wait complete. 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 Processor -- start 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 -- start start 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd520102320 0x7fd520198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 0x7fd52019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd52019fee0 con 0x7fd520102320 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a0020 con 0x7fd520102cc0 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.606+0000 7fd525461700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5201a01b0 con 0x7fd520101120 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:60104/0 (socket says 172.21.15.155:60104) 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 -- 172.21.15.155:0/837062815 learned_addr learned my addr 172.21.15.155:0/837062815 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:30:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51f7fe700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 0x7fd52019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 -- 172.21.15.155:0/837062815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 msgr2=0x7fd52019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 0x7fd52019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 -- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd520102320 msgr2=0x7fd520198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd520102320 0x7fd520198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 -- 172.21.15.155:0/837062815 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5100095f0 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd51effd700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd510003760 tx=0x7fd51000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51001d070 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51000fc30 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5201a0430 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510017650 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.607+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5201a08c0 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.608+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fd5100177b0 con 0x7fd520101120 2024-03-24T06:30:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.608+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd520068ec0 con 0x7fd520101120 2024-03-24T06:30:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.608+0000 7fd507fff700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 0x7fd50803b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.608+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd51004d4f0 con 0x7fd520101120 2024-03-24T06:30:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.609+0000 7fd51e7fc700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 0x7fd50803b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.610+0000 7fd51e7fc700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 0x7fd50803b680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5140097b0 tx=0x7fd514006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:13.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.613+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd51002a950 con 0x7fd520101120 2024-03-24T06:30:13.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.819+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi022=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}) -- 0x7fd52006ae30 con 0x7fd5080391c0 2024-03-24T06:30:13.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.832+0000 7fd507fff700 1 -- 172.21.15.155:0/837062815 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd52006ae30 con 0x7fd5080391c0 2024-03-24T06:30:13.832 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mgr update... 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 msgr2=0x7fd50803b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 0x7fd50803b680 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5140097b0 tx=0x7fd514006d20 comp rx=0 tx=0).stop 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 msgr2=0x7fd520198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd510003760 tx=0x7fd51000f690 comp rx=0 tx=0).stop 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 shutdown_connections 2024-03-24T06:30:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd5080391c0 0x7fd50803b680 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd520102cc0 0x7fd52019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd520101120 0x7fd520198350 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 --2- 172.21.15.155:0/837062815 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd520102320 0x7fd520198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 >> 172.21.15.155:0/837062815 conn(0x7fd5200fc6c0 msgr2=0x7fd52006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 shutdown_connections 2024-03-24T06:30:13.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:30:13.835+0000 7fd525461700 1 -- 172.21.15.155:0/837062815 wait complete. 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:14.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:14.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.443 DEBUG:teuthology.orchestra.run.smithi143:mgr.y> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T06:30:14.445 DEBUG:teuthology.orchestra.run.smithi155:mgr.z> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T06:30:14.447 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T06:30:14.448 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:30:14.448 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T06:30:14.509 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T06:30:14.509 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-24T06:30:14.565 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T06:30:14.565 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:14.565 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-24T06:30:14.565 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:14.566 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:14.566 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 06:29:11.591037396 +0000 2024-03-24T06:30:14.566 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 06:23:51.070909109 +0000 2024-03-24T06:30:14.566 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 06:23:51.070909109 +0000 2024-03-24T06:30:14.566 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 06:23:51.070909109 +0000 2024-03-24T06:30:14.566 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T06:30:14.662 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T06:30:14.662 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T06:30:14.663 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000265134 s, 1.9 MB/s 2024-03-24T06:30:14.667 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T06:30:14.727 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 06:29:11.685036151 +0000 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 06:23:51.642901428 +0000 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 06:23:51.642901428 +0000 2024-03-24T06:30:14.781 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 06:23:51.642901428 +0000 2024-03-24T06:30:14.782 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T06:30:14.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:14.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi022=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:14.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: Saving service mgr spec with placement smithi022=x;smithi143=y;smithi155=z;count:3 2024-03-24T06:30:14.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:14.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:14 smithi155 ceph-mon[36262]: Deploying daemon mgr.z on smithi155 2024-03-24T06:30:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T06:30:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T06:30:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000238402 s, 2.1 MB/s 2024-03-24T06:30:14.877 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T06:30:14.937 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 06:29:11.785034827 +0000 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 06:23:52.228893558 +0000 2024-03-24T06:30:14.988 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 06:23:52.228893558 +0000 2024-03-24T06:30:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 06:23:52.228893558 +0000 2024-03-24T06:30:14.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi022=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: Saving service mgr spec with placement smithi022=x;smithi143=y;smithi155=z;count:3 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:15.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:15.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:15.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:15.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:15.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:14 smithi022 ceph-mon[32003]: Deploying daemon mgr.z on smithi155 2024-03-24T06:30:15.078 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T06:30:15.078 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T06:30:15.079 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000226823 s, 2.3 MB/s 2024-03-24T06:30:15.083 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T06:30:15.143 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi022=x;smithi143=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: Saving service mgr spec with placement smithi022=x;smithi143=y;smithi155=z;count:3 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:15.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:15.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:15.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:15.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:15.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:14 smithi143 ceph-mon[36520]: Deploying daemon mgr.z on smithi155 2024-03-24T06:30:15.192 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T06:30:15.192 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:15.192 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 06:29:11.880033569 +0000 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 06:23:52.815885675 +0000 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 06:23:52.815885675 +0000 2024-03-24T06:30:15.193 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 06:23:52.815885675 +0000 2024-03-24T06:30:15.193 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T06:30:15.278 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T06:30:15.279 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T06:30:15.279 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000254564 s, 2.0 MB/s 2024-03-24T06:30:15.282 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T06:30:15.340 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T06:30:15.340 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T06:30:15.360 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T06:30:15.360 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-03-24T06:30:15.417 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T06:30:15.417 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:15.417 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 06:29:58.313800189 +0000 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 06:22:17.806446256 +0000 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 06:22:17.806446256 +0000 2024-03-24T06:30:15.418 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 06:22:17.806446256 +0000 2024-03-24T06:30:15.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T06:30:15.485 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T06:30:15.485 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T06:30:15.486 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00037296 s, 1.4 MB/s 2024-03-24T06:30:15.487 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T06:30:15.546 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 06:29:58.313800189 +0000 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 06:22:18.142437388 +0000 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 06:22:18.142437388 +0000 2024-03-24T06:30:15.603 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 06:22:18.142437388 +0000 2024-03-24T06:30:15.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T06:30:15.671 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T06:30:15.671 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T06:30:15.671 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000192649 s, 2.7 MB/s 2024-03-24T06:30:15.672 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T06:30:15.729 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 06:29:58.313800189 +0000 2024-03-24T06:30:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 06:22:18.472428677 +0000 2024-03-24T06:30:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 06:22:18.472428677 +0000 2024-03-24T06:30:15.786 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 06:22:18.472428677 +0000 2024-03-24T06:30:15.786 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T06:30:15.852 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T06:30:15.852 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T06:30:15.852 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000238797 s, 2.1 MB/s 2024-03-24T06:30:15.854 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T06:30:15.910 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 06:29:58.314800163 +0000 2024-03-24T06:30:15.965 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 06:22:18.828419281 +0000 2024-03-24T06:30:15.966 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 06:22:18.828419281 +0000 2024-03-24T06:30:15.966 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 06:22:18.828419281 +0000 2024-03-24T06:30:15.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T06:30:16.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:15 smithi155 systemd[1]: Starting Ceph mgr.z for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:30:16.031 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T06:30:16.031 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T06:30:16.032 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00019847 s, 2.6 MB/s 2024-03-24T06:30:16.033 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T06:30:16.089 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T06:30:16.089 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T06:30:16.105 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T06:30:16.105 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-24T06:30:16.160 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T06:30:16.160 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:16.160 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-24T06:30:16.160 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:16.161 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:16.161 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 06:30:15.814116951 +0000 2024-03-24T06:30:16.161 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 06:22:11.206331499 +0000 2024-03-24T06:30:16.161 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 06:22:11.206331499 +0000 2024-03-24T06:30:16.161 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 06:22:11.206331499 +0000 2024-03-24T06:30:16.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T06:30:16.229 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T06:30:16.229 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T06:30:16.229 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000195132 s, 2.6 MB/s 2024-03-24T06:30:16.230 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T06:30:16.285 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 06:30:15.815116930 +0000 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 06:22:11.561323790 +0000 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 06:22:11.561323790 +0000 2024-03-24T06:30:16.340 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 06:22:11.561323790 +0000 2024-03-24T06:30:16.341 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T06:30:16.376 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 podman[38075]: 2024-03-24 06:30:15.99996798 +0000 UTC m=+0.017925158 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:30:16.376 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 podman[38075]: 2024-03-24 06:30:16.124302885 +0000 UTC m=+0.142260041 container create 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T06:30:16.376 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 podman[38075]: 2024-03-24 06:30:16.376854576 +0000 UTC m=+0.394811740 container init 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T06:30:16.402 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T06:30:16.402 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T06:30:16.402 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000187265 s, 2.7 MB/s 2024-03-24T06:30:16.403 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T06:30:16.459 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-24T06:30:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 06:30:15.815116930 +0000 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 06:22:11.903316362 +0000 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 06:22:11.903316362 +0000 2024-03-24T06:30:16.514 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 06:22:11.903316362 +0000 2024-03-24T06:30:16.514 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T06:30:16.580 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T06:30:16.580 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T06:30:16.580 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000206601 s, 2.5 MB/s 2024-03-24T06:30:16.581 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T06:30:16.636 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 06:30:15.816116909 +0000 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 06:22:12.246308913 +0000 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 06:22:12.246308913 +0000 2024-03-24T06:30:16.691 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 06:22:12.246308913 +0000 2024-03-24T06:30:16.692 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T06:30:16.755 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 podman[38075]: 2024-03-24 06:30:16.379963852 +0000 UTC m=+0.397921029 container start 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T06:30:16.755 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mgr[38180]: -- 172.21.15.155:0/2518688220 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc03eca1a0 con 0x55dc03ea1000 2024-03-24T06:30:16.755 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 bash[38075]: 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb 2024-03-24T06:30:16.755 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 systemd[1]: Started Ceph mgr.z for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:30:16.757 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T06:30:16.757 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T06:30:16.757 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000189594 s, 2.7 MB/s 2024-03-24T06:30:16.758 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T06:30:16.814 INFO:tasks.cephadm:Deploying osd.0 on smithi022 with /dev/vg_nvme/lv_4... 2024-03-24T06:30:16.814 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T06:30:17.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:16.944+0000 7f0533c97200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:30:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:17.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:17.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:17.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:17.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:17.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:17.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T06:30:17.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:17.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:17.268 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:30:17.527 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:17.036+0000 7f0533c97200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:30:17.989 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-mon[36262]: Deploying daemon mgr.y on smithi143 2024-03-24T06:30:17.990 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:17.664+0000 7f0533c97200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:30:17.990 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:17.754+0000 7f0533c97200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:30:17.990 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:17.896+0000 7f0533c97200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:30:18.073 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:17 smithi022 ceph-mon[32003]: Deploying daemon mgr.y on smithi143 2024-03-24T06:30:18.140 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:17 smithi143 ceph-mon[36520]: Deploying daemon mgr.y on smithi143 2024-03-24T06:30:18.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:17.989+0000 7f0533c97200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:30:18.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.071+0000 7f0533c97200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:30:18.610 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.329+0000 7f0533c97200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:30:18.610 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.436+0000 7f0533c97200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:30:18.611 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.519+0000 7f0533c97200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:30:18.611 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.611+0000 7f0533c97200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:30:18.793 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:18 smithi143 systemd[1]: Starting Ceph mgr.y for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:30:18.956 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:18 smithi022 ceph-mon[32003]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:19.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-mon[36262]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:19.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:18.785+0000 7f0533c97200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:18 smithi143 ceph-mon[36520]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:18 smithi143 podman[37929]: 2024-03-24 06:30:18.793961647 +0000 UTC m=+0.093055490 container create f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:18 smithi143 podman[37929]: 2024-03-24 06:30:18.718266783 +0000 UTC m=+0.017360640 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:18 smithi143 podman[37929]: 2024-03-24 06:30:18.979697989 +0000 UTC m=+0.278791835 container init f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:18 smithi143 podman[37929]: 2024-03-24 06:30:18.982325913 +0000 UTC m=+0.281419759 container start f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-24T06:30:19.053 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:19 smithi143 ceph-mgr[37948]: -- 172.21.15.143:0/720559766 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab6c10c1a0 con 0x55ab6c0e2800 2024-03-24T06:30:19.054 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:19 smithi143 bash[37929]: f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 2024-03-24T06:30:19.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:19 smithi143 systemd[1]: Started Ceph mgr.y for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0358164 s, 293 MB/s 2024-03-24T06:30:19.792 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T06:30:19.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:19 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:19.540+0000 7fe34f19f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:30:19.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:19 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:19.637+0000 7fe34f19f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:30:20.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:19 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:19.767+0000 7f0533c97200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:30:20.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:19 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:19.904+0000 7f0533c97200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.405 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:20.333+0000 7fe34f19f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:30:20.405 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:20.405 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:20.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.527 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:20.261+0000 7f0533c97200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:30:20.527 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:20.354+0000 7f0533c97200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:30:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:20.534 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_4 2024-03-24T06:30:20.670 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:20.434+0000 7fe34f19f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:30:20.670 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:20.575+0000 7fe34f19f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:30:20.670 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:20.670+0000 7fe34f19f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:30:20.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:20.755+0000 7fe34f19f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:30:20.994 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:30:21.014 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:20.548+0000 7f0533c97200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:30:21.014 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:20.735+0000 7f0533c97200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:30:21.274 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:21.015+0000 7f0533c97200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:30:21.274 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:21.275+0000 7f0533c97200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:30:21.313 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:21.028+0000 7fe34f19f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:30:21.314 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:21.136+0000 7fe34f19f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:30:21.314 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:21.221+0000 7fe34f19f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:30:21.314 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:21.314+0000 7fe34f19f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:30:21.527 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:21.504+0000 7f0533c97200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:30:21.633 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:21.489+0000 7fe34f19f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:30:21.843 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:21.844 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:21.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:21.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:21.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:22.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:22.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T06:30:22.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T06:30:22.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:22.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:22.286+0000 7f0533c97200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:30:22.854 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:22.452+0000 7fe34f19f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:30:22.854 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:22.581+0000 7fe34f19f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:30:23.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:22 smithi022 ceph-mon[32003]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:23.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:22 smithi022 ceph-mon[32003]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T06:30:23.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:22 smithi022 ceph-mon[32003]: Reconfiguring daemon mgr.x on smithi022 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-mon[36262]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-mon[36262]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-mon[36262]: Reconfiguring daemon mgr.x on smithi022 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:22.847+0000 7f0533c97200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:22 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:22.943+0000 7f0533c97200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:30:23.151 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:30:23.151+0000 7f0533c97200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:30:23.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-mon[36520]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:23.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-mon[36520]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T06:30:23.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-mon[36520]: Reconfiguring daemon mgr.x on smithi022 2024-03-24T06:30:23.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:22 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:22.932+0000 7fe34f19f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:30:23.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:23.019+0000 7fe34f19f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 -- 172.21.15.22:0/984871676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 msgr2=0x7fb70c10a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 --2- 172.21.15.22:0/984871676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c10a6d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb70400b0d0 tx=0x7fb70400b3e0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 -- 172.21.15.22:0/984871676 shutdown_connections 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 --2- 172.21.15.22:0/984871676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 --2- 172.21.15.22:0/984871676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c10a6d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 --2- 172.21.15.22:0/984871676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 0x7fb70c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 -- 172.21.15.22:0/984871676 >> 172.21.15.22:0/984871676 conn(0x7fb70c0fd820 msgr2=0x7fb70c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:23.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 -- 172.21.15.22:0/984871676 shutdown_connections 2024-03-24T06:30:23.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.371+0000 7fb710e00700 1 -- 172.21.15.22:0/984871676 wait complete. 2024-03-24T06:30:23.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 Processor -- start 2024-03-24T06:30:23.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 -- start start 2024-03-24T06:30:23.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 0x7fb70c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c1a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1a20b0 con 0x7fb70c102280 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1a21f0 con 0x7fb70c103e20 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.373+0000 7fb710e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1a2380 con 0x7fb70c103480 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb709e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c1a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb709e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35616/0 (socket says 172.21.15.22:35616) 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb709e00700 1 -- 172.21.15.22:0/2811350416 learned_addr learned my addr 172.21.15.22:0/2811350416 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:30:23.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb709400700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 0x7fb70c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:23.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 -- 172.21.15.22:0/2811350416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 msgr2=0x7fb70c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 0x7fb70c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 -- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 msgr2=0x7fb70c19a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c19a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 -- 172.21.15.22:0/2811350416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb704009d00 con 0x7fb70c103e20 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.374+0000 7fb70a800700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c1a1ae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb70400d010 tx=0x7fb70400bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.375+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70400e040 con 0x7fb70c103e20 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.375+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb704003e10 con 0x7fb70c103e20 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.375+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1a2600 con 0x7fb70c103e20 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.375+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70401d9f0 con 0x7fb70c103e20 2024-03-24T06:30:23.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.375+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a2a70 con 0x7fb70c103e20 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.376+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70c068ec0 con 0x7fb70c103e20 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.376+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fb70401db50 con 0x7fb70c103e20 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.377+0000 7fb702a00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 0x7fb6f003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.377+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb70404ce80 con 0x7fb70c103e20 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.377+0000 7fb709e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 0x7fb6f003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:23.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.378+0000 7fb709e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 0x7fb6f003b680 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005950 tx=0x7fb6fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:23.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.382+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb704011a80 con 0x7fb70c103e20 2024-03-24T06:30:23.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.622+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb70c075b50 con 0x7fb6f00391c0 2024-03-24T06:30:23.674 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:23.211+0000 7fe34f19f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:30:23.674 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:23.395+0000 7fe34f19f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:30:23.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:23.869+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 14) ==== 91342+0+0 (secure 0 0 0) 0x7fb704051070 con 0x7fb70c103e20 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:23.674+0000 7fe34f19f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: Standby manager daemon z started 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:23.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: Standby manager daemon z started 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:24.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: Standby manager daemon z started 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1964240728' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:24.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:24.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:24.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:24.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:23.937+0000 7fe34f19f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:30:24.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:24.172+0000 7fe34f19f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:30:25.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:25.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:25.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:25.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:24.964+0000 7fe34f19f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:25.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:25.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:25.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:25.841 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:25.530+0000 7fe34f19f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:30:25.841 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:25.628+0000 7fe34f19f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:30:25.841 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:30:25.841+0000 7fe34f19f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:30:25.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:25.894+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7fb704026400 con 0x7fb70c103e20 2024-03-24T06:30:25.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:25 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:30:25.894 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:25 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:25.895 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:25 smithi022 ceph-mon[32003]: Standby manager daemon y started 2024-03-24T06:30:25.895 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:25 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:30:25.895 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:25 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:30:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-mon[36520]: Standby manager daemon y started 2024-03-24T06:30:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:30:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:25 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:25 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:30:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:25 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:30:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:25 smithi155 ceph-mon[36262]: Standby manager daemon y started 2024-03-24T06:30:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:25 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:30:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:25 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/2992738435' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:30:27.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:26 smithi022 ceph-mon[32003]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:27.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:26 smithi022 ceph-mon[32003]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T06:30:27.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:30:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:26 smithi143 ceph-mon[36520]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:26 smithi143 ceph-mon[36520]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T06:30:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:30:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:26 smithi155 ceph-mon[36262]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:26 smithi155 ceph-mon[36262]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T06:30:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:30:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:27 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/589082891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:27 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:27 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]': finished 2024-03-24T06:30:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:27 smithi022 ceph-mon[32003]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T06:30:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:28.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:27 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/589082891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:27 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:27 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]': finished 2024-03-24T06:30:28.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:27 smithi143 ceph-mon[36520]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T06:30:28.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:27 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/589082891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:27 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]: dispatch 2024-03-24T06:30:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:27 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b50e1d7a-25e6-45e4-8f2e-51198cd3c017"}]': finished 2024-03-24T06:30:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:27 smithi155 ceph-mon[36262]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T06:30:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:28 smithi022 ceph-mon[32003]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:28 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2005635921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:29.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:28 smithi143 ceph-mon[36520]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:29.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:28 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2005635921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:29.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:28 smithi155 ceph-mon[36262]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:29.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:28 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2005635921' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:30 smithi143 ceph-mon[36520]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:30 smithi155 ceph-mon[36262]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:31.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:30 smithi022 ceph-mon[32003]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:33.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:32 smithi143 ceph-mon[36520]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:33.231 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:32 smithi022 ceph-mon[32003]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:32 smithi155 ceph-mon[36262]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:35.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:34 smithi022 ceph-mon[32003]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:34 smithi143 ceph-mon[36520]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:34 smithi155 ceph-mon[36262]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:37.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:36 smithi022 ceph-mon[32003]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:37.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T06:30:37.166 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:36 smithi143 ceph-mon[36520]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T06:30:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:36 smithi155 ceph-mon[36262]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T06:30:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:38.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:37 smithi022 ceph-mon[32003]: Deploying daemon osd.0 on smithi022 2024-03-24T06:30:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:37 smithi143 ceph-mon[36520]: Deploying daemon osd.0 on smithi022 2024-03-24T06:30:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:37 smithi155 ceph-mon[36262]: Deploying daemon osd.0 on smithi022 2024-03-24T06:30:39.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:38 smithi022 ceph-mon[32003]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:38 smithi143 ceph-mon[36520]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:39.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:38 smithi155 ceph-mon[36262]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:40.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:39 smithi022 ceph-mon[32003]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:40.195 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:39 smithi155 ceph-mon[36262]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:39 smithi143 ceph-mon[36520]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:42.351 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:42 smithi022 ceph-mon[32003]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:42 smithi143 ceph-mon[36520]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:42 smithi155 ceph-mon[36262]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:45.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:44 smithi022 ceph-mon[32003]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:45.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:45.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:45.044 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:45.131 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:44 smithi143 ceph-mon[36520]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:45.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:45.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:45.132 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:44 smithi155 ceph-mon[36262]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:47.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:46 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:46 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:47.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:46 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:48.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:47 smithi022 ceph-mon[32003]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T06:30:48.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:47 smithi143 ceph-mon[36520]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T06:30:48.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:47 smithi155 ceph-mon[36262]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T06:30:48.636 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 0 on host 'smithi022' 2024-03-24T06:30:48.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.635+0000 7fb702a00700 1 -- 172.21.15.22:0/2811350416 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb70c075b50 con 0x7fb6f00391c0 2024-03-24T06:30:48.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 msgr2=0x7fb6f003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:48.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 0x7fb6f003b680 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005950 tx=0x7fb6fc009260 comp rx=0 tx=0).stop 2024-03-24T06:30:48.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 msgr2=0x7fb70c1a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:48.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c1a1ae0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb70400d010 tx=0x7fb70400bcd0 comp rx=0 tx=0).stop 2024-03-24T06:30:48.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 shutdown_connections 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb70c102280 0x7fb70c19a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb6f00391c0 0x7fb6f003b680 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb70c103e20 0x7fb70c1a1ae0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 --2- 172.21.15.22:0/2811350416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb70c103480 0x7fb70c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 >> 172.21.15.22:0/2811350416 conn(0x7fb70c0fd820 msgr2=0x7fb70c074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 shutdown_connections 2024-03-24T06:30:48.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:30:48.639+0000 7fb710e00700 1 -- 172.21.15.22:0/2811350416 wait complete. 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:49.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T06:30:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T06:30:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:30:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:49.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:30:49.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:49.344 DEBUG:teuthology.orchestra.run.smithi022:osd.0> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T06:30:49.349 INFO:tasks.cephadm:Deploying osd.1 on smithi143 with /dev/vg_nvme/lv_4... 2024-03-24T06:30:49.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T06:30:49.590 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:30:49.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:49 smithi022 ceph-mon[32003]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:30:49.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:49 smithi022 ceph-mon[32003]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T06:30:49.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.897 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.965 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:49 smithi143 ceph-mon[36520]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:30:49.965 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:49 smithi143 ceph-mon[36520]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T06:30:49.965 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:49.965 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:49 smithi155 ceph-mon[36262]: from='osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:30:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:49 smithi155 ceph-mon[36262]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T06:30:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] boot 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T06:30:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:30:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:30:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] boot 2024-03-24T06:30:51.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T06:30:51.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: osd.0 [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] boot 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T06:30:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131627 s, 797 MB/s 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 2024-03-24T06:30:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T06:30:51.891 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:51 smithi143 ceph-mon[36520]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T06:30:51.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_4 2024-03-24T06:30:52.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:51 smithi022 ceph-mon[32003]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T06:30:52.152 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:30:52.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:51 smithi155 ceph-mon[36262]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T06:30:53.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:52 smithi022 ceph-mon[32003]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:53.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:52 smithi143 ceph-mon[36520]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:53.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:52 smithi155 ceph-mon[36262]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:53.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- 172.21.15.143:0/1214080938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2401055f0 msgr2=0x7fb2401079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 --2- 172.21.15.143:0/1214080938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2401055f0 0x7fb2401079e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb234009990 tx=0x7fb234009ca0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- 172.21.15.143:0/1214080938 shutdown_connections 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 --2- 172.21.15.143:0/1214080938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb240107f20 0x7fb24010a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 --2- 172.21.15.143:0/1214080938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240102cc0 0x7fb2401050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 --2- 172.21.15.143:0/1214080938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2401055f0 0x7fb2401079e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- 172.21.15.143:0/1214080938 >> 172.21.15.143:0/1214080938 conn(0x7fb2400fc6f0 msgr2=0x7fb2400feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- 172.21.15.143:0/1214080938 shutdown_connections 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- 172.21.15.143:0/1214080938 wait complete. 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 Processor -- start 2024-03-24T06:30:53.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.661+0000 7fb246469700 1 -- start start 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2401055f0 0x7fb240198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240107f20 0x7fb24019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb24019fdb0 con 0x7fb2401055f0 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb24019fef0 con 0x7fb240102cc0 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2401a0080 con 0x7fb240107f20 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41892/0 (socket says 172.21.15.143:41892) 2024-03-24T06:30:53.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 -- 172.21.15.143:0/1181785054 learned_addr learned my addr 172.21.15.143:0/1181785054 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb244a06700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240107f20 0x7fb24019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 -- 172.21.15.143:0/1181785054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240107f20 msgr2=0x7fb24019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240107f20 0x7fb24019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 -- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2401055f0 msgr2=0x7fb240198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2401055f0 0x7fb240198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 -- 172.21.15.143:0/1181785054 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340095f0 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23ffff700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb23000d980 tx=0x7fb23000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230004220 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb230007640 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2401a0360 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.662+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2300056f0 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.663+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2401a0860 con 0x7fb240102cc0 2024-03-24T06:30:53.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.663+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2400fe270 con 0x7fb240102cc0 2024-03-24T06:30:53.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.664+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7fb23000b750 con 0x7fb240102cc0 2024-03-24T06:30:53.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.665+0000 7fb23d7fa700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 0x7fb2280a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:30:53.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.665+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fb2300cc070 con 0x7fb240102cc0 2024-03-24T06:30:53.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.666+0000 7fb23f7fe700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 0x7fb2280a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:30:53.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.667+0000 7fb23f7fe700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 0x7fb2280a4580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2340098e0 tx=0x7fb234005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:30:53.669 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.670+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb230096ab0 con 0x7fb240102cc0 2024-03-24T06:30:53.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:30:53.872+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb24002dca0 con 0x7fb2280a20c0 2024-03-24T06:30:55.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:55.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:55.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:30:55.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:55.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:30:55.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:30:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:55 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi022 2024-03-24T06:30:56.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:55 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 17095M 2024-03-24T06:30:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:55 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi022 2024-03-24T06:30:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:55 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 17095M 2024-03-24T06:30:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:55 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi022 2024-03-24T06:30:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:55 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 17095M 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/492314994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]': finished 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T06:30:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:30:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/492314994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]': finished 2024-03-24T06:30:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T06:30:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/492314994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]: dispatch 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "572a8c88-6fa5-4e41-9c09-562f3cdd36f3"}]': finished 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T06:30:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:30:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:57 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/2237576459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:57 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/2237576459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:58.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:57 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/2237576459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:30:59.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:30:58 smithi022 ceph-mon[32003]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:30:58 smithi143 ceph-mon[36520]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:30:59.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:30:58 smithi155 ceph-mon[36262]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:00 smithi022 ceph-mon[32003]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:00 smithi143 ceph-mon[36520]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:01.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:00 smithi155 ceph-mon[36262]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:03.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:02 smithi022 ceph-mon[32003]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:03.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:02 smithi143 ceph-mon[36520]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:03.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:02 smithi155 ceph-mon[36262]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:04.159 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T06:31:04.160 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T06:31:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:04.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T06:31:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:04 smithi022 ceph-mon[32003]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:04 smithi022 ceph-mon[32003]: Deploying daemon osd.1 on smithi143 2024-03-24T06:31:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:04 smithi143 ceph-mon[36520]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:04 smithi143 ceph-mon[36520]: Deploying daemon osd.1 on smithi143 2024-03-24T06:31:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:04 smithi155 ceph-mon[36262]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:04 smithi155 ceph-mon[36262]: Deploying daemon osd.1 on smithi143 2024-03-24T06:31:07.064 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:06 smithi143 ceph-mon[36520]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:07.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:06 smithi155 ceph-mon[36262]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:06 smithi022 ceph-mon[32003]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:08 smithi143 ceph-mon[36520]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:08 smithi155 ceph-mon[36262]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:08 smithi022 ceph-mon[32003]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:11.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:10 smithi143 ceph-mon[36520]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:11.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:10 smithi155 ceph-mon[36262]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:11.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:10 smithi022 ceph-mon[32003]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:12.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.108+0000 7fb23d7fa700 1 -- 172.21.15.143:0/1181785054 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb24002dca0 con 0x7fb2280a20c0 2024-03-24T06:31:12.109 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 1 on host 'smithi143' 2024-03-24T06:31:12.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.110+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 msgr2=0x7fb2280a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:12.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.110+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 0x7fb2280a4580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb2340098e0 tx=0x7fb234005fb0 comp rx=0 tx=0).stop 2024-03-24T06:31:12.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.110+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 msgr2=0x7fb240198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:12.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.110+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb23000d980 tx=0x7fb23000dc90 comp rx=0 tx=0).stop 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 shutdown_connections 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb2280a20c0 0x7fb2280a4580 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb240107f20 0x7fb24019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb240102cc0 0x7fb240198220 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 --2- 172.21.15.143:0/1181785054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2401055f0 0x7fb240198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 >> 172.21.15.143:0/1181785054 conn(0x7fb2400fc6f0 msgr2=0x7fb2401035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 shutdown_connections 2024-03-24T06:31:12.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:31:12.111+0000 7fb246469700 1 -- 172.21.15.143:0/1181785054 wait complete. 2024-03-24T06:31:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:12.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:12.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.412 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:12.412 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.412 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.412 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:12.412 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:12.413 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:12.710 DEBUG:teuthology.orchestra.run.smithi143:osd.1> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T06:31:12.713 INFO:tasks.cephadm:Deploying osd.2 on smithi155 with /dev/vg_nvme/lv_4... 2024-03-24T06:31:12.713 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T06:31:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:31:13.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:13.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:13 smithi155 ceph-mon[36262]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:13.362 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:13 smithi155 ceph-mon[36262]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:13 smithi022 ceph-mon[32003]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:13 smithi022 ceph-mon[32003]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:13 smithi143 ceph-mon[36520]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:13.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:13 smithi143 ceph-mon[36520]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T06:31:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T06:31:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T06:31:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:14 smithi155 ceph-mon[36262]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:14 smithi022 ceph-mon[32003]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: from='osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-mon[36520]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121434 s, 863 MB/s 2024-03-24T06:31:14.828 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T06:31:15.099 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1[41155]: 2024-03-24T06:31:14.806+0000 7f71da894700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T06:31:15.099 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:14 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1[41155]: 2024-03-24T06:31:14.810+0000 7f71d267f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: from='osd.1 ' entity='osd.1' 2024-03-24T06:31:15.378 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:31:15.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T06:31:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: from='osd.1 ' entity='osd.1' 2024-03-24T06:31:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.405 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: from='osd.1 ' entity='osd.1' 2024-03-24T06:31:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:15.642 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:31:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi143 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 17080M 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] boot 2024-03-24T06:31:16.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T06:31:16.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:16 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:31:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi143 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 17080M 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] boot 2024-03-24T06:31:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T06:31:16.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:16 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi143 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.476 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 17080M 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: osd.1 [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] boot 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T06:31:16.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:16 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:31:17.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 -- 172.21.15.155:0/1009126792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778102330 msgr2=0x7f0778102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 --2- 172.21.15.155:0/1009126792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778102330 0x7f0778102790 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f076c009990 tx=0x7f076c009ca0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 -- 172.21.15.155:0/1009126792 shutdown_connections 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 --2- 172.21.15.155:0/1009126792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0778102cd0 0x7f0778109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 --2- 172.21.15.155:0/1009126792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778101130 0x7f0778101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 --2- 172.21.15.155:0/1009126792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778102330 0x7f0778102790 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 -- 172.21.15.155:0/1009126792 >> 172.21.15.155:0/1009126792 conn(0x7f07780fc6f0 msgr2=0x7f07780feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 -- 172.21.15.155:0/1009126792 shutdown_connections 2024-03-24T06:31:17.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.232+0000 7f077d21c700 1 -- 172.21.15.155:0/1009126792 wait complete. 2024-03-24T06:31:17.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.233+0000 7f077d21c700 1 Processor -- start 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.233+0000 7f077d21c700 1 -- start start 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.233+0000 7f077d21c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.233+0000 7f077d21c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0778102330 0x7f0778198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778102cd0 0x7f077819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f077819fd90 con 0x7f0778102330 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f077819fed0 con 0x7f0778102cd0 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f07781a0060 con 0x7f0778101130 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:17.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42206/0 (socket says 172.21.15.155:42206) 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 -- 172.21.15.155:0/486030895 learned_addr learned my addr 172.21.15.155:0/486030895 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077759e700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778102cd0 0x7f077819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 -- 172.21.15.155:0/486030895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778102cd0 msgr2=0x7f077819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778102cd0 0x7f077819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 -- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0778102330 msgr2=0x7f0778198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0778102330 0x7f0778198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 -- 172.21.15.155:0/486030895 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076c0095f0 con 0x7f0778101130 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f0776d9d700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f076800d9b0 tx=0x7f076800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768004020 con 0x7f0778101130 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0768009730 con 0x7f0778101130 2024-03-24T06:31:17.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07781a0340 con 0x7f0778101130 2024-03-24T06:31:17.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768003be0 con 0x7f0778101130 2024-03-24T06:31:17.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.234+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07781a08c0 con 0x7f0778101130 2024-03-24T06:31:17.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.235+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07781084f0 con 0x7f0778101130 2024-03-24T06:31:17.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.237+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f0768009ba0 con 0x7f0778101130 2024-03-24T06:31:17.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.238+0000 7f075ffff700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 0x7f07600a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:17.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.238+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f07680cc3f0 con 0x7f0778101130 2024-03-24T06:31:17.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.238+0000 7f077659c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 0x7f07600a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:17.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.239+0000 7f077659c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 0x7f07600a4580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f076c0095c0 tx=0x7f076c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:31:17.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.241+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0768096c10 con 0x7f0778101130 2024-03-24T06:31:17.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:17.442+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f077802dc60 con 0x7f07600a20c0 2024-03-24T06:31:18.340 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:18.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T06:31:18.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:18.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:18.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:18.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:18 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:18 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:18.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T06:31:18.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:18.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:18.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:18.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:18 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/420125560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]': finished 2024-03-24T06:31:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T06:31:20.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/420125560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]': finished 2024-03-24T06:31:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T06:31:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/420125560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]: dispatch 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43917ed1-92cb-42b5-98e4-a92deee31e5a"}]': finished 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T06:31:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:21.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:21 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/3526519324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:21.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:21 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/3526519324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:21 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/3526519324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:22 smithi022 ceph-mon[32003]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:22 smithi143 ceph-mon[36520]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:22 smithi155 ceph-mon[36262]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:24 smithi155 ceph-mon[36262]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:24 smithi022 ceph-mon[32003]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:24 smithi143 ceph-mon[36520]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:26.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:26 smithi022 ceph-mon[32003]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:26 smithi143 ceph-mon[36520]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:26 smithi155 ceph-mon[36262]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:28 smithi022 ceph-mon[32003]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:28 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T06:31:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:28 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:28 smithi022 ceph-mon[32003]: Deploying daemon osd.2 on smithi155 2024-03-24T06:31:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:28 smithi143 ceph-mon[36520]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:28 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T06:31:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:28 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:28 smithi143 ceph-mon[36520]: Deploying daemon osd.2 on smithi155 2024-03-24T06:31:28.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:28 smithi155 ceph-mon[36262]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:28.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:28 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T06:31:28.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:28 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:28.477 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:28 smithi155 ceph-mon[36262]: Deploying daemon osd.2 on smithi155 2024-03-24T06:31:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:30 smithi155 ceph-mon[36262]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:30.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:30 smithi022 ceph-mon[32003]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:30 smithi143 ceph-mon[36520]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:32.218 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:32 smithi155 ceph-mon[36262]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:32 smithi143 ceph-mon[36520]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:32 smithi022 ceph-mon[32003]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:33.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:33.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:33.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:34 smithi155 ceph-mon[36262]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:34.800 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:34 smithi022 ceph-mon[32003]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:34 smithi143 ceph-mon[36520]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:35.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.727+0000 7f075ffff700 1 -- 172.21.15.155:0/486030895 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f077802dc60 con 0x7f07600a20c0 2024-03-24T06:31:35.727 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 2 on host 'smithi155' 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 msgr2=0x7f07600a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 0x7f07600a4580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f076c0095c0 tx=0x7f076c009510 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 msgr2=0x7f0778198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f076800d9b0 tx=0x7f076800dcc0 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 shutdown_connections 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f07600a20c0 0x7f07600a4580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0778102cd0 0x7f077819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0778101130 0x7f0778198200 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 --2- 172.21.15.155:0/486030895 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0778102330 0x7f0778198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:35.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 >> 172.21.15.155:0/486030895 conn(0x7f07780fc6f0 msgr2=0x7f0778105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:31:35.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 shutdown_connections 2024-03-24T06:31:35.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:31:35.729+0000 7f077d21c700 1 -- 172.21.15.155:0/486030895 wait complete. 2024-03-24T06:31:36.367 DEBUG:teuthology.orchestra.run.smithi155:osd.2> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T06:31:36.369 INFO:tasks.cephadm:Deploying osd.3 on smithi022 with /dev/vg_nvme/lv_3... 2024-03-24T06:31:36.369 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T06:31:36.825 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:31:36.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:36.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:36.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:36.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:36.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:36.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:36.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:36 smithi022 ceph-mon[32003]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:36 smithi155 ceph-mon[36262]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:36 smithi143 ceph-mon[36520]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T06:31:37.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:37 smithi022 ceph-mon[32003]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T06:31:37.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:37 smithi022 ceph-mon[32003]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T06:31:37.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:37.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:37 smithi022 ceph-mon[32003]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:37.988 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:37 smithi022 ceph-mon[32003]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:38.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-mon[36262]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T06:31:38.140 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-mon[36262]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T06:31:38.141 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:38.141 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-mon[36262]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:38.141 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-mon[36262]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:38.141 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2[41606]: 2024-03-24T06:31:37.780+0000 7f02a7aa8700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T06:31:38.142 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:37 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2[41606]: 2024-03-24T06:31:37.784+0000 7f02a389b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:31:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:37 smithi143 ceph-mon[36520]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T06:31:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:37 smithi143 ceph-mon[36520]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T06:31:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:37 smithi143 ceph-mon[36520]: from='osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:37 smithi143 ceph-mon[36520]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:31:38.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:38 smithi022 ceph-mon[32003]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:38.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:38 smithi022 ceph-mon[32003]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:31:38.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:38 smithi022 ceph-mon[32003]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T06:31:38.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:38 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:38.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:38 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:39.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:38 smithi155 ceph-mon[36262]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:39.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:38 smithi155 ceph-mon[36262]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:31:39.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:38 smithi155 ceph-mon[36262]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T06:31:39.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:38 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:39.004 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:38 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:38 smithi143 ceph-mon[36520]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:38 smithi143 ceph-mon[36520]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:31:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:38 smithi143 ceph-mon[36520]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T06:31:39.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:38 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:39.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:38 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T06:31:39.458 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T06:31:39.459 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T06:31:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T06:31:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0379104 s, 277 MB/s 2024-03-24T06:31:39.459 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T06:31:40.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] boot 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:40.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] boot 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:40.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:40.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:40.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.227 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:31:40.227 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: osd.2 [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] boot 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:40.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:40.304 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_3 2024-03-24T06:31:40.766 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:31:41.084 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:41.084 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi155 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 17080M 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:41.085 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:40 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi155 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 17080M 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:40 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi155 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 17080M 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:31:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:40 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:41.827 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:41 smithi022 sudo[50974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:41.827 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:41 smithi022 sudo[50974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:41.827 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:41 smithi022 sudo[50974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:41.827 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:41 smithi022 sudo[50974]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:42.126 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:41 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:42.126 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:41 smithi022 ceph-mon[32003]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T06:31:42.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:41 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:42.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:41 smithi143 ceph-mon[36520]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T06:31:42.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:41 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:31:42.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:41 smithi155 ceph-mon[36262]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T06:31:42.394 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:42 smithi022 sudo[50996]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:42.394 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:42 smithi022 sudo[50996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:42.394 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:42 smithi022 sudo[50996]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:42.394 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:31:42 smithi022 sudo[50996]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:42.658 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44872]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:42.658 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44872]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:42.658 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44872]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:42.658 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44872]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.686+0000 7febc1c00700 1 -- 172.21.15.22:0/915340216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 msgr2=0x7febbc105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.686+0000 7febc1c00700 1 --2- 172.21.15.22:0/915340216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc105ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feba80099c0 tx=0x7feba8009cd0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.686+0000 7febc1c00700 1 -- 172.21.15.22:0/915340216 shutdown_connections 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.686+0000 7febc1c00700 1 --2- 172.21.15.22:0/915340216 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc10c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.687+0000 7febc1c00700 1 --2- 172.21.15.22:0/915340216 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 0x7febbc1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.687+0000 7febc1c00700 1 --2- 172.21.15.22:0/915340216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc105ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.687+0000 7febc1c00700 1 -- 172.21.15.22:0/915340216 >> 172.21.15.22:0/915340216 conn(0x7febbc0ffa20 msgr2=0x7febbc101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.687+0000 7febc1c00700 1 -- 172.21.15.22:0/915340216 shutdown_connections 2024-03-24T06:31:42.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.687+0000 7febc1c00700 1 -- 172.21.15.22:0/915340216 wait complete. 2024-03-24T06:31:42.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.688+0000 7febc1c00700 1 Processor -- start 2024-03-24T06:31:42.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.688+0000 7febc1c00700 1 -- start start 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 0x7febbc19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1a1f70 con 0x7febbc106020 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1a20b0 con 0x7febbc105680 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febc1c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1a2240 con 0x7febbc104480 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febbaa00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 0x7febbc19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:38876/0 (socket says 172.21.15.22:38876) 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 -- 172.21.15.22:0/108930471 learned_addr learned my addr 172.21.15.22:0/108930471 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febbb400700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 -- 172.21.15.22:0/108930471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 msgr2=0x7febbc19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 0x7febbc19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 -- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 msgr2=0x7febbc1a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc1a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febba000700 1 -- 172.21.15.22:0/108930471 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba80095f0 con 0x7febbc105680 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.689+0000 7febbb400700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:31:42.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.690+0000 7febba000700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feba80036c0 tx=0x7feba800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.692+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba801d070 con 0x7febbc105680 2024-03-24T06:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.692+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba8022470 con 0x7febbc105680 2024-03-24T06:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.692+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba8017650 con 0x7febbc105680 2024-03-24T06:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.692+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc1a24c0 con 0x7febbc105680 2024-03-24T06:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.692+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc1a2880 con 0x7febbc105680 2024-03-24T06:31:42.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.693+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc10b840 con 0x7febbc105680 2024-03-24T06:31:42.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.695+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7feba80225e0 con 0x7febbc105680 2024-03-24T06:31:42.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.696+0000 7febb3400700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 0x7feba00a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:31:42.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.696+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7feba80cd920 con 0x7febbc105680 2024-03-24T06:31:42.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.696+0000 7febbaa00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 0x7feba00a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:31:42.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.697+0000 7febbaa00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 0x7feba00a4690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febbc1054e0 tx=0x7febac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:31:42.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.700+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feba805e280 con 0x7febbc105680 2024-03-24T06:31:42.928 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44875]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:42.928 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44875]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:42.928 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44875]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:42.928 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:31:42 smithi143 sudo[44875]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:42 smithi143 ceph-mon[36520]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:42 smithi143 ceph-mon[36520]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T06:31:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:42 smithi143 ceph-mon[36520]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T06:31:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:42.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:42.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:31:42.937+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7febbc02db60 con 0x7feba00a21d0 2024-03-24T06:31:42.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:42 smithi022 ceph-mon[32003]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:42.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:42 smithi022 ceph-mon[32003]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T06:31:42.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:42 smithi022 ceph-mon[32003]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T06:31:42.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:42.937 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:43.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:42 smithi155 ceph-mon[36262]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:43.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:42 smithi155 ceph-mon[36262]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T06:31:43.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:42 smithi155 ceph-mon[36262]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T06:31:43.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:43.187 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:43.187 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:42 smithi155 sudo[45701]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:43.187 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:42 smithi155 sudo[45701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:43.187 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:42 smithi155 sudo[45701]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:43.187 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:42 smithi155 sudo[45701]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:43.527 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:43 smithi155 sudo[45704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:43.527 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:43 smithi155 sudo[45704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:43.527 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:43 smithi155 sudo[45704]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:43.527 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:31:43 smithi155 sudo[45704]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:44.069 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:43 smithi022 sudo[51144]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:44.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:43 smithi022 sudo[51144]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:44.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:43 smithi022 sudo[51144]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:44.070 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:43 smithi022 sudo[51144]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 sudo[51167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 sudo[51167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 sudo[51167]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 sudo[51167]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:44 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:44.659 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:44.659 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:44.659 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45707]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:44.660 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45707]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:31:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:31:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:31:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:44 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:45.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:45.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:45.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:44 smithi155 sudo[45710]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44878]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T06:31:45.406 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T06:31:45.407 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T06:31:45.407 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 sudo[44881]: pam_unix(sudo:session): session closed for user root 2024-03-24T06:31:45.407 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:45 smithi143 ceph-mon[36520]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.752 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.753 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:45 smithi022 ceph-mon[32003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:31:45.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:45 smithi155 ceph-mon[36262]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T06:31:46.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:46 smithi143 ceph-mon[36520]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:46.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:46 smithi155 ceph-mon[36262]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:46.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:46 smithi022 ceph-mon[32003]: pgmap v96: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2149168835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:47 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:47 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]': finished 2024-03-24T06:31:47.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:47 smithi143 ceph-mon[36520]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T06:31:47.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:47 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:31:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2149168835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:47 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:47 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]': finished 2024-03-24T06:31:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:47 smithi155 ceph-mon[36262]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T06:31:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:47 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:31:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2149168835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:47 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]: dispatch 2024-03-24T06:31:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:47 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85fc0b5c-cb79-4dc9-84c8-95968df70ee2"}]': finished 2024-03-24T06:31:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:47 smithi022 ceph-mon[32003]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T06:31:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:47 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:31:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:48 smithi143 ceph-mon[36520]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 353 KiB/s wr, 24 op/s 2024-03-24T06:31:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:48 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1435310190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:48 smithi155 ceph-mon[36262]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 353 KiB/s wr, 24 op/s 2024-03-24T06:31:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:48 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1435310190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:48.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:48 smithi022 ceph-mon[32003]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 353 KiB/s wr, 24 op/s 2024-03-24T06:31:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:48 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1435310190' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:31:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:50 smithi155 ceph-mon[36262]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 255 KiB/s wr, 17 op/s 2024-03-24T06:31:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:50 smithi022 ceph-mon[32003]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 255 KiB/s wr, 17 op/s 2024-03-24T06:31:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:50 smithi143 ceph-mon[36520]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.6 KiB/s rd, 255 KiB/s wr, 17 op/s 2024-03-24T06:31:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:52 smithi155 ceph-mon[36262]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 199 KiB/s wr, 13 op/s 2024-03-24T06:31:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:52 smithi022 ceph-mon[32003]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 199 KiB/s wr, 13 op/s 2024-03-24T06:31:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:52 smithi143 ceph-mon[36520]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 199 KiB/s wr, 13 op/s 2024-03-24T06:31:54.624 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:54 smithi022 ceph-mon[32003]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:54.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:54 smithi155 ceph-mon[36262]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:54.927 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:54 smithi143 ceph-mon[36520]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:56 smithi155 ceph-mon[36262]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T06:31:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:56.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:56 smithi155 ceph-mon[36262]: Deploying daemon osd.3 on smithi022 2024-03-24T06:31:56.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:56 smithi022 ceph-mon[32003]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:56.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T06:31:56.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:56.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:56 smithi022 ceph-mon[32003]: Deploying daemon osd.3 on smithi022 2024-03-24T06:31:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:56 smithi143 ceph-mon[36520]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-24T06:31:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T06:31:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:31:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:56 smithi143 ceph-mon[36520]: Deploying daemon osd.3 on smithi022 2024-03-24T06:31:58.746 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:31:58 smithi022 ceph-mon[32003]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:58.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:31:58 smithi155 ceph-mon[36262]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:31:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:31:58 smithi143 ceph-mon[36520]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:00.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:00 smithi022 ceph-mon[32003]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:00.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:00 smithi155 ceph-mon[36262]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:00 smithi143 ceph-mon[36520]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:02 smithi155 ceph-mon[36262]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:02 smithi022 ceph-mon[32003]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:02 smithi143 ceph-mon[36520]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:04.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:04 smithi022 ceph-mon[32003]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:04.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.773 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.774 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:04 smithi155 ceph-mon[36262]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:04 smithi143 ceph-mon[36520]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:06 smithi155 ceph-mon[36262]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:06.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:06 smithi022 ceph-mon[32003]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:06 smithi143 ceph-mon[36520]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:08.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.846 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:08 smithi155 ceph-mon[36262]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:08 smithi022 ceph-mon[32003]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:08 smithi143 ceph-mon[36520]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T06:32:09.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.141+0000 7febb3400700 1 -- 172.21.15.22:0/108930471 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7febbc02db60 con 0x7feba00a21d0 2024-03-24T06:32:09.143 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 3 on host 'smithi022' 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 msgr2=0x7feba00a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 0x7feba00a4690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febbc1054e0 tx=0x7febac009260 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 msgr2=0x7febbc19a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feba80036c0 tx=0x7feba800f690 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 shutdown_connections 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febbc106020 0x7febbc1a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feba00a21d0 0x7feba00a4690 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febbc104480 0x7febbc19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 --2- 172.21.15.22:0/108930471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febbc105680 0x7febbc19a920 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.145+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 >> 172.21.15.22:0/108930471 conn(0x7febbc0ffa20 msgr2=0x7febbc108980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.146+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 shutdown_connections 2024-03-24T06:32:09.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:32:09.146+0000 7febc1c00700 1 -- 172.21.15.22:0/108930471 wait complete. 2024-03-24T06:32:09.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T06:32:09.849 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T06:32:09.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:32:09.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:09.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.850 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:09.914 DEBUG:teuthology.orchestra.run.smithi022:osd.3> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T06:32:09.918 INFO:tasks.cephadm:Deploying osd.4 on smithi143 with /dev/vg_nvme/lv_3... 2024-03-24T06:32:09.918 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:09.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:10.163 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:32:10.473 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3[53302]: 2024-03-24T06:32:10.308+0000 7ff5c7200700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T06:32:10.473 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3[53302]: 2024-03-24T06:32:10.329+0000 7ff5bd000700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:32:10.882 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:10.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:32:10.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T06:32:10.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.883 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:10 smithi155 ceph-mon[36262]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:10 smithi022 ceph-mon[32003]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:10 smithi143 ceph-mon[36520]: from='osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945]' entity='osd.3' 2024-03-24T06:32:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:11 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:32:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:11 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:32:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:11 smithi022 ceph-mon[32003]: osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] boot 2024-03-24T06:32:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:11 smithi022 ceph-mon[32003]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T06:32:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:11 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:32:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:11 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:32:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:11 smithi143 ceph-mon[36520]: osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] boot 2024-03-24T06:32:11.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:11 smithi143 ceph-mon[36520]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T06:32:11.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:12.007 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T06:32:12.007 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T06:32:12.007 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T06:32:12.008 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T06:32:12.008 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113048 s, 928 MB/s 2024-03-24T06:32:12.008 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T06:32:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:11 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:32:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:11 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:32:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:11 smithi155 ceph-mon[36262]: osd.3 [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] boot 2024-03-24T06:32:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:11 smithi155 ceph-mon[36262]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T06:32:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:32:12.609 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:12 smithi143 ceph-mon[36520]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:12.609 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:12 smithi143 ceph-mon[36520]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T06:32:12.637 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_3 2024-03-24T06:32:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:12 smithi022 ceph-mon[32003]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:12 smithi022 ceph-mon[32003]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T06:32:12.917 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:32:13.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:12 smithi155 ceph-mon[36262]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:13.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:12 smithi155 ceph-mon[36262]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T06:32:13.856 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:13 smithi143 ceph-mon[36520]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T06:32:13.861 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:13 smithi022 ceph-mon[32003]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T06:32:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:13 smithi155 ceph-mon[36262]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T06:32:14.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.354+0000 7f032fb28700 1 -- 172.21.15.143:0/1071418417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 msgr2=0x7f03281038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.354+0000 7f032fb28700 1 --2- 172.21.15.143:0/1071418417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281038b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f03180099c0 tx=0x7f0318009cd0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 -- 172.21.15.143:0/1071418417 shutdown_connections 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 --2- 172.21.15.143:0/1071418417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0328103df0 0x7f032810a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 --2- 172.21.15.143:0/1071418417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 0x7f0328102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 --2- 172.21.15.143:0/1071418417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281038b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 -- 172.21.15.143:0/1071418417 >> 172.21.15.143:0/1071418417 conn(0x7f03280fd810 msgr2=0x7f03280ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 -- 172.21.15.143:0/1071418417 shutdown_connections 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 -- 172.21.15.143:0/1071418417 wait complete. 2024-03-24T06:32:14.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 Processor -- start 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.355+0000 7f032fb28700 1 -- start start 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 0x7f03281982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0328103df0 0x7f032819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f032819fe40 con 0x7f0328103df0 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f032819ff80 con 0x7f0328103450 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03281a0110 con 0x7f0328102250 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43626/0 (socket says 172.21.15.143:43626) 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 -- 172.21.15.143:0/1442608270 learned_addr learned my addr 172.21.15.143:0/1442608270 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T06:32:14.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d8c4700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 0x7f03281982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 -- 172.21.15.143:0/1442608270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 msgr2=0x7f03281982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 0x7f03281982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 -- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0328103df0 msgr2=0x7f032819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0328103df0 0x7f032819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 -- 172.21.15.143:0/1442608270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03180095f0 con 0x7f0328103450 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032d0c3700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0318003790 tx=0x7f03180047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031801d070 con 0x7f0328103450 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0318022470 con 0x7f0328103450 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.356+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03281a0390 con 0x7f0328103450 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.357+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031800f6a0 con 0x7f0328103450 2024-03-24T06:32:14.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.357+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03281a08b0 con 0x7f0328103450 2024-03-24T06:32:14.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.358+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0328109610 con 0x7f0328103450 2024-03-24T06:32:14.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.359+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f03180225e0 con 0x7f0328103450 2024-03-24T06:32:14.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.359+0000 7f031effd700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 0x7f03140a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:14.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.359+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f03180cdf80 con 0x7f0328103450 2024-03-24T06:32:14.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.360+0000 7f032d8c4700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 0x7f03140a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:14.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.361+0000 7f032d8c4700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 0x7f03140a4610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03281032b0 tx=0x7f0324009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:32:14.363 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.364+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0318098200 con 0x7f0328103450 2024-03-24T06:32:14.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:14.572+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f032802dc70 con 0x7f03140a2150 2024-03-24T06:32:14.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:14 smithi022 ceph-mon[32003]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:14.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:14 smithi022 ceph-mon[32003]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T06:32:14.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:14 smithi143 ceph-mon[36520]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:14.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:14 smithi143 ceph-mon[36520]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T06:32:15.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:14 smithi155 ceph-mon[36262]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:15.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:14 smithi155 ceph-mon[36262]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T06:32:15.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T06:32:15.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi022 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 8547M 2024-03-24T06:32:15.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:15.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:15.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:15 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi022 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 8547M 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:15.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:15 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi022 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 8547M 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:16.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:15 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:16 smithi155 ceph-mon[36262]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:17.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:16 smithi022 ceph-mon[32003]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:16 smithi143 ceph-mon[36520]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 520 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:17 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/2908981894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:17 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:17 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]': finished 2024-03-24T06:32:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:17 smithi155 ceph-mon[36262]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T06:32:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:18.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:17 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/2908981894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:17 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:17 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]': finished 2024-03-24T06:32:18.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:17 smithi022 ceph-mon[32003]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T06:32:18.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:17 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/2908981894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:17 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]: dispatch 2024-03-24T06:32:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:17 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c774c865-b9c3-4054-a328-e0c3a3c84d56"}]': finished 2024-03-24T06:32:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:17 smithi143 ceph-mon[36520]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T06:32:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:19.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:18 smithi155 ceph-mon[36262]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T06:32:19.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:18 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/2130102075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:19.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:18 smithi022 ceph-mon[32003]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T06:32:19.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:18 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/2130102075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:19.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:18 smithi143 ceph-mon[36520]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-24T06:32:19.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:18 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/2130102075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:21.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:20 smithi155 ceph-mon[36262]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T06:32:21.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:20 smithi022 ceph-mon[32003]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T06:32:21.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:20 smithi143 ceph-mon[36520]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T06:32:23.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:22 smithi155 ceph-mon[36262]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-24T06:32:23.133 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:22 smithi143 ceph-mon[36520]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-24T06:32:23.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:22 smithi022 ceph-mon[32003]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-24T06:32:25.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:24 smithi155 ceph-mon[36262]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T06:32:25.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T06:32:25.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:25.089 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:24 smithi143 ceph-mon[36520]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T06:32:25.089 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T06:32:25.089 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:25.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:24 smithi022 ceph-mon[32003]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-24T06:32:25.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T06:32:25.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:25.912 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:25 smithi143 ceph-mon[36520]: Deploying daemon osd.4 on smithi143 2024-03-24T06:32:26.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:25 smithi155 ceph-mon[36262]: Deploying daemon osd.4 on smithi143 2024-03-24T06:32:26.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:25 smithi022 ceph-mon[32003]: Deploying daemon osd.4 on smithi143 2024-03-24T06:32:26.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:26 smithi143 ceph-mon[36520]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:32:27.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:26 smithi155 ceph-mon[36262]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:32:27.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:26 smithi022 ceph-mon[32003]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:32:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:28 smithi143 ceph-mon[36520]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:29.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:28 smithi155 ceph-mon[36262]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:29.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:28 smithi022 ceph-mon[32003]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:30 smithi022 ceph-mon[32003]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:30 smithi143 ceph-mon[36520]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:30 smithi155 ceph-mon[36262]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:33.102 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:32 smithi143 ceph-mon[36520]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:33.103 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.103 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.103 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:33.103 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:32 smithi155 ceph-mon[36262]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:32 smithi022 ceph-mon[32003]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:33.767 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.767+0000 7f031effd700 1 -- 172.21.15.143:0/1442608270 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f032802dc70 con 0x7f03140a2150 2024-03-24T06:32:33.767 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 4 on host 'smithi143' 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 msgr2=0x7f03140a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 0x7f03140a4610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03281032b0 tx=0x7f0324009190 comp rx=0 tx=0).stop 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 msgr2=0x7f03281987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0318003790 tx=0x7f03180047b0 comp rx=0 tx=0).stop 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 shutdown_connections 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0328103df0 0x7f032819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f03140a2150 0x7f03140a4610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0328102250 0x7f03281982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:33.770 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 --2- 172.21.15.143:0/1442608270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0328103450 0x7f03281987f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:33.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 >> 172.21.15.143:0/1442608270 conn(0x7f03280fd810 msgr2=0x7f0328106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:32:33.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 shutdown_connections 2024-03-24T06:32:33.771 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:32:33.770+0000 7f032fb28700 1 -- 172.21.15.143:0/1442608270 wait complete. 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.081 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:33 smithi143 ceph-mon[36520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:33 smithi155 ceph-mon[36262]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:33 smithi022 ceph-mon[32003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T06:32:34.374 DEBUG:teuthology.orchestra.run.smithi143:osd.4> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T06:32:34.377 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_3... 2024-03-24T06:32:34.378 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T06:32:34.624 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:35.012 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:34 smithi022 ceph-mon[32003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T06:32:35.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:35.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T06:32:35.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:35.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:34 smithi155 ceph-mon[36262]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:35.147 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:35.147 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T06:32:35.147 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: from='osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:35.148 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T06:32:35.148 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:35.148 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:34 smithi143 ceph-mon[36520]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:32:36.758 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4[47722]: 2024-03-24T06:32:36.492+0000 7f367c1a4700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T06:32:36.758 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4[47722]: 2024-03-24T06:32:36.499+0000 7f3673f8f700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123618 s, 848 MB/s 2024-03-24T06:32:36.911 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T06:32:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:36 smithi155 ceph-mon[36262]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:36 smithi155 ceph-mon[36262]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:32:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:36 smithi155 ceph-mon[36262]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T06:32:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:37.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:36 smithi155 ceph-mon[36262]: from='osd.4 ' entity='osd.4' 2024-03-24T06:32:37.039 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-mon[36520]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:37.039 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-mon[36520]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:32:37.039 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-mon[36520]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T06:32:37.040 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:37.040 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:36 smithi143 ceph-mon[36520]: from='osd.4 ' entity='osd.4' 2024-03-24T06:32:37.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:36 smithi022 ceph-mon[32003]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:37.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:36 smithi022 ceph-mon[32003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:32:37.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:36 smithi022 ceph-mon[32003]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T06:32:37.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:37.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:36 smithi022 ceph-mon[32003]: from='osd.4 ' entity='osd.4' 2024-03-24T06:32:37.613 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-03-24T06:32:37.860 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:32:37.899 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:32:37.899 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] boot 2024-03-24T06:32:37.900 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T06:32:37.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:37.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:32:38.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] boot 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T06:32:38.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:38.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:32:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: osd.4 [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] boot 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T06:32:38.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:32:38.180 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:38.953 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:38 smithi155 ceph-mon[36262]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:38.953 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:38 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi143 2024-03-24T06:32:38.953 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:38 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 8540M 2024-03-24T06:32:39.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:38 smithi022 ceph-mon[32003]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:39.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:38 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi143 2024-03-24T06:32:39.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:38 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 8540M 2024-03-24T06:32:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:38 smithi143 ceph-mon[36520]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-24T06:32:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:38 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi143 2024-03-24T06:32:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:38 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 8540M 2024-03-24T06:32:39.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- 172.21.15.155:0/1397906956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101de0 msgr2=0x7f16981096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:39.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- 172.21.15.155:0/1397906956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101de0 0x7f16981096a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f169000b0a0 tx=0x7f169000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- 172.21.15.155:0/1397906956 shutdown_connections 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- 172.21.15.155:0/1397906956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1698102700 0x7f1698102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- 172.21.15.155:0/1397906956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101de0 0x7f16981096a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- 172.21.15.155:0/1397906956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101440 0x7f16981018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- 172.21.15.155:0/1397906956 >> 172.21.15.155:0/1397906956 conn(0x7f16980fbe90 msgr2=0x7f16980fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- 172.21.15.155:0/1397906956 shutdown_connections 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- 172.21.15.155:0/1397906956 wait complete. 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 Processor -- start 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- start start 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101de0 0x7f169819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:39.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1698102700 0x7f169819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f169819fe20 con 0x7f1698102700 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f169819ff90 con 0x7f1698101de0 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.461+0000 7f169f8f2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f16981a0150 con 0x7f1698101440 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:60192/0 (socket says 172.21.15.155:60192) 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 -- 172.21.15.155:0/3383144867 learned_addr learned my addr 172.21.15.155:0/3383144867 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169ce8d700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101de0 0x7f169819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 -- 172.21.15.155:0/3383144867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101de0 msgr2=0x7f169819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101de0 0x7f169819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 -- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1698102700 msgr2=0x7f169819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:39.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1698102700 0x7f169819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 -- 172.21.15.155:0/3383144867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1690009d00 con 0x7f1698101440 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169d68e700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f169400b500 tx=0x7f169400b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694011830 con 0x7f1698101440 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1694011e70 con 0x7f1698101440 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16981a0430 con 0x7f1698101440 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169400f2e0 con 0x7f1698101440 2024-03-24T06:32:39.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.462+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16981a09b0 con 0x7f1698101440 2024-03-24T06:32:39.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.464+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1698108540 con 0x7f1698101440 2024-03-24T06:32:39.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.464+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f16940105f0 con 0x7f1698101440 2024-03-24T06:32:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.465+0000 7f168e7fc700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 0x7f16840a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:32:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.465+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f16940cc6c0 con 0x7f1698101440 2024-03-24T06:32:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.465+0000 7f169ce8d700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 0x7f16840a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:32:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.466+0000 7f169ce8d700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 0x7f16840a4610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1688005950 tx=0x7f168800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:32:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.470+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1694096700 con 0x7f1698101440 2024-03-24T06:32:39.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:39.671+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f169802dc00 con 0x7f16840a2150 2024-03-24T06:32:40.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:39 smithi022 ceph-mon[32003]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T06:32:40.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:40.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:40.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:39 smithi143 ceph-mon[36520]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T06:32:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:39 smithi155 ceph-mon[36262]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T06:32:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:32:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:32:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:40.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:40 smithi155 ceph-mon[36262]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:40.872 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:40 smithi155 ceph-mon[36262]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:41.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:40 smithi022 ceph-mon[32003]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:41.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:40 smithi022 ceph-mon[32003]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:40 smithi143 ceph-mon[36520]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:40 smithi143 ceph-mon[36520]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:32:43.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:43.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/216493923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]': finished 2024-03-24T06:32:43.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T06:32:43.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/216493923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]': finished 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T06:32:43.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:32:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/216493923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]: dispatch 2024-03-24T06:32:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c149b822-fc96-49aa-bb16-e1318e3572e4"}]': finished 2024-03-24T06:32:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T06:32:43.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:32:44.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/1114463932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:44.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:43 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/1114463932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:44.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:43 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/1114463932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:32:45.079 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:44 smithi143 ceph-mon[36520]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:45.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:44 smithi022 ceph-mon[32003]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:44 smithi155 ceph-mon[36262]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:47.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:46 smithi022 ceph-mon[32003]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:47.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:46 smithi143 ceph-mon[36520]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:46 smithi155 ceph-mon[36262]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:49.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:48 smithi155 ceph-mon[36262]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:49.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:48 smithi022 ceph-mon[32003]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:49.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:48 smithi143 ceph-mon[36520]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T06:32:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:50.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T06:32:50.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:50.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T06:32:50.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:32:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:50 smithi155 ceph-mon[36262]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:50 smithi155 ceph-mon[36262]: Deploying daemon osd.5 on smithi155 2024-03-24T06:32:51.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:50 smithi022 ceph-mon[32003]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:51.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:50 smithi022 ceph-mon[32003]: Deploying daemon osd.5 on smithi155 2024-03-24T06:32:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:50 smithi143 ceph-mon[36520]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:50 smithi143 ceph-mon[36520]: Deploying daemon osd.5 on smithi155 2024-03-24T06:32:52.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:52 smithi155 ceph-mon[36262]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:53.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:52 smithi022 ceph-mon[32003]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:53.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:52 smithi143 ceph-mon[36520]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:54 smithi155 ceph-mon[36262]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:54 smithi022 ceph-mon[32003]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:54 smithi143 ceph-mon[36520]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:56 smithi022 ceph-mon[32003]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:56 smithi143 ceph-mon[36520]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:56 smithi155 ceph-mon[36262]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:32:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:32:58.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:58 smithi155 ceph-mon[36262]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:58 smithi022 ceph-mon[32003]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:58.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:58 smithi143 ceph-mon[36520]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:32:59.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.006+0000 7f168e7fc700 1 -- 172.21.15.155:0/3383144867 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f169802dc00 con 0x7f16840a2150 2024-03-24T06:32:59.008 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-03-24T06:32:59.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 msgr2=0x7f16840a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:59.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 0x7f16840a4610 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1688005950 tx=0x7f168800b410 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 msgr2=0x7f1698100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f169400b500 tx=0x7f169400b810 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 shutdown_connections 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1698102700 0x7f169819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16840a2150 0x7f16840a4610 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1698101440 0x7f1698100ef0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 --2- 172.21.15.155:0/3383144867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1698101de0 0x7f169819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 >> 172.21.15.155:0/3383144867 conn(0x7f16980fbe90 msgr2=0x7f1698105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 shutdown_connections 2024-03-24T06:32:59.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:32:59.009+0000 7f169f8f2700 1 -- 172.21.15.155:0/3383144867 wait complete. 2024-03-24T06:32:59.677 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T06:32:59.680 INFO:tasks.cephadm:Deploying osd.6 on smithi022 with /dev/vg_nvme/lv_2... 2024-03-24T06:32:59.680 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T06:33:00.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:00.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:32:59 smithi022 ceph-mon[32003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:00.165 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:32:59 smithi155 ceph-mon[36262]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:32:59 smithi143 ceph-mon[36520]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T06:33:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:01 smithi022 ceph-mon[32003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T06:33:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:01 smithi022 ceph-mon[32003]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T06:33:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:01 smithi022 ceph-mon[32003]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:01.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:01 smithi022 ceph-mon[32003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:01.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-mon[36262]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T06:33:01.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-mon[36262]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T06:33:01.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:01.361 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-mon[36262]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:01.362 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-mon[36262]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:01.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:01 smithi143 ceph-mon[36520]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T06:33:01.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:01 smithi143 ceph-mon[36520]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T06:33:01.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:01.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:01 smithi143 ceph-mon[36520]: from='osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:01.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:01 smithi143 ceph-mon[36520]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:33:02.027 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5[48599]: 2024-03-24T06:33:01.776+0000 7f207c185700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T06:33:02.027 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:33:01 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5[48599]: 2024-03-24T06:33:01.783+0000 7f2073f70700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:02 smithi022 ceph-mon[32003]: from='osd.5 ' entity='osd.5' 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:02 smithi155 ceph-mon[36262]: from='osd.5 ' entity='osd.5' 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:02 smithi143 ceph-mon[36520]: from='osd.5 ' entity='osd.5' 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] boot 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T06:33:03.375 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi155 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 8540M 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.376 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033453 s, 313 MB/s 2024-03-24T06:33:03.377 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] boot 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T06:33:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi155 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 8540M 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: osd.5 [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] boot 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi155 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 8540M 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:03.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:04.252 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_2 2024-03-24T06:33:04.349 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:04 smithi022 ceph-mon[32003]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:04.349 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:04 smithi022 ceph-mon[32003]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T06:33:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:04 smithi143 ceph-mon[36520]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:04 smithi143 ceph-mon[36520]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T06:33:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:04 smithi155 ceph-mon[36262]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-24T06:33:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:04 smithi155 ceph-mon[36262]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T06:33:04.720 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:33:05.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:05 smithi022 ceph-mon[32003]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T06:33:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:05 smithi022 ceph-mon[32003]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T06:33:05.427 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:05 smithi143 ceph-mon[36520]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T06:33:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:05 smithi143 ceph-mon[36520]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T06:33:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:05 smithi155 ceph-mon[36262]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T06:33:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:05 smithi155 ceph-mon[36262]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T06:33:06.534 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:06 smithi022 ceph-mon[32003]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:06.535 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:06 smithi022 ceph-mon[32003]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T06:33:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:06 smithi143 ceph-mon[36520]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:06 smithi143 ceph-mon[36520]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T06:33:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:06 smithi155 ceph-mon[36262]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:06 smithi155 ceph-mon[36262]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 -- 172.21.15.22:0/1795742037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 msgr2=0x7f034810c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 --2- 172.21.15.22:0/1795742037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f034810c8d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f034000b0a0 tx=0x7f034000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 -- 172.21.15.22:0/1795742037 shutdown_connections 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 --2- 172.21.15.22:0/1795742037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348104480 0x7f03481048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 --2- 172.21.15.22:0/1795742037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f034810c8d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 --2- 172.21.15.22:0/1795742037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348105680 0x7f0348105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 -- 172.21.15.22:0/1795742037 >> 172.21.15.22:0/1795742037 conn(0x7f03480ffa20 msgr2=0x7f0348101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 -- 172.21.15.22:0/1795742037 shutdown_connections 2024-03-24T06:33:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.069+0000 7f034da00700 1 -- 172.21.15.22:0/1795742037 wait complete. 2024-03-24T06:33:07.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.070+0000 7f034da00700 1 Processor -- start 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.070+0000 7f034da00700 1 -- start start 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 0x7f034819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f03481a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f03481a2060 con 0x7f0348105680 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f03481a21a0 con 0x7f0348104480 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f034da00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03481a2330 con 0x7f0348106020 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f0346000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f0346000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46868/0 (socket says 172.21.15.22:46868) 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f0346000700 1 -- 172.21.15.22:0/2437161456 learned_addr learned my addr 172.21.15.22:0/2437161456 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f0346a00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 0x7f034819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.071+0000 7f0347400700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f03481a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 -- 172.21.15.22:0/2437161456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 msgr2=0x7f03481a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f03481a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 -- 172.21.15.22:0/2437161456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 msgr2=0x7f034819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 0x7f034819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 -- 172.21.15.22:0/2437161456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0340009d00 con 0x7f0348105680 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346a00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 0x7f034819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:33:07.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f0346000700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f033400e9d0 tx=0x7f033400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:33:07.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334009790 con 0x7f0348105680 2024-03-24T06:33:07.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.072+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03481a2610 con 0x7f0348105680 2024-03-24T06:33:07.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.073+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033400ee70 con 0x7f0348105680 2024-03-24T06:33:07.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.073+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334018680 con 0x7f0348105680 2024-03-24T06:33:07.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.073+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03481a2b60 con 0x7f0348105680 2024-03-24T06:33:07.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.074+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034810b7e0 con 0x7f0348105680 2024-03-24T06:33:07.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.076+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f033400cbd0 con 0x7f0348105680 2024-03-24T06:33:07.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.077+0000 7f033f400700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 0x7f032c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:07.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.078+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f0334014070 con 0x7f0348105680 2024-03-24T06:33:07.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.078+0000 7f0346a00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 0x7f032c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:07.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.078+0000 7f0346a00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 0x7f032c0a4690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03481054e0 tx=0x7f033800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:33:07.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.081+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0334096fb0 con 0x7f0348105680 2024-03-24T06:33:07.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:07.320+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f034802dc00 con 0x7f032c0a21d0 2024-03-24T06:33:07.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:07.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:07.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:07 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:07.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:07.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:07.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:07 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:07 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:08.619 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:08 smithi022 ceph-mon[32003]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:33:08.619 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:08 smithi022 ceph-mon[32003]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:08.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:08 smithi155 ceph-mon[36262]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:33:08.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:08 smithi155 ceph-mon[36262]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:08 smithi143 ceph-mon[36520]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:33:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:08 smithi143 ceph-mon[36520]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:10.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:10 smithi022 ceph-mon[32003]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T06:33:10.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:10 smithi155 ceph-mon[36262]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T06:33:10.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:10 smithi143 ceph-mon[36520]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-24T06:33:12.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:12 smithi022 ceph-mon[32003]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T06:33:12.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]: dispatch 2024-03-24T06:33:12.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]': finished 2024-03-24T06:33:12.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:12 smithi022 ceph-mon[32003]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T06:33:12.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:12.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:12 smithi155 ceph-mon[36262]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T06:33:12.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]: dispatch 2024-03-24T06:33:12.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]': finished 2024-03-24T06:33:12.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:12 smithi155 ceph-mon[36262]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T06:33:12.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:12 smithi143 ceph-mon[36520]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T06:33:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]: dispatch 2024-03-24T06:33:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2076962347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5"}]': finished 2024-03-24T06:33:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:12 smithi143 ceph-mon[36520]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T06:33:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:13.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:13 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2930849935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:13.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:13 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2930849935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:13.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:13 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2930849935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:14.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:14 smithi155 ceph-mon[36262]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:33:14.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:14 smithi022 ceph-mon[32003]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:33:14.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:14 smithi143 ceph-mon[36520]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:33:16.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:16 smithi022 ceph-mon[32003]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T06:33:16.730 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:16 smithi143 ceph-mon[36520]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T06:33:16.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:16 smithi155 ceph-mon[36262]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-24T06:33:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:18 smithi155 ceph-mon[36262]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:18.872 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:18 smithi022 ceph-mon[32003]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:18.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:18 smithi143 ceph-mon[36520]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:20.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:20 smithi022 ceph-mon[32003]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:20.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:20 smithi155 ceph-mon[36262]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:20.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:20 smithi143 ceph-mon[36520]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T06:33:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:21.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T06:33:21.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T06:33:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:22.565 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:22 smithi022 ceph-mon[32003]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:22.565 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:22 smithi022 ceph-mon[32003]: Deploying daemon osd.6 on smithi022 2024-03-24T06:33:22.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:22 smithi155 ceph-mon[36262]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:22.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:22 smithi155 ceph-mon[36262]: Deploying daemon osd.6 on smithi022 2024-03-24T06:33:22.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:22 smithi143 ceph-mon[36520]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:22.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:22 smithi143 ceph-mon[36520]: Deploying daemon osd.6 on smithi022 2024-03-24T06:33:24.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:24 smithi022 ceph-mon[32003]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:24.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:24 smithi155 ceph-mon[36262]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:24.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:24 smithi143 ceph-mon[36520]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:26.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:26 smithi155 ceph-mon[36262]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:26.863 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:26 smithi022 ceph-mon[32003]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:26.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:26 smithi143 ceph-mon[36520]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:28 smithi155 ceph-mon[36262]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:28.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:28 smithi022 ceph-mon[32003]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:28 smithi143 ceph-mon[36520]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:30 smithi022 ceph-mon[32003]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:30 smithi155 ceph-mon[36262]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:30.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:30 smithi143 ceph-mon[36520]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:31.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:32.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:32.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:32.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:32.853 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:32 smithi022 ceph-mon[32003]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:32 smithi143 ceph-mon[36520]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:33.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:32 smithi155 ceph-mon[36262]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:34.788 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:34 smithi022 ceph-mon[32003]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:34.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:34.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:34.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:34 smithi022 ceph-mon[32003]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:34.789 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:34 smithi022 ceph-mon[32003]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:34 smithi143 ceph-mon[36520]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:34 smithi143 ceph-mon[36520]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:34 smithi143 ceph-mon[36520]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:34 smithi155 ceph-mon[36262]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:34 smithi155 ceph-mon[36262]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:35.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:34 smithi155 ceph-mon[36262]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T06:33:35.880 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 6 on host 'smithi022' 2024-03-24T06:33:35.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.879+0000 7f033f400700 1 -- 172.21.15.22:0/2437161456 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f034802dc00 con 0x7f032c0a21d0 2024-03-24T06:33:35.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.882+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 msgr2=0x7f032c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:35.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.882+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 0x7f032c0a4690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f03481054e0 tx=0x7f033800b410 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.882+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 msgr2=0x7f034819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.882+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f033400e9d0 tx=0x7f033400ece0 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 shutdown_connections 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f032c0a21d0 0x7f032c0a4690 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0348106020 0x7f03481a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0348104480 0x7f034819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 --2- 172.21.15.22:0/2437161456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0348105680 0x7f034819aa10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 >> 172.21.15.22:0/2437161456 conn(0x7f03480ffa20 msgr2=0x7f0348108920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 shutdown_connections 2024-03-24T06:33:35.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:33:35.883+0000 7f034da00700 1 -- 172.21.15.22:0/2437161456 wait complete. 2024-03-24T06:33:36.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:36.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T06:33:36.518 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.519 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:36.736 DEBUG:teuthology.orchestra.run.smithi022:osd.6> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T06:33:36.740 INFO:tasks.cephadm:Deploying osd.7 on smithi143 with /dev/vg_nvme/lv_2... 2024-03-24T06:33:36.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T06:33:36.769 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:36.769 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T06:33:36.769 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T06:33:36.769 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:36.769 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.770 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T06:33:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T06:33:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:36.982 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:33:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:37 smithi143 ceph-mon[36520]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:33:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:37 smithi143 ceph-mon[36520]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T06:33:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:37 smithi155 ceph-mon[36262]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:33:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:37 smithi155 ceph-mon[36262]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T06:33:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:37.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:37 smithi022 ceph-mon[32003]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:33:37.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:37 smithi022 ceph-mon[32003]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T06:33:37.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:37.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] boot 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T06:33:38.729 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:38 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] boot 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T06:33:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:38 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: osd.6 [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] boot 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T06:33:38.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:38 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113303 s, 925 MB/s 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 2024-03-24T06:33:39.003 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T06:33:39.578 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:39 smithi143 ceph-mon[36520]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T06:33:39.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_2 2024-03-24T06:33:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:39 smithi155 ceph-mon[36262]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T06:33:39.890 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:33:39.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:39 smithi022 ceph-mon[32003]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T06:33:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:40 smithi143 ceph-mon[36520]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:40.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:40 smithi155 ceph-mon[36262]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:40 smithi022 ceph-mon[32003]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- 172.21.15.143:0/2405093102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc102330 msgr2=0x7ff2bc102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 --2- 172.21.15.143:0/2405093102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc102330 0x7ff2bc102790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff2b00099c0 tx=0x7ff2b0009cd0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- 172.21.15.143:0/2405093102 shutdown_connections 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 --2- 172.21.15.143:0/2405093102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2bc102cd0 0x7ff2bc109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 --2- 172.21.15.143:0/2405093102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc101130 0x7ff2bc101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 --2- 172.21.15.143:0/2405093102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc102330 0x7ff2bc102790 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- 172.21.15.143:0/2405093102 >> 172.21.15.143:0/2405093102 conn(0x7ff2bc0fc6f0 msgr2=0x7ff2bc0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- 172.21.15.143:0/2405093102 shutdown_connections 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- 172.21.15.143:0/2405093102 wait complete. 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 Processor -- start 2024-03-24T06:33:41.460 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.459+0000 7ff2c2892700 1 -- start start 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc102330 0x7ff2bc19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2bc102cd0 0x7ff2bc19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc19fd50 con 0x7ff2bc102cd0 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc19fec0 con 0x7ff2bc101130 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1a0080 con 0x7ff2bc102330 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:41.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36400/0 (socket says 172.21.15.143:36400) 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 -- 172.21.15.143:0/3163784099 learned_addr learned my addr 172.21.15.143:0/3163784099 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bb7fe700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc102330 0x7ff2bc19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 -- 172.21.15.143:0/3163784099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc102330 msgr2=0x7ff2bc19cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc102330 0x7ff2bc19cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 -- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2bc102cd0 msgr2=0x7ff2bc19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2bc102cd0 0x7ff2bc19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 -- 172.21.15.143:0/3163784099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b00095f0 con 0x7ff2bc101130 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2bbfff700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00d980 tx=0x7ff2ac00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac004020 con 0x7ff2bc101130 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2ac009730 con 0x7ff2bc101130 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac003be0 con 0x7ff2bc101130 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.460+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc1a0360 con 0x7ff2bc101130 2024-03-24T06:33:41.463 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.461+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a0830 con 0x7ff2bc101130 2024-03-24T06:33:41.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.462+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc108570 con 0x7ff2bc101130 2024-03-24T06:33:41.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.462+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff2ac009ba0 con 0x7ff2bc101130 2024-03-24T06:33:41.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.463+0000 7ff2b97fa700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 0x7ff2a40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:33:41.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.463+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7ff2ac0ccc40 con 0x7ff2bc101130 2024-03-24T06:33:41.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.464+0000 7ff2bb7fe700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 0x7ff2a40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:33:41.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.465+0000 7ff2bb7fe700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 0x7ff2a40a4610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009990 tx=0x7ff2b0005260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:33:41.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.468+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff2ac096920 con 0x7ff2bc101130 2024-03-24T06:33:41.672 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:33:41.672+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff2bc02dc00 con 0x7ff2a40a2150 2024-03-24T06:33:42.619 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi022 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 5698M 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.620 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.621 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:42.621 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:42 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi022 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 5698M 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:42.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:42 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi022 2024-03-24T06:33:42.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 5698M 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:33:42.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:42 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/3286448840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]': finished 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T06:33:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/3286448840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]': finished 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T06:33:44.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/3286448840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]: dispatch 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f669e98-5416-456f-8f7c-e3d8fd6608c6"}]': finished 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T06:33:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:33:45.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:45 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/1181149134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:45.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:45 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/1181149134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:46.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:45 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/1181149134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:33:46.829 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:46 smithi143 ceph-mon[36520]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:46.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:46 smithi022 ceph-mon[32003]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:47.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:46 smithi155 ceph-mon[36262]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:48.634 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:48 smithi143 ceph-mon[36520]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:48 smithi022 ceph-mon[32003]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:49.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:48 smithi155 ceph-mon[36262]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:50 smithi022 ceph-mon[32003]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:50 smithi143 ceph-mon[36520]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:50 smithi155 ceph-mon[36262]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:52 smithi022 ceph-mon[32003]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T06:33:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:52.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:52 smithi022 ceph-mon[32003]: Deploying daemon osd.7 on smithi143 2024-03-24T06:33:52.899 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:52 smithi143 ceph-mon[36520]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:52.900 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T06:33:52.900 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:52.900 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:52 smithi143 ceph-mon[36520]: Deploying daemon osd.7 on smithi143 2024-03-24T06:33:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:52 smithi155 ceph-mon[36262]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T06:33:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:33:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:52 smithi155 ceph-mon[36262]: Deploying daemon osd.7 on smithi143 2024-03-24T06:33:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:54 smithi143 ceph-mon[36520]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:54.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:54 smithi022 ceph-mon[32003]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:54 smithi155 ceph-mon[36262]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:56.804 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:56 smithi143 ceph-mon[36520]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:56.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:56 smithi022 ceph-mon[32003]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:56 smithi155 ceph-mon[36262]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:58 smithi022 ceph-mon[32003]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:33:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:58 smithi143 ceph-mon[36520]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:33:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:58 smithi155 ceph-mon[36262]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:33:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:33:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:33:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:33:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:00.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:00 smithi143 ceph-mon[36520]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:00.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:00 smithi022 ceph-mon[32003]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:00 smithi155 ceph-mon[36262]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:00.963 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 7 on host 'smithi143' 2024-03-24T06:34:00.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.963+0000 7ff2b97fa700 1 -- 172.21.15.143:0/3163784099 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff2bc02dc00 con 0x7ff2a40a2150 2024-03-24T06:34:00.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 msgr2=0x7ff2a40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:00.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 0x7ff2a40a4610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009990 tx=0x7ff2b0005260 comp rx=0 tx=0).stop 2024-03-24T06:34:00.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 msgr2=0x7ff2bc06b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00d980 tx=0x7ff2ac00dc90 comp rx=0 tx=0).stop 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 shutdown_connections 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2bc102cd0 0x7ff2bc19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff2a40a2150 0x7ff2a40a4610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc101130 0x7ff2bc06b4a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 --2- 172.21.15.143:0/3163784099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2bc102330 0x7ff2bc19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 >> 172.21.15.143:0/3163784099 conn(0x7ff2bc0fc6f0 msgr2=0x7ff2bc1056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 shutdown_connections 2024-03-24T06:34:00.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:34:00.966+0000 7ff2c2892700 1 -- 172.21.15.143:0/3163784099 wait complete. 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:01 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:01 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:01.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:01 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:01.606 DEBUG:teuthology.orchestra.run.smithi143:osd.7> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T06:34:01.609 INFO:tasks.cephadm:Deploying osd.8 on smithi155 with /dev/vg_nvme/lv_2... 2024-03-24T06:34:01.609 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T06:34:01.856 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:34:03.093 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:03.094 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T06:34:03.094 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.094 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T06:34:03.094 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:03.094 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-mon[36520]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.094 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7[54601]: 2024-03-24T06:34:02.929+0000 7f2d8be61700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T06:34:03.094 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:34:02 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7[54601]: 2024-03-24T06:34:02.940+0000 7f2d87c54700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:03.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:02 smithi155 ceph-mon[36262]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:03.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T06:34:03.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: from='osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T06:34:03.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:03.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:02 smithi022 ceph-mon[32003]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012321 s, 851 MB/s 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 2024-03-24T06:34:03.832 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T06:34:04.166 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:03 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:34:04.166 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:03 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:34:04.166 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:03 smithi155 ceph-mon[36262]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:34:04.167 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:03 smithi155 ceph-mon[36262]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T06:34:04.167 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:03 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:34:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:03 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:34:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:03 smithi143 ceph-mon[36520]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:34:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:03 smithi143 ceph-mon[36520]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T06:34:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:04.300 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:03 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:34:04.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:03 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:34:04.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:03 smithi022 ceph-mon[32003]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:34:04.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:03 smithi022 ceph-mon[32003]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T06:34:04.301 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:04.463 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-03-24T06:34:04.707 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:34:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] boot 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:05.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] boot 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:05.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: osd.7 [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] boot 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:05.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:05.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:05.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:05.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:06.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:05 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi143 2024-03-24T06:34:06.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:05 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 5693M 2024-03-24T06:34:06.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:05 smithi155 ceph-mon[36262]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T06:34:06.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:05 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi143 2024-03-24T06:34:06.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:05 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 5693M 2024-03-24T06:34:06.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:05 smithi022 ceph-mon[32003]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 -- 172.21.15.155:0/3403514759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102cd0 msgr2=0x7f10e8109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 --2- 172.21.15.155:0/3403514759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102cd0 0x7f10e8109580 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f10e000b0a0 tx=0x7f10e000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 -- 172.21.15.155:0/3403514759 shutdown_connections 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 --2- 172.21.15.155:0/3403514759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102cd0 0x7f10e8109580 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 --2- 172.21.15.155:0/3403514759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 0x7f10e8101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 --2- 172.21.15.155:0/3403514759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10e8102330 0x7f10e8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.422+0000 7f10efd34700 1 -- 172.21.15.155:0/3403514759 >> 172.21.15.155:0/3403514759 conn(0x7f10e80fc6f0 msgr2=0x7f10e80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- 172.21.15.155:0/3403514759 shutdown_connections 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- 172.21.15.155:0/3403514759 wait complete. 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 Processor -- start 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- start start 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 0x7f10e806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10e8102cd0 0x7f10e819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f10e819ff20 con 0x7f10e8102cd0 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f10e81a0090 con 0x7f10e8101130 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.423+0000 7f10efd34700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10e81a0250 con 0x7f10e8102330 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:45306/0 (socket says 172.21.15.155:45306) 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 -- 172.21.15.155:0/2249112407 learned_addr learned my addr 172.21.15.155:0/2249112407 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10edad0700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 0x7f10e806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:06.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 -- 172.21.15.155:0/2249112407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 msgr2=0x7f10e806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 0x7f10e806b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 -- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10e8102cd0 msgr2=0x7f10e819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10e8102cd0 0x7f10e819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 -- 172.21.15.155:0/2249112407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e0009d00 con 0x7f10e8102330 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10ed2cf700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b530 tx=0x7f10e400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4011830 con 0x7f10e8102330 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e4011e70 con 0x7f10e8102330 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e81a0530 con 0x7f10e8102330 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e400f2e0 con 0x7f10e8102330 2024-03-24T06:34:06.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.424+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e81a0ab0 con 0x7f10e8102330 2024-03-24T06:34:06.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.425+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e81084f0 con 0x7f10e8102330 2024-03-24T06:34:06.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.426+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f10e40105f0 con 0x7f10e8102330 2024-03-24T06:34:06.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.427+0000 7f10deffd700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 0x7f10d40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:06.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.427+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f10e40cd140 con 0x7f10e8102330 2024-03-24T06:34:06.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.427+0000 7f10edad0700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 0x7f10d40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:05 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi143 2024-03-24T06:34:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:05 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 5693M 2024-03-24T06:34:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:05 smithi143 ceph-mon[36520]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T06:34:06.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.428+0000 7f10edad0700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 0x7f10d40a4610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10d8006fd0 tx=0x7f10d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:34:06.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.431+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f10e4096c50 con 0x7f10e8102330 2024-03-24T06:34:06.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:06.635+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f10e802dc70 con 0x7f10d40a2150 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:07.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:06 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:07.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:06 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:07.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:07.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T06:34:07.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T06:34:07.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:07.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:07.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:06 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:07 smithi155 ceph-mon[36262]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:07 smithi155 ceph-mon[36262]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:07 smithi155 ceph-mon[36262]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T06:34:08.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:07 smithi022 ceph-mon[32003]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:08.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:07 smithi022 ceph-mon[32003]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:08.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:07 smithi022 ceph-mon[32003]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T06:34:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:07 smithi143 ceph-mon[36520]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:07 smithi143 ceph-mon[36520]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:07 smithi143 ceph-mon[36520]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T06:34:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:09 smithi022 ceph-mon[32003]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T06:34:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:09 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:34:09.427 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:09 smithi143 ceph-mon[36520]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T06:34:09.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:09 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:34:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:09 smithi155 ceph-mon[36262]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T06:34:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:09 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/1364631238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]': finished 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/2871476239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/1364631238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]': finished 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T06:34:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:10.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/2871476239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/1364631238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]: dispatch 2024-03-24T06:34:10.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75386ae9-abf1-4061-b5a6-ae879ab75235"}]': finished 2024-03-24T06:34:10.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T06:34:10.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:10.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/2871476239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:12 smithi022 ceph-mon[32003]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:12 smithi143 ceph-mon[36520]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:12 smithi155 ceph-mon[36262]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:14 smithi155 ceph-mon[36262]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:14 smithi143 ceph-mon[36520]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:14.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:14 smithi022 ceph-mon[32003]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:16 smithi155 ceph-mon[36262]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:16.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:16 smithi022 ceph-mon[32003]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:16.427 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:16 smithi143 ceph-mon[36520]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:17.418 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T06:34:17.418 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:17.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T06:34:17.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:17.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T06:34:17.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:18 smithi143 ceph-mon[36520]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:18 smithi143 ceph-mon[36520]: Deploying daemon osd.8 on smithi155 2024-03-24T06:34:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:18 smithi155 ceph-mon[36262]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:18 smithi155 ceph-mon[36262]: Deploying daemon osd.8 on smithi155 2024-03-24T06:34:18.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:18 smithi022 ceph-mon[32003]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:18.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:18 smithi022 ceph-mon[32003]: Deploying daemon osd.8 on smithi155 2024-03-24T06:34:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:20 smithi143 ceph-mon[36520]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:20 smithi155 ceph-mon[36262]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:20.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:20 smithi022 ceph-mon[32003]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:22.355 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:22 smithi155 ceph-mon[36262]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:22.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:22 smithi022 ceph-mon[32003]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:22 smithi143 ceph-mon[36520]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:23 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:23 smithi155 ceph-mon[36262]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:23 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:23 smithi022 ceph-mon[32003]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:23 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:23 smithi143 ceph-mon[36520]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:26.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.977+0000 7f10deffd700 1 -- 172.21.15.155:0/2249112407 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f10e802dc70 con 0x7f10d40a2150 2024-03-24T06:34:26.977 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 8 on host 'smithi155' 2024-03-24T06:34:26.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 msgr2=0x7f10d40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 0x7f10d40a4610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10d8006fd0 tx=0x7f10d8008040 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 msgr2=0x7f10e819d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b530 tx=0x7f10e400b8f0 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 shutdown_connections 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10e8102cd0 0x7f10e819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f10d40a2150 0x7f10d40a4610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e8101130 0x7f10e806b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 --2- 172.21.15.155:0/2249112407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10e8102330 0x7f10e819d0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 >> 172.21.15.155:0/2249112407 conn(0x7f10e80fc6f0 msgr2=0x7f10e8105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 shutdown_connections 2024-03-24T06:34:26.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:34:26.979+0000 7f10efd34700 1 -- 172.21.15.155:0/2249112407 wait complete. 2024-03-24T06:34:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:27 smithi143 ceph-mon[36520]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:27 smithi143 ceph-mon[36520]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:27 smithi022 ceph-mon[32003]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:27 smithi022 ceph-mon[32003]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:27 smithi155 ceph-mon[36262]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:27 smithi155 ceph-mon[36262]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T06:34:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:27.613 DEBUG:teuthology.orchestra.run.smithi155:osd.8> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T06:34:27.616 INFO:tasks.cephadm:Deploying osd.9 on smithi022 with /dev/vg_nvme/lv_1... 2024-03-24T06:34:27.616 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T06:34:28.111 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:34:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T06:34:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T06:34:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-mon[36262]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T06:34:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T06:34:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:28 smithi022 ceph-mon[32003]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: from='osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:28 smithi143 ceph-mon[36520]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:34:29.130 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8[55486]: 2024-03-24T06:34:28.779+0000 7f3acd448700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T06:34:29.130 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:34:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8[55486]: 2024-03-24T06:34:28.788+0000 7f3ac4a32700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:34:29.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:29 smithi022 ceph-mon[32003]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:34:29.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:29 smithi022 ceph-mon[32003]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T06:34:29.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:29 smithi022 ceph-mon[32003]: from='osd.8 ' entity='osd.8' 2024-03-24T06:34:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:29 smithi143 ceph-mon[36520]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:34:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:29 smithi143 ceph-mon[36520]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T06:34:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:29 smithi143 ceph-mon[36520]: from='osd.8 ' entity='osd.8' 2024-03-24T06:34:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:29 smithi155 ceph-mon[36262]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:34:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:29 smithi155 ceph-mon[36262]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T06:34:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:29 smithi155 ceph-mon[36262]: from='osd.8 ' entity='osd.8' 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] boot 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:30 smithi155 ceph-mon[36262]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T06:34:30.357 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:34:30.357 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:34:30.357 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:30.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] boot 2024-03-24T06:34:30.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T06:34:30.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:30.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:30 smithi022 ceph-mon[32003]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: osd.8 [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] boot 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:34:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:30 smithi143 ceph-mon[36520]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0373799 s, 281 MB/s 2024-03-24T06:34:31.258 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T06:34:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T06:34:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:31.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:31.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:31.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:31.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:34:31.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:31.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:31.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:34:31.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:32.094 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_1 2024-03-24T06:34:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:34:32.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:32 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi155 2024-03-24T06:34:32.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:32 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 5693M 2024-03-24T06:34:32.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:32 smithi022 ceph-mon[32003]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:32.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:32 smithi022 ceph-mon[32003]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T06:34:32.739 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:32 smithi022 ceph-mon[32003]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T06:34:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:32 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi155 2024-03-24T06:34:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:32 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 5693M 2024-03-24T06:34:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:32 smithi155 ceph-mon[36262]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:32 smithi155 ceph-mon[36262]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T06:34:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:32 smithi155 ceph-mon[36262]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T06:34:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:32 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi155 2024-03-24T06:34:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:32 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 5693M 2024-03-24T06:34:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:32 smithi143 ceph-mon[36520]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:32 smithi143 ceph-mon[36520]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T06:34:32.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:32 smithi143 ceph-mon[36520]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T06:34:34.516 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:34 smithi022 ceph-mon[32003]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:34 smithi155 ceph-mon[36262]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:34.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 -- 172.21.15.22:0/1742451972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0105680 msgr2=0x7fa8d0105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:34.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 --2- 172.21.15.22:0/1742451972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0105680 0x7fa8d0105ae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0099c0 tx=0x7fa8bc009cd0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 -- 172.21.15.22:0/1742451972 shutdown_connections 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 --2- 172.21.15.22:0/1742451972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0106020 0x7fa8d010c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 --2- 172.21.15.22:0/1742451972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0104480 0x7fa8d01048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 --2- 172.21.15.22:0/1742451972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0105680 0x7fa8d0105ae0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 -- 172.21.15.22:0/1742451972 >> 172.21.15.22:0/1742451972 conn(0x7fa8d00ffa20 msgr2=0x7fa8d0101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 -- 172.21.15.22:0/1742451972 shutdown_connections 2024-03-24T06:34:34.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.829+0000 7fa8d5600700 1 -- 172.21.15.22:0/1742451972 wait complete. 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.830+0000 7fa8d5600700 1 Processor -- start 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 -- start start 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 0x7fa8d019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0106020 0x7fa8d01a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01a6300 con 0x7fa8d0106020 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01a6440 con 0x7fa8d0104480 2024-03-24T06:34:34.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8d5600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d01a65d0 con 0x7fa8d0105680 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8cea00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.831+0000 7fa8cea00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:45276/0 (socket says 172.21.15.22:45276) 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8cea00700 1 -- 172.21.15.22:0/2531451755 learned_addr learned my addr 172.21.15.22:0/2531451755 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 0x7fa8d019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8cf400700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0106020 0x7fa8d01a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 -- 172.21.15.22:0/2531451755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 msgr2=0x7fa8d019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 -- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0106020 msgr2=0x7fa8d01a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0106020 0x7fa8d01a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 -- 172.21.15.22:0/2531451755 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc0095f0 con 0x7fa8d0105680 2024-03-24T06:34:34.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8cea00700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:34:34.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.832+0000 7fa8ce000700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 0x7fa8d019ecb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc003760 tx=0x7fa8bc0055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:34:34.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.833+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc01d070 con 0x7fa8d0105680 2024-03-24T06:34:34.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.833+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8bc003e90 con 0x7fa8d0105680 2024-03-24T06:34:34.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.833+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d01a6850 con 0x7fa8d0105680 2024-03-24T06:34:34.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.833+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc00fc20 con 0x7fa8d0105680 2024-03-24T06:34:34.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.833+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d01a6d40 con 0x7fa8d0105680 2024-03-24T06:34:34.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.834+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d0068ec0 con 0x7fa8d0105680 2024-03-24T06:34:34.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.836+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa8bc00b9a0 con 0x7fa8d0105680 2024-03-24T06:34:34.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.837+0000 7fa8c7400700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 0x7fa8b40a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:34:34.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.838+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fa8bc0ce220 con 0x7fa8d0105680 2024-03-24T06:34:34.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.838+0000 7fa8cea00700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 0x7fa8b40a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:34:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.838+0000 7fa8cea00700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 0x7fa8b40a4690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005950 tx=0x7fa8c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:34:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:34.840+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa8bc097cb0 con 0x7fa8d0105680 2024-03-24T06:34:34.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:34 smithi143 ceph-mon[36520]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:35.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:34:35.079+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa8d0079d70 con 0x7fa8b40a21d0 2024-03-24T06:34:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:35 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:35 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:35 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:35.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:35.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:35.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:35 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:35 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:34:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:35 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:34:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:35 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:36.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:36 smithi022 ceph-mon[32003]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T06:34:36.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:36 smithi022 ceph-mon[32003]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:36 smithi155 ceph-mon[36262]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T06:34:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:36 smithi155 ceph-mon[36262]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:36 smithi143 ceph-mon[36520]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-24T06:34:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:36 smithi143 ceph-mon[36520]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:34:38.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:38 smithi155 ceph-mon[36262]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T06:34:38.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:38 smithi022 ceph-mon[32003]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T06:34:38.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:38 smithi143 ceph-mon[36520]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T06:34:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:39 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]: dispatch 2024-03-24T06:34:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:39 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]': finished 2024-03-24T06:34:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:39 smithi155 ceph-mon[36262]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T06:34:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:34:39.882 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:39 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]: dispatch 2024-03-24T06:34:39.882 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:39 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]': finished 2024-03-24T06:34:39.882 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:39 smithi022 ceph-mon[32003]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T06:34:39.882 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:34:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:39 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]: dispatch 2024-03-24T06:34:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:39 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4101927355' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0937ff6c-3ac5-4da0-b56c-f8286d053eff"}]': finished 2024-03-24T06:34:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:39 smithi143 ceph-mon[36520]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T06:34:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:34:40.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:40 smithi155 ceph-mon[36262]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:34:40.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:40 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/529811383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:40 smithi022 ceph-mon[32003]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:34:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:40 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/529811383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:40.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:40 smithi143 ceph-mon[36520]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T06:34:40.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:40 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/529811383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:34:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:42 smithi155 ceph-mon[36262]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T06:34:42.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:42 smithi022 ceph-mon[32003]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T06:34:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:42 smithi143 ceph-mon[36520]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-24T06:34:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:44 smithi155 ceph-mon[36262]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:34:44.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:44 smithi022 ceph-mon[32003]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:34:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:44 smithi143 ceph-mon[36520]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T06:34:46.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:46 smithi155 ceph-mon[36262]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:46.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:46 smithi022 ceph-mon[32003]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:46.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:46 smithi143 ceph-mon[36520]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:48 smithi155 ceph-mon[36262]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:48 smithi022 ceph-mon[32003]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:48.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:48 smithi143 ceph-mon[36520]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:49.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T06:34:49.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:49.927 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T06:34:49.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T06:34:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:34:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:50 smithi022 ceph-mon[32003]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:50 smithi022 ceph-mon[32003]: Deploying daemon osd.9 on smithi022 2024-03-24T06:34:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:50 smithi143 ceph-mon[36520]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:50 smithi143 ceph-mon[36520]: Deploying daemon osd.9 on smithi022 2024-03-24T06:34:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:50 smithi155 ceph-mon[36262]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:50 smithi155 ceph-mon[36262]: Deploying daemon osd.9 on smithi022 2024-03-24T06:34:52.647 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:52 smithi022 ceph-mon[32003]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:52 smithi143 ceph-mon[36520]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:52 smithi155 ceph-mon[36262]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:54.640 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:54 smithi022 ceph-mon[32003]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:54 smithi143 ceph-mon[36520]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:54 smithi155 ceph-mon[36262]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:56.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:56 smithi022 ceph-mon[32003]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:56 smithi143 ceph-mon[36520]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:56 smithi155 ceph-mon[36262]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:58 smithi022 ceph-mon[32003]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:34:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:58 smithi143 ceph-mon[36520]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:34:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:58 smithi155 ceph-mon[36262]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:34:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:34:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:34:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:34:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:00.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:00 smithi022 ceph-mon[32003]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:00 smithi143 ceph-mon[36520]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:00 smithi155 ceph-mon[36262]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:02.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:02 smithi022 ceph-mon[32003]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:02.820 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:02 smithi022 ceph-mon[32003]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T06:35:02.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:02 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:02.821 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:02 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:02 smithi143 ceph-mon[36520]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:02 smithi143 ceph-mon[36520]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T06:35:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:02 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:02 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:02 smithi155 ceph-mon[36262]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:02 smithi155 ceph-mon[36262]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T06:35:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:02 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:02 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.082 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 9 on host 'smithi022' 2024-03-24T06:35:03.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.081+0000 7fa8c7400700 1 -- 172.21.15.22:0/2531451755 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa8d0079d70 con 0x7fa8b40a21d0 2024-03-24T06:35:03.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.084+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 msgr2=0x7fa8b40a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.084+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 0x7fa8b40a4690 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0005950 tx=0x7fa8c0009260 comp rx=0 tx=0).stop 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 msgr2=0x7fa8d019ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 0x7fa8d019ecb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc003760 tx=0x7fa8bc0055f0 comp rx=0 tx=0).stop 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 shutdown_connections 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8d0106020 0x7fa8d01a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8b40a21d0 0x7fa8b40a4690 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:03.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d0104480 0x7fa8d019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 --2- 172.21.15.22:0/2531451755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8d0105680 0x7fa8d019ecb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 >> 172.21.15.22:0/2531451755 conn(0x7fa8d00ffa20 msgr2=0x7fa8d0078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 shutdown_connections 2024-03-24T06:35:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:03.085+0000 7fa8d5600700 1 -- 172.21.15.22:0/2531451755 wait complete. 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.823 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:03 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.882 DEBUG:teuthology.orchestra.run.smithi022:osd.9> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T06:35:03.887 INFO:tasks.cephadm:Deploying osd.10 on smithi143 with /dev/vg_nvme/lv_1... 2024-03-24T06:35:03.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:03.917 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:03 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:04.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:03 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:04.132 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:35:04.415 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:35:04.342+0000 7f4fa0c00700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T06:35:04.415 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:35:04.374+0000 7f4f96a00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:04.738 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:04 smithi022 ceph-mon[32003]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:04 smithi143 ceph-mon[36520]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:04 smithi155 ceph-mon[36262]: from='osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113]' entity='osd.9' 2024-03-24T06:35:05.850 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:35:05.850 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:35:05.850 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:05.850 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] boot 2024-03-24T06:35:05.850 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T06:35:05.851 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:05 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:05.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:35:05.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:35:05.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:05.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] boot 2024-03-24T06:35:05.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T06:35:05.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:05 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:06.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:35:06.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:35:06.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:06.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: osd.9 [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] boot 2024-03-24T06:35:06.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T06:35:06.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:05 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:35:06.219 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T06:35:06.220 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T06:35:06.220 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T06:35:06.220 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T06:35:06.220 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113786 s, 922 MB/s 2024-03-24T06:35:06.220 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T06:35:06.872 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:06 smithi143 ceph-mon[36520]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:06.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:06 smithi022 ceph-mon[32003]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:06.900 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_1 2024-03-24T06:35:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:06 smithi155 ceph-mon[36262]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-24T06:35:07.182 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:35:07.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:07 smithi022 ceph-mon[32003]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T06:35:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:07 smithi143 ceph-mon[36520]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T06:35:08.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:07 smithi155 ceph-mon[36262]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T06:35:08.806 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- 172.21.15.143:0/1126472157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee40101ad0 msgr2=0x7fee40101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- 172.21.15.143:0/1126472157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee40101ad0 0x7fee40101f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee300099c0 tx=0x7fee30009cd0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- 172.21.15.143:0/1126472157 shutdown_connections 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- 172.21.15.143:0/1126472157 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee401008d0 0x7fee40100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- 172.21.15.143:0/1126472157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 0x7fee40108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- 172.21.15.143:0/1126472157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee40101ad0 0x7fee40101f30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- 172.21.15.143:0/1126472157 >> 172.21.15.143:0/1126472157 conn(0x7fee400fbe50 msgr2=0x7fee400fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- 172.21.15.143:0/1126472157 shutdown_connections 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- 172.21.15.143:0/1126472157 wait complete. 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 Processor -- start 2024-03-24T06:35:08.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- start start 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee40101ad0 0x7fee40198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 0x7fee4019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fee4019fdb0 con 0x7fee40101ad0 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fee4019fef0 con 0x7fee401008d0 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.805+0000 7fee48161700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fee401a0080 con 0x7fee40102470 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54772/0 (socket says 172.21.15.143:54772) 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 -- 172.21.15.143:0/1232333768 learned_addr learned my addr 172.21.15.143:0/1232333768 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee466fe700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 0x7fee4019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:08.808 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 -- 172.21.15.143:0/1232333768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 msgr2=0x7fee4019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 0x7fee4019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 -- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee40101ad0 msgr2=0x7fee40198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee456fc700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee40101ad0 0x7fee40198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee40101ad0 0x7fee40198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 -- 172.21.15.143:0/1232333768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee300095f0 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee45efd700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00b500 tx=0x7fee3c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c011830 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c011e70 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee401a0360 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c00f2e0 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.806+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee401a0860 con 0x7fee401008d0 2024-03-24T06:35:08.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.807+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee40068ec0 con 0x7fee401008d0 2024-03-24T06:35:08.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.808+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fee3c0105f0 con 0x7fee401008d0 2024-03-24T06:35:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.809+0000 7fee36ffd700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 0x7fee2c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.809+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fee3c0ccee0 con 0x7fee401008d0 2024-03-24T06:35:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.809+0000 7fee456fc700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 0x7fee2c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:08.811 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.810+0000 7fee456fc700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 0x7fee2c0a4610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee30006010 tx=0x7fee30009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:08.813 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:08.814+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fee3c0966b0 con 0x7fee401008d0 2024-03-24T06:35:08.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:08 smithi022 ceph-mon[32003]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:08 smithi143 ceph-mon[36520]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:09.018 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:09.019+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fee40107f20 con 0x7fee2c0a2150 2024-03-24T06:35:09.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:08 smithi155 ceph-mon[36262]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.897 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:09.898 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:09 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:09.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:09 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:10.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:09 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:10.866 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:10 smithi143 ceph-mon[36520]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:10.866 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:10 smithi143 ceph-mon[36520]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:10.866 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:10 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi022 2024-03-24T06:35:10.867 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:10 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 4273M 2024-03-24T06:35:10.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:10 smithi022 ceph-mon[32003]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:10.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:10 smithi022 ceph-mon[32003]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:10 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi022 2024-03-24T06:35:10.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:10 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 4273M 2024-03-24T06:35:10.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:10 smithi155 ceph-mon[36262]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:10.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:10 smithi155 ceph-mon[36262]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:10.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:10 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi022 2024-03-24T06:35:10.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:10 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 4273M 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/2119664838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]': finished 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T06:35:12.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:12.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.143:0/3504693494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/2119664838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]': finished 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:12.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.143:0/3504693494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:13.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:13.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/2119664838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:13.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]: dispatch 2024-03-24T06:35:13.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "095f0ee4-d45c-48ea-961d-f60c8c0869bb"}]': finished 2024-03-24T06:35:13.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T06:35:13.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:13.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.143:0/3504693494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:14.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:14 smithi022 ceph-mon[32003]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:14.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:14 smithi143 ceph-mon[36520]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:15.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:14 smithi155 ceph-mon[36262]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:16.888 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:16 smithi143 ceph-mon[36520]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:16.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:16 smithi022 ceph-mon[32003]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:17.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:16 smithi155 ceph-mon[36262]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:18.890 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:18 smithi143 ceph-mon[36520]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:18.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:18 smithi022 ceph-mon[32003]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:19.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:18 smithi155 ceph-mon[36262]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:19.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:19 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T06:35:19.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:19 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:20.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:19 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T06:35:20.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:19 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:20.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:19 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T06:35:20.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:19 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:20.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:20 smithi143 ceph-mon[36520]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:20.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:20 smithi143 ceph-mon[36520]: Deploying daemon osd.10 on smithi143 2024-03-24T06:35:21.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:20 smithi155 ceph-mon[36262]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:21.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:20 smithi155 ceph-mon[36262]: Deploying daemon osd.10 on smithi143 2024-03-24T06:35:21.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:20 smithi022 ceph-mon[32003]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:21.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:20 smithi022 ceph-mon[32003]: Deploying daemon osd.10 on smithi143 2024-03-24T06:35:22.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:22 smithi143 ceph-mon[36520]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:23.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:22 smithi155 ceph-mon[36262]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:23.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:22 smithi022 ceph-mon[32003]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:24.703 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:24 smithi143 ceph-mon[36520]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:25.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:24 smithi155 ceph-mon[36262]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:25.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:24 smithi022 ceph-mon[32003]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:26 smithi022 ceph-mon[32003]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:26 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:26 smithi143 ceph-mon[36520]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:26 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:26 smithi155 ceph-mon[36262]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:26 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:28 smithi022 ceph-mon[32003]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:28 smithi143 ceph-mon[36520]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:28 smithi155 ceph-mon[36262]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:28.907 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 10 on host 'smithi143' 2024-03-24T06:35:28.908 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.907+0000 7fee36ffd700 1 -- 172.21.15.143:0/1232333768 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fee40107f20 con 0x7fee2c0a2150 2024-03-24T06:35:28.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 msgr2=0x7fee2c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:28.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 0x7fee2c0a4610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee30006010 tx=0x7fee30009f10 comp rx=0 tx=0).stop 2024-03-24T06:35:28.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 msgr2=0x7fee40198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00b500 tx=0x7fee3c00b810 comp rx=0 tx=0).stop 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 shutdown_connections 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fee2c0a2150 0x7fee2c0a4610 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee40102470 0x7fee4019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee401008d0 0x7fee40198220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 --2- 172.21.15.143:0/1232333768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee40101ad0 0x7fee40198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 >> 172.21.15.143:0/1232333768 conn(0x7fee400fbe50 msgr2=0x7fee400fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 shutdown_connections 2024-03-24T06:35:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T06:35:28.910+0000 7fee48161700 1 -- 172.21.15.143:0/1232333768 wait complete. 2024-03-24T06:35:29.547 DEBUG:teuthology.orchestra.run.smithi143:osd.10> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T06:35:29.550 INFO:tasks.cephadm:Deploying osd.11 on smithi155 with /dev/vg_nvme/lv_1... 2024-03-24T06:35:29.550 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T06:35:29.798 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:29 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:29 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:29 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:30.991 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:30 smithi155 ceph-mon[36262]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:30 smithi022 ceph-mon[32003]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T06:35:31.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: from='osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:31.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T06:35:31.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:31.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-mon[36520]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T06:35:31.179 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:35:30.875+0000 7ff267c3e700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T06:35:31.179 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:35:30 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:35:30.886+0000 7ff263a31700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:35:31.888 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T06:35:31.888 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T06:35:31.888 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T06:35:31.888 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T06:35:31.888 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124609 s, 841 MB/s 2024-03-24T06:35:31.889 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 2024-03-24T06:35:31.889 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T06:35:31.909 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:31 smithi143 ceph-mon[36520]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:35:31.909 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:31 smithi143 ceph-mon[36520]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T06:35:31.909 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:31.909 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:32.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:31 smithi022 ceph-mon[32003]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:35:32.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:31 smithi022 ceph-mon[32003]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T06:35:32.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:32.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:32.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:31 smithi155 ceph-mon[36262]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T06:35:32.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:31 smithi155 ceph-mon[36262]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T06:35:32.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:32.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:32.552 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_1 2024-03-24T06:35:32.800 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] boot 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T06:35:33.114 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:32 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] boot 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T06:35:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:32 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:33.175 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:35:33.175 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:35:33.175 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 894 GiB / 894 GiB avail 2024-03-24T06:35:33.176 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:33.176 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: osd.10 [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] boot 2024-03-24T06:35:33.176 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T06:35:33.176 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:32 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:35:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T06:35:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:33 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:34.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.223 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:33 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T06:35:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:33 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 -- 172.21.15.155:0/3511921099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103430 msgr2=0x7efd28103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/3511921099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103430 0x7efd28103890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009990 tx=0x7efd1c009ca0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 -- 172.21.15.155:0/3511921099 shutdown_connections 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/3511921099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd28102230 0x7efd28102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/3511921099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28103dd0 0x7efd2810a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/3511921099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103430 0x7efd28103890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 -- 172.21.15.155:0/3511921099 >> 172.21.15.155:0/3511921099 conn(0x7efd280fd810 msgr2=0x7efd280ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 -- 172.21.15.155:0/3511921099 shutdown_connections 2024-03-24T06:35:34.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 -- 172.21.15.155:0/3511921099 wait complete. 2024-03-24T06:35:34.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.429+0000 7efd2ca8a700 1 Processor -- start 2024-03-24T06:35:34.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 -- start start 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28102230 0x7efd28198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd28103430 0x7efd28198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efd2819fec0 con 0x7efd28103430 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efd281a0000 con 0x7efd28102230 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2ca8a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd281a0190 con 0x7efd28103dd0 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:52348/0 (socket says 172.21.15.155:52348) 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 -- 172.21.15.155:0/698871782 learned_addr learned my addr 172.21.15.155:0/698871782 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd2659c700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28102230 0x7efd28198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 -- 172.21.15.155:0/698871782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28102230 msgr2=0x7efd28198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:34.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28102230 0x7efd28198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 -- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd28103430 msgr2=0x7efd28198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd28103430 0x7efd28198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 -- 172.21.15.155:0/698871782 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1c0095f0 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.430+0000 7efd26d9d700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd2000b9a0 tx=0x7efd2000bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.431+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2000c930 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.431+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd281a0470 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.431+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd20014440 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.431+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd200075d0 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.431+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd281a0a40 con 0x7efd28103dd0 2024-03-24T06:35:34.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.432+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd28109710 con 0x7efd28103dd0 2024-03-24T06:35:34.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.433+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7efd20014920 con 0x7efd28103dd0 2024-03-24T06:35:34.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.433+0000 7efd177fe700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 0x7efd100a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:34.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.434+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7efd20006e40 con 0x7efd28103dd0 2024-03-24T06:35:34.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.434+0000 7efd2659c700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 0x7efd100a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:34.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.435+0000 7efd2659c700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 0x7efd100a4610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd28103290 tx=0x7efd18009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.437+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efd20096730 con 0x7efd28103dd0 2024-03-24T06:35:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:34.642+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efd2802dc00 con 0x7efd100a2150 2024-03-24T06:35:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi143 2024-03-24T06:35:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 4270M 2024-03-24T06:35:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T06:35:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T06:35:35.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:35.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:35.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi143 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 4270M 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi143 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 4270M 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T06:35:35.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:36.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:35 smithi022 ceph-mon[32003]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:36.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:35 smithi022 ceph-mon[32003]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T06:35:36.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:35 smithi143 ceph-mon[36520]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:36.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:35 smithi143 ceph-mon[36520]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T06:35:36.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:35 smithi155 ceph-mon[36262]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:35:36.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:35 smithi155 ceph-mon[36262]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T06:35:37.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:36 smithi022 ceph-mon[32003]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:35:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:36 smithi143 ceph-mon[36520]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:35:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:36 smithi155 ceph-mon[36262]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T06:35:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:37 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/2213433686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:37 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:37 smithi155 ceph-mon[36262]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]': finished 2024-03-24T06:35:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:37 smithi155 ceph-mon[36262]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T06:35:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:37 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:37 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/2213433686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:37 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:37 smithi022 ceph-mon[32003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]': finished 2024-03-24T06:35:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:37 smithi022 ceph-mon[32003]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T06:35:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:37 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:37 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/2213433686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:37 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]: dispatch 2024-03-24T06:35:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:37 smithi143 ceph-mon[36520]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8"}]': finished 2024-03-24T06:35:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:37 smithi143 ceph-mon[36520]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T06:35:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:37 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:39.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:38 smithi155 ceph-mon[36262]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-24T06:35:39.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:38 smithi155 ceph-mon[36262]: from='client.? 172.21.15.155:0/1036394238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:39.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:38 smithi022 ceph-mon[32003]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-24T06:35:39.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:38 smithi022 ceph-mon[32003]: from='client.? 172.21.15.155:0/1036394238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:38 smithi143 ceph-mon[36520]: pgmap v267: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-24T06:35:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:38 smithi143 ceph-mon[36520]: from='client.? 172.21.15.155:0/1036394238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T06:35:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:40 smithi155 ceph-mon[36262]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T06:35:41.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:40 smithi022 ceph-mon[32003]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T06:35:41.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:40 smithi143 ceph-mon[36520]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T06:35:43.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:42 smithi155 ceph-mon[36262]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:43.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:42 smithi022 ceph-mon[32003]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:43.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:42 smithi143 ceph-mon[36520]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:45.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:44 smithi155 ceph-mon[36262]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:45.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:44 smithi022 ceph-mon[32003]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:45.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:44 smithi143 ceph-mon[36520]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:46.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T06:35:46.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:45 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:46.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T06:35:46.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:45 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:46.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T06:35:46.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:45 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:47.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:46 smithi155 ceph-mon[36262]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:47.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:46 smithi155 ceph-mon[36262]: Deploying daemon osd.11 on smithi155 2024-03-24T06:35:47.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:46 smithi022 ceph-mon[32003]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:47.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:46 smithi022 ceph-mon[32003]: Deploying daemon osd.11 on smithi155 2024-03-24T06:35:47.343 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:46 smithi143 ceph-mon[36520]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:47.343 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:46 smithi143 ceph-mon[36520]: Deploying daemon osd.11 on smithi155 2024-03-24T06:35:49.260 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:48 smithi155 ceph-mon[36262]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:49.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:48 smithi022 ceph-mon[32003]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:49.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:48 smithi143 ceph-mon[36520]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:51.238 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:50 smithi155 ceph-mon[36262]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:51.239 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.239 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.239 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:51.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:50 smithi022 ceph-mon[32003]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:51.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:51.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:50 smithi143 ceph-mon[36520]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:51.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:51.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:53.214 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:52 smithi155 ceph-mon[36262]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:53.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:52 smithi022 ceph-mon[32003]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:53.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:52 smithi143 ceph-mon[36520]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:54.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:54 smithi155 ceph-mon[36262]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:54.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:54 smithi022 ceph-mon[32003]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:54 smithi143 ceph-mon[36520]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:54.527 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 11 on host 'smithi155' 2024-03-24T06:35:54.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.527+0000 7efd177fe700 1 -- 172.21.15.155:0/698871782 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7efd2802dc00 con 0x7efd100a2150 2024-03-24T06:35:54.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 msgr2=0x7efd100a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:54.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 0x7efd100a4610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd28103290 tx=0x7efd18009190 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 msgr2=0x7efd2819f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd2000b9a0 tx=0x7efd2000bcb0 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 shutdown_connections 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efd100a2150 0x7efd100a4610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd28103dd0 0x7efd2819f8f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd28102230 0x7efd28198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 --2- 172.21.15.155:0/698871782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd28103430 0x7efd28198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 >> 172.21.15.155:0/698871782 conn(0x7efd280fd810 msgr2=0x7efd28106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 shutdown_connections 2024-03-24T06:35:54.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:35:54.530+0000 7efd2ca8a700 1 -- 172.21.15.155:0/698871782 wait complete. 2024-03-24T06:35:55.217 DEBUG:teuthology.orchestra.run.smithi155:osd.11> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T06:35:55.220 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T06:35:55.220 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T06:35:55.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.686 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:56.762 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:56.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T06:35:56.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:56.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T06:35:56.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:56.763 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:56 smithi022 ceph-mon[32003]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:56.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-mon[36262]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:56.777 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:35:56.540+0000 7f575c2c9700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T06:35:56.778 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:35:56 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:35:56.553+0000 7f57580bc700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T06:35:56.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:56.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T06:35:56.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: from='osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:56.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T06:35:56.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:56.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:56 smithi143 ceph-mon[36520]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T06:35:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:57 smithi155 ceph-mon[36262]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:35:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:57 smithi155 ceph-mon[36262]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T06:35:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:57.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:57 smithi022 ceph-mon[32003]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:35:57.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:57 smithi022 ceph-mon[32003]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T06:35:57.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:57.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:57.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:57 smithi143 ceph-mon[36520]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T06:35:57.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:57 smithi143 ceph-mon[36520]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T06:35:57.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:57.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 -- 172.21.15.22:0/2508508197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 msgr2=0x7f587810c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 --2- 172.21.15.22:0/2508508197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f587810c8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f587000b0a0 tx=0x7f587000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 -- 172.21.15.22:0/2508508197 shutdown_connections 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 --2- 172.21.15.22:0/2508508197 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 0x7f58781048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 --2- 172.21.15.22:0/2508508197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f587810c8d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 --2- 172.21.15.22:0/2508508197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 0x7f5878105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 -- 172.21.15.22:0/2508508197 >> 172.21.15.22:0/2508508197 conn(0x7f58780ffa20 msgr2=0x7f5878101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:58.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 -- 172.21.15.22:0/2508508197 shutdown_connections 2024-03-24T06:35:58.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.147+0000 7f587e200700 1 -- 172.21.15.22:0/2508508197 wait complete. 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 Processor -- start 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 -- start start 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 0x7f587819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 0x7f587819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a2070 con 0x7f5878104480 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a21b0 con 0x7f5878105680 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.148+0000 7f587e200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a2340 con 0x7f5878106020 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 0x7f587819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44082/0 (socket says 172.21.15.22:44082) 2024-03-24T06:35:58.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 -- 172.21.15.22:0/2169823080 learned_addr learned my addr 172.21.15.22:0/2169823080 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5876a00700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 0x7f587819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 -- 172.21.15.22:0/2169823080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 msgr2=0x7f587819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 0x7f587819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 -- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 msgr2=0x7f587819a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 0x7f587819a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 -- 172.21.15.22:0/2169823080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5870009d00 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.149+0000 7f5877e00700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f587000b070 tx=0x7f587000bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.150+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587000e040 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.150+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5870004e60 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.150+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58781a25c0 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.150+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58700052a0 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.150+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58781a29e0 con 0x7f5878106020 2024-03-24T06:35:58.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.151+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f587810b770 con 0x7f5878106020 2024-03-24T06:35:58.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.153+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f587001d470 con 0x7f5878106020 2024-03-24T06:35:58.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.154+0000 7f586fe00700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 0x7f585c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:35:58.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.154+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f58700cef30 con 0x7f5878106020 2024-03-24T06:35:58.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.154+0000 7f5877400700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 0x7f585c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:35:58.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.155+0000 7f5877400700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 0x7f585c0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58781054e0 tx=0x7f5868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:35:58.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.158+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5870098390 con 0x7f5878106020 2024-03-24T06:35:58.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.385+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5878068ec0 con 0x7f5878106020 2024-03-24T06:35:58.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.385+0000 7f586fe00700 1 -- 172.21.15.22:0/2169823080 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f587009ba50 con 0x7f5878106020 2024-03-24T06:35:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:35:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 msgr2=0x7f585c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 0x7f585c0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58781054e0 tx=0x7f5868009260 comp rx=0 tx=0).stop 2024-03-24T06:35:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 msgr2=0x7f58781a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:35:58.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f587000b070 tx=0x7f587000bce0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 shutdown_connections 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5878104480 0x7f587819a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f585c0a2150 0x7f585c0a4610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5878106020 0x7f58781a1aa0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 --2- 172.21.15.22:0/2169823080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5878105680 0x7f587819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.390+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 >> 172.21.15.22:0/2169823080 conn(0x7f58780ffa20 msgr2=0x7f58781088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.391+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 shutdown_connections 2024-03-24T06:35:58.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:35:58.391+0000 7f587e200700 1 -- 172.21.15.22:0/2169823080 wait complete. 2024-03-24T06:35:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: purged_snaps scrub starts 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: purged_snaps scrub ok 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] boot 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:58 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2169823080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: purged_snaps scrub starts 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: purged_snaps scrub ok 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] boot 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:58.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:58 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2169823080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: purged_snaps scrub starts 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: purged_snaps scrub ok 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 983 GiB / 983 GiB avail 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: osd.11 [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] boot 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:35:58.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2169823080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T06:35:59.238 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711262157,"num_in_osds":12,"osd_in_since":1711262137,"num_remapped_pgs":0} 2024-03-24T06:35:59.239 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T06:35:59.686 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:35:59.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T06:35:59.832 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: Detected new or changed devices on smithi155 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.833 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:35:59 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: Detected new or changed devices on smithi155 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:35:59.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:35:59 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:00.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T06:36:00.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: Detected new or changed devices on smithi155 2024-03-24T06:36:00.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:00.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:00.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:00.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:35:59 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:00.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:00 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 4270M 2024-03-24T06:36:00.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:00 smithi022 ceph-mon[32003]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:00.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:00.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:00.878 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:00 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 4270M 2024-03-24T06:36:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:00 smithi143 ceph-mon[36520]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:00.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:00 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 4270M 2024-03-24T06:36:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:00 smithi155 ceph-mon[36262]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:01.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 -- 172.21.15.22:0/894196550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 msgr2=0x7f2388107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 --2- 172.21.15.22:0/894196550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f2388107520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f237c00b0a0 tx=0x7f237c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 -- 172.21.15.22:0/894196550 shutdown_connections 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 --2- 172.21.15.22:0/894196550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f2388107520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 --2- 172.21.15.22:0/894196550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f238810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 --2- 172.21.15.22:0/894196550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 0x7f2388101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 -- 172.21.15.22:0/894196550 >> 172.21.15.22:0/894196550 conn(0x7f2388078030 msgr2=0x7f2388078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 -- 172.21.15.22:0/894196550 shutdown_connections 2024-03-24T06:36:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.016+0000 7f2390200700 1 -- 172.21.15.22:0/894196550 wait complete. 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 Processor -- start 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 -- start start 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 0x7f23881a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f238819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f238819af60 con 0x7f2388107140 2024-03-24T06:36:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f238819b0a0 con 0x7f2388101aa0 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f2390200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f238819b260 con 0x7f2388101100 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.018+0000 7f238ce00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51894/0 (socket says 172.21.15.22:51894) 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 -- 172.21.15.22:0/3104333550 learned_addr learned my addr 172.21.15.22:0/3104333550 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238e200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f238819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238d800700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 0x7f23881a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 -- 172.21.15.22:0/3104333550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 msgr2=0x7f23881a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 0x7f23881a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 -- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 msgr2=0x7f238819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f238819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 -- 172.21.15.22:0/3104333550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2380009520 con 0x7f2388101aa0 2024-03-24T06:36:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.019+0000 7f238ce00700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f237400e9a0 tx=0x7f237400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.020+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374009790 con 0x7f2388101aa0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.020+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2374010430 con 0x7f2388101aa0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.020+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237c009d00 con 0x7f2388101aa0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.020+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237400cea0 con 0x7f2388101aa0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.020+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23881a7610 con 0x7f2388101aa0 2024-03-24T06:36:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.021+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2388104480 con 0x7f2388101aa0 2024-03-24T06:36:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.023+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2374010910 con 0x7f2388101aa0 2024-03-24T06:36:02.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.024+0000 7f2386000700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 0x7f23700a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:02.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.025+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2374014070 con 0x7f2388101aa0 2024-03-24T06:36:02.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.025+0000 7f238d800700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 0x7f23700a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:02.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.025+0000 7f238d800700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 0x7f23700a4610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f237c009fd0 tx=0x7f237c006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.027+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2374096e40 con 0x7f2388101aa0 2024-03-24T06:36:02.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.255+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f238819be90 con 0x7f2388101aa0 2024-03-24T06:36:02.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.256+0000 7f2386000700 1 -- 172.21.15.22:0/3104333550 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18465 (secure 0 0 0) 0x7f23740105a0 con 0x7f2388101aa0 2024-03-24T06:36:02.257 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:02.257 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":81,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:35:58.557028+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:36:02.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 msgr2=0x7f23700a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:02.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 0x7f23700a4610 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f237c009fd0 tx=0x7f237c006210 comp rx=0 tx=0).stop 2024-03-24T06:36:02.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 msgr2=0x7f23881a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f237400e9a0 tx=0x7f237400ecb0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 shutdown_connections 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2388107140 0x7f238819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23700a2150 0x7f23700a4610 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2388101100 0x7f23881a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 --2- 172.21.15.22:0/3104333550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2388101aa0 0x7f23881a0c80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.260+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 >> 172.21.15.22:0/3104333550 conn(0x7f2388078030 msgr2=0x7f23881002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.261+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 shutdown_connections 2024-03-24T06:36:02.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:02.261+0000 7f2390200700 1 -- 172.21.15.22:0/3104333550 wait complete. 2024-03-24T06:36:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:02 smithi022 ceph-mon[32003]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:02 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3104333550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:02 smithi143 ceph-mon[36520]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:02 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3104333550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:02 smithi155 ceph-mon[36262]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:03.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:02 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3104333550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:03.116 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T06:36:03.116 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T06:36:03.584 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:36:04.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:04 smithi022 ceph-mon[32003]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:04 smithi143 ceph-mon[36520]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:05.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:04 smithi155 ceph-mon[36262]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 -- 172.21.15.22:0/2999950430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7410074ec0 msgr2=0x7f7410112bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 --2- 172.21.15.22:0/2999950430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7410074ec0 0x7f7410112bb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f74000098f0 tx=0x7f7400009c00 comp rx=0 tx=0).stop 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 -- 172.21.15.22:0/2999950430 shutdown_connections 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 --2- 172.21.15.22:0/2999950430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74101130f0 0x7f74101154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 --2- 172.21.15.22:0/2999950430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f7410074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 --2- 172.21.15.22:0/2999950430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7410074ec0 0x7f7410112bb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 -- 172.21.15.22:0/2999950430 >> 172.21.15.22:0/2999950430 conn(0x7f74100fdbd0 msgr2=0x7f74100ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:05.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 -- 172.21.15.22:0/2999950430 shutdown_connections 2024-03-24T06:36:05.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.790+0000 7f7416200700 1 -- 172.21.15.22:0/2999950430 wait complete. 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.791+0000 7f7416200700 1 Processor -- start 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.791+0000 7f7416200700 1 -- start start 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f74101a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 0x7f74101a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 0x7f741019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f741019f2d0 con 0x7f7410074ec0 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f741019f440 con 0x7f74100745a0 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7416200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f741019f600 con 0x7f74101130f0 2024-03-24T06:36:05.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7407e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 0x7f74101a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7414e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f74101a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7415800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 0x7f741019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7407e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 0x7f74101a4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58536/0 (socket says 172.21.15.22:58536) 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7407e00700 1 -- 172.21.15.22:0/2543556725 learned_addr learned my addr 172.21.15.22:0/2543556725 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.792+0000 7f7414e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f74101a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:38164/0 (socket says 172.21.15.22:38164) 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 -- 172.21.15.22:0/2543556725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 msgr2=0x7f74101a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f74101a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 -- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 msgr2=0x7f74101a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 0x7f74101a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 -- 172.21.15.22:0/2543556725 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7400009520 con 0x7f74101130f0 2024-03-24T06:36:05.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7415800700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 0x7f741019eb20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f740800ef90 tx=0x7f740800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408018070 con 0x7f74101130f0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741019f880 con 0x7f74101130f0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.793+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740800f040 con 0x7f74101130f0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.794+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740800cd40 con 0x7f74101130f0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.794+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74101ab9d0 con 0x7f74101130f0 2024-03-24T06:36:05.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.795+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74101055d0 con 0x7f74101130f0 2024-03-24T06:36:05.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.797+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7408003a00 con 0x7f74101130f0 2024-03-24T06:36:05.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.798+0000 7f740e000700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 0x7f73f00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:05.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.798+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f74080cd3d0 con 0x7f74101130f0 2024-03-24T06:36:05.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.798+0000 7f7414e00700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 0x7f73f00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:05.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.799+0000 7f7414e00700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 0x7f73f00a4610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009c60 tx=0x7f73f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:05.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:05.803+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7408096830 con 0x7f74101130f0 2024-03-24T06:36:06.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.026+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f741019ffe0 con 0x7f74101130f0 2024-03-24T06:36:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.027+0000 7f740e000700 1 -- 172.21.15.22:0/2543556725 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f7408099ef0 con 0x7f74101130f0 2024-03-24T06:36:06.027 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:36:06.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 msgr2=0x7f73f00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:06.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 0x7f73f00a4610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009c60 tx=0x7f73f8009260 comp rx=0 tx=0).stop 2024-03-24T06:36:06.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 msgr2=0x7f741019eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 0x7f741019eb20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f740800ef90 tx=0x7f740800c3a0 comp rx=0 tx=0).stop 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 shutdown_connections 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f73f00a2150 0x7f73f00a4610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74101130f0 0x7f741019eb20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74100745a0 0x7f74101a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 --2- 172.21.15.22:0/2543556725 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7410074ec0 0x7f74101a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 >> 172.21.15.22:0/2543556725 conn(0x7f74100fdbd0 msgr2=0x7f74100ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 shutdown_connections 2024-03-24T06:36:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:06.030+0000 7f7416200700 1 -- 172.21.15.22:0/2543556725 wait complete. 2024-03-24T06:36:06.809 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:06 smithi022 ceph-mon[32003]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:06.810 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:06 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2543556725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:06.884 INFO:tasks.cephadm:Adding mds.a on smithi022 2024-03-24T06:36:06.884 INFO:tasks.cephadm:Adding mds.d on smithi022 2024-03-24T06:36:06.884 INFO:tasks.cephadm:Adding mds.g on smithi022 2024-03-24T06:36:06.884 INFO:tasks.cephadm:Adding mds.j on smithi022 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.b on smithi143 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.e on smithi143 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.h on smithi143 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.k on smithi143 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.c on smithi155 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.f on smithi155 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.i on smithi155 2024-03-24T06:36:06.885 INFO:tasks.cephadm:Adding mds.l on smithi155 2024-03-24T06:36:06.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l' 2024-03-24T06:36:06.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:06 smithi155 ceph-mon[36262]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:06.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:06 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2543556725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:06 smithi143 ceph-mon[36520]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:06 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2543556725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:07.133 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 -- 172.21.15.155:0/2298627916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5730101ad0 msgr2=0x7f5730101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 --2- 172.21.15.155:0/2298627916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5730101ad0 0x7f5730101f30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f572400aaa0 tx=0x7f572400adb0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 -- 172.21.15.155:0/2298627916 shutdown_connections 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 --2- 172.21.15.155:0/2298627916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57301008d0 0x7f5730100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 --2- 172.21.15.155:0/2298627916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730102470 0x7f5730108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 --2- 172.21.15.155:0/2298627916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5730101ad0 0x7f5730101f30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 -- 172.21.15.155:0/2298627916 >> 172.21.15.155:0/2298627916 conn(0x7f57300fbe50 msgr2=0x7f57300fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:08.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 -- 172.21.15.155:0/2298627916 shutdown_connections 2024-03-24T06:36:08.847 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:08 smithi155 ceph-mon[36262]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:08.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.846+0000 7f5735f37700 1 -- 172.21.15.155:0/2298627916 wait complete. 2024-03-24T06:36:08.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.847+0000 7f5735f37700 1 Processor -- start 2024-03-24T06:36:08.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.847+0000 7f5735f37700 1 -- start start 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 0x7f573019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5730102470 0x7f57301a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f57301a2060 con 0x7f5730102470 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f57301a21a0 con 0x7f5730101ad0 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f57301a2330 con 0x7f57301008d0 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:08.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:49168/0 (socket says 172.21.15.155:49168) 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 -- 172.21.15.155:0/3884501159 learned_addr learned my addr 172.21.15.155:0/3884501159 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572effd700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 0x7f573019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 -- 172.21.15.155:0/3884501159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 msgr2=0x7f573019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 0x7f573019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 -- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5730102470 msgr2=0x7f57301a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5730102470 0x7f57301a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 -- 172.21.15.155:0/3884501159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572400a700 con 0x7f57301008d0 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572effd700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 0x7f573019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:36:08.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572f7fe700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f572000b9f0 tx=0x7f572000bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572000ca30 con 0x7f57301008d0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57301a2610 con 0x7f57301008d0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572000cb90 con 0x7f57301008d0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.848+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57200125a0 con 0x7f57301008d0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.849+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57301a2b90 con 0x7f57301008d0 2024-03-24T06:36:08.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.849+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5730107e30 con 0x7f57301008d0 2024-03-24T06:36:08.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.851+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f572000cd00 con 0x7f57301008d0 2024-03-24T06:36:08.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.851+0000 7f572cff9700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 0x7f57180a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:08.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.851+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5720006e40 con 0x7f57301008d0 2024-03-24T06:36:08.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.852+0000 7f572effd700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 0x7f57180a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:08.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.853+0000 7f572effd700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 0x7f57180a4610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5724003dc0 tx=0x7f5724005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:08.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:08.855+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f57200968b0 con 0x7f57301008d0 2024-03-24T06:36:08.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:08 smithi022 ceph-mon[32003]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:08.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:08 smithi143 ceph-mon[36520]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.059+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}) -- 0x7f573002dd00 con 0x7f57180a2150 2024-03-24T06:36:09.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.076+0000 7f572cff9700 1 -- 172.21.15.155:0/3884501159 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f573002dd00 con 0x7f57180a2150 2024-03-24T06:36:09.075 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mds.all update... 2024-03-24T06:36:09.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 msgr2=0x7f57180a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:09.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 0x7f57180a4610 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5724003dc0 tx=0x7f5724005fb0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 msgr2=0x7f573019a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f572000b9f0 tx=0x7f572000bdb0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 shutdown_connections 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5730102470 0x7f57301a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f57180a2150 0x7f57180a4610 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57301008d0 0x7f573019a4d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 --2- 172.21.15.155:0/3884501159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5730101ad0 0x7f573019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 >> 172.21.15.155:0/3884501159 conn(0x7f57300fbe50 msgr2=0x7f5730104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 shutdown_connections 2024-03-24T06:36:09.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T06:36:09.078+0000 7f5735f37700 1 -- 172.21.15.155:0/3884501159 wait complete. 2024-03-24T06:36:09.908 DEBUG:teuthology.orchestra.run.smithi022:mds.a> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T06:36:09.916 DEBUG:teuthology.orchestra.run.smithi022:mds.d> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T06:36:09.964 DEBUG:teuthology.orchestra.run.smithi022:mds.g> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T06:36:10.009 DEBUG:teuthology.orchestra.run.smithi022:mds.j> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T06:36:10.054 DEBUG:teuthology.orchestra.run.smithi143:mds.b> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T06:36:10.057 DEBUG:teuthology.orchestra.run.smithi143:mds.e> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T06:36:10.100 DEBUG:teuthology.orchestra.run.smithi143:mds.h> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T06:36:10.143 DEBUG:teuthology.orchestra.run.smithi143:mds.k> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T06:36:10.186 DEBUG:teuthology.orchestra.run.smithi155:mds.c> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T06:36:10.189 DEBUG:teuthology.orchestra.run.smithi155:mds.f> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T06:36:10.212 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:10 smithi155 ceph-mon[36262]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:36:10.212 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:10 smithi155 ceph-mon[36262]: Saving service mds.all spec with placement smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-24T06:36:10.213 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:10 smithi155 ceph-mon[36262]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:10.213 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:10.213 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:10 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:10.213 DEBUG:teuthology.orchestra.run.smithi155:mds.i> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T06:36:10.256 DEBUG:teuthology.orchestra.run.smithi155:mds.l> sudo journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T06:36:10.298 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T06:36:10.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:36:10.382 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:10 smithi022 ceph-mon[32003]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:36:10.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:10 smithi022 ceph-mon[32003]: Saving service mds.all spec with placement smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-24T06:36:10.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:10 smithi022 ceph-mon[32003]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:10.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:10.383 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:10 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:10 smithi143 ceph-mon[36520]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:36:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:10 smithi143 ceph-mon[36520]: Saving service mds.all spec with placement smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-24T06:36:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:10 smithi143 ceph-mon[36520]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:10 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 -- 172.21.15.22:0/3093585744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 msgr2=0x7fb9e40764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3093585744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e40764f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009900 tx=0x7fb9d402bc40 comp rx=0 tx=0).stop 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 -- 172.21.15.22:0/3093585744 shutdown_connections 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3093585744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 0x7fb9e4078550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3093585744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e4077770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3093585744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e40764f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.547+0000 7fb9e9000640 1 -- 172.21.15.22:0/3093585744 >> 172.21.15.22:0/3093585744 conn(0x7fb9e40feec0 msgr2=0x7fb9e4101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.548+0000 7fb9e9000640 1 -- 172.21.15.22:0/3093585744 shutdown_connections 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.548+0000 7fb9e9000640 1 -- 172.21.15.22:0/3093585744 wait complete. 2024-03-24T06:36:10.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.549+0000 7fb9e9000640 1 Processor -- start 2024-03-24T06:36:10.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.549+0000 7fb9e9000640 1 -- start start 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.549+0000 7fb9e9000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.549+0000 7fb9e9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e9000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 0x7fb9e41a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e9000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e41a5920 con 0x7fb9e40760d0 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e9000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e41a5a90 con 0x7fb9e4077310 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e9000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e41a5c50 con 0x7fb9e4077cb0 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 0x7fb9e41a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:38192/0 (socket says 172.21.15.22:38192) 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 -- 172.21.15.22:0/3196637078 learned_addr learned my addr 172.21.15.22:0/3196637078 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1e00640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58554/0 (socket says 172.21.15.22:58554) 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 -- 172.21.15.22:0/3196637078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 msgr2=0x7fb9e41a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 0x7fb9e41a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 -- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 msgr2=0x7fb9e419de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e419de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.550+0000 7fb9e1400640 1 -- 172.21.15.22:0/3196637078 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc009840 con 0x7fb9e4077310 2024-03-24T06:36:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9e1400640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000e9f0 tx=0x7fb9d000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0009700 con 0x7fb9e4077310 2024-03-24T06:36:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d4009560 con 0x7fb9e4077310 2024-03-24T06:36:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9d000ccc0 con 0x7fb9e4077310 2024-03-24T06:36:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0018820 con 0x7fb9e4077310 2024-03-24T06:36:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.551+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e41a6270 con 0x7fb9e4077310 2024-03-24T06:36:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.553+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e41104a0 con 0x7fb9e4077310 2024-03-24T06:36:10.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.554+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb9d0010430 con 0x7fb9e4077310 2024-03-24T06:36:10.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.556+0000 7fb9daa00640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 0x7fb9c80a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:10.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.557+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb9d0014070 con 0x7fb9e4077310 2024-03-24T06:36:10.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.557+0000 7fb9e1e00640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 0x7fb9c80a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:10.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.557+0000 7fb9e1e00640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 0x7fb9c80a4600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4000c00 tx=0x7fb9d402c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.560+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb9d0053280 con 0x7fb9e4077310 2024-03-24T06:36:10.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.765+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9e402d2f0 con 0x7fb9e4077310 2024-03-24T06:36:10.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.766+0000 7fb9daa00640 1 -- 172.21.15.22:0/3196637078 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18465 (secure 0 0 0) 0x7fb9d0096fe0 con 0x7fb9e4077310 2024-03-24T06:36:10.767 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:10.767 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":81,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:35:58.557028+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 msgr2=0x7fb9c80a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 0x7fb9c80a4600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4000c00 tx=0x7fb9d402c7f0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 msgr2=0x7fb9e419e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000e9f0 tx=0x7fb9d000edf0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 shutdown_connections 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9c80a2140 0x7fb9c80a4600 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9e4077cb0 0x7fb9e41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9e4077310 0x7fb9e419e370 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 --2- 172.21.15.22:0/3196637078 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb9e40760d0 0x7fb9e419de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 >> 172.21.15.22:0/3196637078 conn(0x7fb9e40feec0 msgr2=0x7fb9e41005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 shutdown_connections 2024-03-24T06:36:10.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:10.770+0000 7fb9e9000640 1 -- 172.21.15.22:0/3196637078 wait complete. 2024-03-24T06:36:10.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T06:36:10.791 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 -- 172.21.15.22:0/2635498154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 msgr2=0x7f239810bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 --2- 172.21.15.22:0/2635498154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f239810bd90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2390009ca0 tx=0x7f239002c100 comp rx=0 tx=0).stop 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 -- 172.21.15.22:0/2635498154 shutdown_connections 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 --2- 172.21.15.22:0/2635498154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f239810bd90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 --2- 172.21.15.22:0/2635498154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 0x7f2398104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 --2- 172.21.15.22:0/2635498154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f2398103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 -- 172.21.15.22:0/2635498154 >> 172.21.15.22:0/2635498154 conn(0x7f23980feec0 msgr2=0x7f2398101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:11.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 -- 172.21.15.22:0/2635498154 shutdown_connections 2024-03-24T06:36:11.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.015+0000 7f239fa00640 1 -- 172.21.15.22:0/2635498154 wait complete. 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.016+0000 7f239fa00640 1 Processor -- start 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.016+0000 7f239fa00640 1 -- start start 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 0x7f2398199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f23981a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f23981a14b0 con 0x7f2398105500 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f23981a1620 con 0x7f2398104b40 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239fa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f23981a17e0 con 0x7f2398103950 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239da00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f23981a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:60280/0 (socket says 172.21.15.22:60280) 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f239d000640 1 -- 172.21.15.22:0/3287302118 learned_addr learned my addr 172.21.15.22:0/3287302118 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.017+0000 7f2397e00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 0x7f2398199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 -- 172.21.15.22:0/3287302118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 msgr2=0x7f2398199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 0x7f2398199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 -- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 msgr2=0x7f23981a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f23981a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 -- 172.21.15.22:0/3287302118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2384009490 con 0x7f2398103950 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239d000640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f238800e9c0 tx=0x7f238800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23880096e0 con 0x7f2398103950 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f238800cd80 con 0x7f2398103950 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388010430 con 0x7f2398103950 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2390009900 con 0x7f2398103950 2024-03-24T06:36:11.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.018+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23981a1df0 con 0x7f2398103950 2024-03-24T06:36:11.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.019+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239810ac70 con 0x7f2398103950 2024-03-24T06:36:11.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.021+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2388003680 con 0x7f2398103950 2024-03-24T06:36:11.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.023+0000 7f2395600640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 0x7f237c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.023+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2388014070 con 0x7f2398103950 2024-03-24T06:36:11.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.023+0000 7f2397e00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 0x7f237c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.024+0000 7f2397e00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 0x7f237c0a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f239819aed0 tx=0x7f2384005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:11.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.027+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2388096fa0 con 0x7f2398103950 2024-03-24T06:36:11.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.230+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f239802d240 con 0x7f2398103950 2024-03-24T06:36:11.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.231+0000 7f2395600640 1 -- 172.21.15.22:0/3287302118 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f238809a660 con 0x7f2398103950 2024-03-24T06:36:11.231 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 msgr2=0x7f237c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 0x7f237c0a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f239819aed0 tx=0x7f2384005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 msgr2=0x7f23981999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f238800e9c0 tx=0x7f238800ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 shutdown_connections 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f237c0a2140 0x7f237c0a4600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2398105500 0x7f23981a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2398104b40 0x7f2398199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 --2- 172.21.15.22:0/3287302118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2398103950 0x7f23981999c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 >> 172.21.15.22:0/3287302118 conn(0x7f23980feec0 msgr2=0x7f2398107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 shutdown_connections 2024-03-24T06:36:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.234+0000 7f239fa00640 1 -- 172.21.15.22:0/3287302118 wait complete. 2024-03-24T06:36:11.256 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:11.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: Deploying daemon mds.l on smithi155 2024-03-24T06:36:11.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:11 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3196637078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: Deploying daemon mds.l on smithi155 2024-03-24T06:36:11.344 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:11 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3196637078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:11.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: Deploying daemon mds.l on smithi155 2024-03-24T06:36:11.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:11 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3196637078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 -- 172.21.15.22:0/327359753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4101570 msgr2=0x7f7ed4101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 --2- 172.21.15.22:0/327359753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4101570 0x7f7ed4101990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009ca0 tx=0x7f7ec802c100 comp rx=0 tx=0).stop 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 -- 172.21.15.22:0/327359753 shutdown_connections 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 --2- 172.21.15.22:0/327359753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4103120 0x7f7ed41099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 --2- 172.21.15.22:0/327359753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4102760 0x7f7ed4102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 --2- 172.21.15.22:0/327359753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4101570 0x7f7ed4101990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 -- 172.21.15.22:0/327359753 >> 172.21.15.22:0/327359753 conn(0x7f7ed40fcae0 msgr2=0x7f7ed40fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 -- 172.21.15.22:0/327359753 shutdown_connections 2024-03-24T06:36:11.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.487+0000 7f7edb000640 1 -- 172.21.15.22:0/327359753 wait complete. 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 Processor -- start 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 -- start start 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 0x7f7ed4199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 0x7f7ed41a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41a12f0 con 0x7f7ed4103120 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41a1460 con 0x7f7ed4102760 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.488+0000 7f7edb000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed41a1620 con 0x7f7ed4101570 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 0x7f7ed41a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 0x7f7ed4199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:60300/0 (socket says 172.21.15.22:60300) 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9c00640 1 -- 172.21.15.22:0/226816288 learned_addr learned my addr 172.21.15.22:0/226816288 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 -- 172.21.15.22:0/226816288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 msgr2=0x7f7ed41998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 -- 172.21.15.22:0/226816288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 msgr2=0x7f7ed4199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 0x7f7ed4199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7eda600640 1 -- 172.21.15.22:0/226816288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec8009900 con 0x7f7ed4103120 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9c00640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.489+0000 7f7ed9200640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 0x7f7ed4199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7eda600640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 0x7f7ed41a0db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00e970 tx=0x7f7ecc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc009620 con 0x7f7ed4103120 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ecc00ccc0 con 0x7f7ed4103120 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed41a1900 con 0x7f7ed4103120 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc004a00 con 0x7f7ed4103120 2024-03-24T06:36:11.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.490+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed41a1dc0 con 0x7f7ed4103120 2024-03-24T06:36:11.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.491+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed410cf80 con 0x7f7ed4103120 2024-03-24T06:36:11.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.493+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7ecc005020 con 0x7f7ed4103120 2024-03-24T06:36:11.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.496+0000 7f7ed2000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 0x7f7ebc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:11.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.496+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7ecc014070 con 0x7f7ed4103120 2024-03-24T06:36:11.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.496+0000 7f7ed9c00640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 0x7f7ebc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:11.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.497+0000 7f7ed9c00640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 0x7f7ebc0a4580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005e30 tx=0x7f7ec80326d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:11.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.499+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ecc096d40 con 0x7f7ed4103120 2024-03-24T06:36:11.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.739+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ed410d110 con 0x7f7ed4103120 2024-03-24T06:36:11.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.740+0000 7f7ed2000640 1 -- 172.21.15.22:0/226816288 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7ecc09a400 con 0x7f7ed4103120 2024-03-24T06:36:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:11.740 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T06:36:11.740 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 1 2024-03-24T06:36:11.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 msgr2=0x7f7ebc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 0x7f7ebc0a4580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005e30 tx=0x7f7ec80326d0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 msgr2=0x7f7ed41a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 0x7f7ed41a0db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00e970 tx=0x7f7ecc00ee30 comp rx=0 tx=0).stop 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 shutdown_connections 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7ebc0a20c0 0x7f7ebc0a4580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ed4103120 0x7f7ed41a0db0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ed4102760 0x7f7ed4199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 --2- 172.21.15.22:0/226816288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ed4101570 0x7f7ed41998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:11.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 >> 172.21.15.22:0/226816288 conn(0x7f7ed40fcae0 msgr2=0x7f7ed4073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:11.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 shutdown_connections 2024-03-24T06:36:11.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:11.743+0000 7f7edb000640 1 -- 172.21.15.22:0/226816288 wait complete. 2024-03-24T06:36:12.230 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:12 smithi155 ceph-mon[36262]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:12.230 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3287302118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:12.230 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/226816288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:12 smithi022 ceph-mon[32003]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3287302118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/226816288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:12 smithi143 ceph-mon[36520]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3287302118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/226816288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:12.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 systemd[1]: Starting Ceph mds.l for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:12.935 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 podman[66950]: 2024-03-24 06:36:12.531835309 +0000 UTC m=+0.017080894 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:12.935 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 podman[66950]: 2024-03-24 06:36:12.69225899 +0000 UTC m=+0.177504568 container create 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:36:12.936 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 podman[66950]: 2024-03-24 06:36:12.936434931 +0000 UTC m=+0.421680509 container init 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T06:36:13.277 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 podman[66950]: 2024-03-24 06:36:12.939114187 +0000 UTC m=+0.424359775 container start 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:36:13.277 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 ceph-mds[66969]: -- 172.21.15.155:0/4175753107 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a0bfb2e1a0 con 0x55a0bfb05000 2024-03-24T06:36:13.277 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:12 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: starting mds.l at 2024-03-24T06:36:13.278 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:13 smithi155 bash[66950]: 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 2024-03-24T06:36:13.278 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:36:13 smithi155 systemd[1]: Started Ceph mds.l for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:13.763 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:13.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 -- 172.21.15.22:0/1196957600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 msgr2=0x7f3e3c103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:13.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 --2- 172.21.15.22:0/1196957600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c103d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009900 tx=0x7f3e2c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:36:13.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 -- 172.21.15.22:0/1196957600 shutdown_connections 2024-03-24T06:36:13.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 --2- 172.21.15.22:0/1196957600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c10bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 --2- 172.21.15.22:0/1196957600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 --2- 172.21.15.22:0/1196957600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c103d70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 -- 172.21.15.22:0/1196957600 >> 172.21.15.22:0/1196957600 conn(0x7f3e3c0feec0 msgr2=0x7f3e3c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 -- 172.21.15.22:0/1196957600 shutdown_connections 2024-03-24T06:36:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.990+0000 7f3e40a00640 1 -- 172.21.15.22:0/1196957600 wait complete. 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.991+0000 7f3e40a00640 1 Processor -- start 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.991+0000 7f3e40a00640 1 -- start start 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.991+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e40a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e40a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c1a1510 con 0x7f3e3c104b40 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e40a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c1a1680 con 0x7f3e3c105500 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e40a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3c1a1840 con 0x7f3e3c103950 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e39400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e39400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58602/0 (socket says 172.21.15.22:58602) 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e39400640 1 -- 172.21.15.22:0/3201002751 learned_addr learned my addr 172.21.15.22:0/3201002751 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e39e00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 -- 172.21.15.22:0/3201002751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 msgr2=0x7f3e3c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 -- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 msgr2=0x7f3e3c199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e3a800640 1 -- 172.21.15.22:0/3201002751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2c009560 con 0x7f3e3c105500 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.992+0000 7f3e39400640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e3a800640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c1a0fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009fd0 tx=0x7f3e3400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3400ce20 con 0x7f3e3c105500 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e34005020 con 0x7f3e3c105500 2024-03-24T06:36:13.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3c1a1b20 con 0x7f3e3c105500 2024-03-24T06:36:13.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34010690 con 0x7f3e3c105500 2024-03-24T06:36:13.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.993+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3c1a2030 con 0x7f3e3c105500 2024-03-24T06:36:13.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.995+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e3c10ac70 con 0x7f3e3c105500 2024-03-24T06:36:13.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.996+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3e34005500 con 0x7f3e3c105500 2024-03-24T06:36:13.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.997+0000 7f3e32a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 0x7f3e200a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:13.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.997+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3e34014070 con 0x7f3e3c105500 2024-03-24T06:36:13.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.998+0000 7f3e39e00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 0x7f3e200a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:13.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:13.998+0000 7f3e39e00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 0x7f3e200a4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c02c250 tx=0x7f3e2c02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.002+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3e34097330 con 0x7f3e3c105500 2024-03-24T06:36:14.099 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6832/715907814,v1:172.21.15.155:6833/715907814] up:boot 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: fsmap 1 up:standby 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:14.100 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:13 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6832/715907814,v1:172.21.15.155:6833/715907814] up:boot 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: fsmap 1 up:standby 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.137 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:14.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:14.138 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:13 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:14.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.246+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e3c02d1d0 con 0x7f3e3c105500 2024-03-24T06:36:14.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.247+0000 7f3e32a00640 1 -- 172.21.15.22:0/3201002751 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f3e34005190 con 0x7f3e3c105500 2024-03-24T06:36:14.247 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:14.247 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T06:36:14.247 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 2 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.249+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 msgr2=0x7f3e200a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 0x7f3e200a4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c02c250 tx=0x7f3e2c02c7f0 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 msgr2=0x7f3e3c1a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c1a0fd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009fd0 tx=0x7f3e3400ef10 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 shutdown_connections 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3e200a20c0 0x7f3e200a4580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e3c105500 0x7f3e3c1a0fd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e3c104b40 0x7f3e3c199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 --2- 172.21.15.22:0/3201002751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c103950 0x7f3e3c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 >> 172.21.15.22:0/3201002751 conn(0x7f3e3c0feec0 msgr2=0x7f3e3c107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 shutdown_connections 2024-03-24T06:36:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:14.250+0000 7f3e40a00640 1 -- 172.21.15.22:0/3201002751 wait complete. 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6832/715907814,v1:172.21.15.155:6833/715907814] up:boot 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: fsmap 1 up:standby 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:13 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:14 smithi143 ceph-mon[36520]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:14 smithi143 ceph-mon[36520]: Deploying daemon mds.k on smithi143 2024-03-24T06:36:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:14 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3201002751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:15.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:14 smithi155 ceph-mon[36262]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:15.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:14 smithi155 ceph-mon[36262]: Deploying daemon mds.k on smithi143 2024-03-24T06:36:15.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:14 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3201002751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:14 smithi022 ceph-mon[32003]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:14 smithi022 ceph-mon[32003]: Deploying daemon mds.k on smithi143 2024-03-24T06:36:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:14 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3201002751' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:15.658 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 systemd[1]: Starting Ceph mds.k for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:15.928 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 podman[65653]: 2024-03-24 06:36:15.561336838 +0000 UTC m=+0.017660155 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:15.928 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 podman[65653]: 2024-03-24 06:36:15.683875746 +0000 UTC m=+0.140199048 container create d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:36:15.928 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 podman[65653]: 2024-03-24 06:36:15.878350339 +0000 UTC m=+0.334673639 container init d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T06:36:15.929 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 podman[65653]: 2024-03-24 06:36:15.880908564 +0000 UTC m=+0.337231872 container start d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:36:15.929 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 ceph-mds[65672]: -- 172.21.15.143:0/3761697445 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa4a0d81a0 con 0x55aa4a0ae800 2024-03-24T06:36:15.929 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: starting mds.k at 2024-03-24T06:36:16.270 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:16 smithi155 ceph-mon[36262]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:16.341 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:16 smithi022 ceph-mon[32003]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:16 smithi143 ceph-mon[36520]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:16.428 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 bash[65653]: d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 2024-03-24T06:36:16.428 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:36:15 smithi143 systemd[1]: Started Ceph mds.k for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:16.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 -- 172.21.15.22:0/4270023539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 msgr2=0x7f09dc104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:16.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 --2- 172.21.15.22:0/4270023539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc104fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:16.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 -- 172.21.15.22:0/4270023539 shutdown_connections 2024-03-24T06:36:16.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 --2- 172.21.15.22:0/4270023539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 0x7f09dc10bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 --2- 172.21.15.22:0/4270023539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc104fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 --2- 172.21.15.22:0/4270023539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 -- 172.21.15.22:0/4270023539 >> 172.21.15.22:0/4270023539 conn(0x7f09dc0feec0 msgr2=0x7f09dc101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:16.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 -- 172.21.15.22:0/4270023539 shutdown_connections 2024-03-24T06:36:16.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.481+0000 7f09e1a00640 1 -- 172.21.15.22:0/4270023539 wait complete. 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.482+0000 7f09e1a00640 1 Processor -- start 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.482+0000 7f09e1a00640 1 -- start start 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.482+0000 7f09e1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.482+0000 7f09e1a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.482+0000 7f09e1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 0x7f09dc1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09e1a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1a1540 con 0x7f09dc103950 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09e1a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1a16b0 con 0x7f09dc104b40 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09e1a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1a1870 con 0x7f09dc105500 2024-03-24T06:36:16.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09db400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 0x7f09dc1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60470/0 (socket says 172.21.15.22:60470) 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 -- 172.21.15.22:0/1931032691 learned_addr learned my addr 172.21.15.22:0/1931032691 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09da000640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 -- 172.21.15.22:0/1931032691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 msgr2=0x7f09dc1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 0x7f09dc1a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 -- 172.21.15.22:0/1931032691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 msgr2=0x7f09dc199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.483+0000 7f09daa00640 1 -- 172.21.15.22:0/1931032691 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c8009560 con 0x7f09dc103950 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09daa00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e9c0 tx=0x7f09cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc009700 con 0x7f09dc103950 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09cc00cd80 con 0x7f09dc103950 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc1a1b50 con 0x7f09dc103950 2024-03-24T06:36:16.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc010430 con 0x7f09dc103950 2024-03-24T06:36:16.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.484+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc1a2080 con 0x7f09dc103950 2024-03-24T06:36:16.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.486+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc10ac70 con 0x7f09dc103950 2024-03-24T06:36:16.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.487+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f09cc003680 con 0x7f09dc103950 2024-03-24T06:36:16.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.490+0000 7f09d3400640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 0x7f09c00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:16.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.490+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f09cc014070 con 0x7f09dc103950 2024-03-24T06:36:16.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.490+0000 7f09da000640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 0x7f09c00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:16.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.491+0000 7f09da000640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 0x7f09c00a4600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09c8000c00 tx=0x7f09c802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:16.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.494+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f09cc096f40 con 0x7f09dc103950 2024-03-24T06:36:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.738+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09dc10f380 con 0x7f09dc103950 2024-03-24T06:36:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.739+0000 7f09d3400640 1 -- 172.21.15.22:0/1931032691 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f09cc09a600 con 0x7f09dc103950 2024-03-24T06:36:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:16.739 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T06:36:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 3 2024-03-24T06:36:16.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 msgr2=0x7f09c00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 0x7f09c00a4600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09c8000c00 tx=0x7f09c802c7e0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 msgr2=0x7f09dc199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00e9c0 tx=0x7f09cc00ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 shutdown_connections 2024-03-24T06:36:16.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09c00a2140 0x7f09c00a4600 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09dc105500 0x7f09dc1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc104b40 0x7f09dc199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 --2- 172.21.15.22:0/1931032691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09dc103950 0x7f09dc199a50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 >> 172.21.15.22:0/1931032691 conn(0x7f09dc0feec0 msgr2=0x7f09dc107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 shutdown_connections 2024-03-24T06:36:16.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:16.742+0000 7f09e1a00640 1 -- 172.21.15.22:0/1931032691 wait complete. 2024-03-24T06:36:17.082 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6832/3570497194,v1:172.21.15.143:6833/3570497194] up:boot 2024-03-24T06:36:17.082 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: fsmap 2 up:standby 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: Deploying daemon mds.j on smithi022 2024-03-24T06:36:17.083 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:17 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1931032691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:17.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6832/3570497194,v1:172.21.15.143:6833/3570497194] up:boot 2024-03-24T06:36:17.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: fsmap 2 up:standby 2024-03-24T06:36:17.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:36:17.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: Deploying daemon mds.j on smithi022 2024-03-24T06:36:17.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:17 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1931032691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6832/3570497194,v1:172.21.15.143:6833/3570497194] up:boot 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: fsmap 2 up:standby 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:17.280 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:17.281 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:17.281 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:17.281 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: Deploying daemon mds.j on smithi022 2024-03-24T06:36:17.281 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:17 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1931032691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:18 smithi155 ceph-mon[36262]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:18.310 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:18 smithi022 ceph-mon[32003]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:18 smithi143 ceph-mon[36520]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:18.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.017+0000 7f895d200640 1 -- 172.21.15.22:0/4049034430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 msgr2=0x7f8958103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.017+0000 7f895d200640 1 --2- 172.21.15.22:0/4049034430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f8958103d70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8948009900 tx=0x7f894802bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.017+0000 7f895d200640 1 -- 172.21.15.22:0/4049034430 shutdown_connections 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 --2- 172.21.15.22:0/4049034430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958105500 0x7f895810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 --2- 172.21.15.22:0/4049034430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958104b40 0x7f8958104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 --2- 172.21.15.22:0/4049034430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f8958103d70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 -- 172.21.15.22:0/4049034430 >> 172.21.15.22:0/4049034430 conn(0x7f89580feec0 msgr2=0x7f8958101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 -- 172.21.15.22:0/4049034430 shutdown_connections 2024-03-24T06:36:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.018+0000 7f895d200640 1 -- 172.21.15.22:0/4049034430 wait complete. 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 Processor -- start 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 -- start start 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f89581999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958105500 0x7f89581a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f89581a14b0 con 0x7f8958103950 2024-03-24T06:36:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f89581a1620 con 0x7f8958105500 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.019+0000 7f895d200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f89581a17e0 con 0x7f8958104b40 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39196/0 (socket says 172.21.15.22:39196) 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 -- 172.21.15.22:0/2941636919 learned_addr learned my addr 172.21.15.22:0/2941636919 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955e00640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f89581999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8956800640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958105500 0x7f89581a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 -- 172.21.15.22:0/2941636919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958105500 msgr2=0x7f89581a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958105500 0x7f89581a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 -- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 msgr2=0x7f89581999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f89581999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.020+0000 7f8955400640 1 -- 172.21.15.22:0/2941636919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8948009560 con 0x7f8958104b40 2024-03-24T06:36:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f8955400640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f894400e9f0 tx=0x7f894400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944009700 con 0x7f8958104b40 2024-03-24T06:36:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894400cd80 con 0x7f8958104b40 2024-03-24T06:36:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89581a1ac0 con 0x7f8958104b40 2024-03-24T06:36:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944018920 con 0x7f8958104b40 2024-03-24T06:36:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.021+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89581a1fd0 con 0x7f8958104b40 2024-03-24T06:36:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.022+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895810ac70 con 0x7f8958104b40 2024-03-24T06:36:19.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.026+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8944018a80 con 0x7f8958104b40 2024-03-24T06:36:19.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.028+0000 7f894ea00640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 0x7f893c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:19.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.028+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8944014070 con 0x7f8958104b40 2024-03-24T06:36:19.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.028+0000 7f8955e00640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 0x7f893c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:19.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.028+0000 7f8955e00640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 0x7f893c0a4580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8948005ab0 tx=0x7f8948005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:19.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.031+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8944053280 con 0x7f8958104b40 2024-03-24T06:36:19.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.282+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f895810f350 con 0x7f8958104b40 2024-03-24T06:36:19.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.283+0000 7f894ea00640 1 -- 172.21.15.22:0/2941636919 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f8944097090 con 0x7f8958104b40 2024-03-24T06:36:19.283 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:19.283 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T06:36:19.284 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 3 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.286+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 msgr2=0x7f893c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.286+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 0x7f893c0a4580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8948005ab0 tx=0x7f8948005fd0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.286+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 msgr2=0x7f8958199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.286+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f894400e9f0 tx=0x7f894400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 shutdown_connections 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f893c0a20c0 0x7f893c0a4580 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8958105500 0x7f89581a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8958104b40 0x7f8958199f00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 --2- 172.21.15.22:0/2941636919 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8958103950 0x7f89581999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 >> 172.21.15.22:0/2941636919 conn(0x7f89580feec0 msgr2=0x7f8958107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 shutdown_connections 2024-03-24T06:36:19.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:19.287+0000 7f895d200640 1 -- 172.21.15.22:0/2941636919 wait complete. 2024-03-24T06:36:19.859 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 podman[72182]: 2024-03-24 06:36:19.568398755 +0000 UTC m=+0.164402415 container create 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T06:36:19.859 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 podman[72182]: 2024-03-24 06:36:19.858751869 +0000 UTC m=+0.454755536 container init 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:36:20.140 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 podman[72182]: 2024-03-24 06:36:19.871211929 +0000 UTC m=+0.467215582 container start 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T06:36:20.141 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 ceph-mds[72201]: -- 172.21.15.22:0/1826684544 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56361ea5c1a0 con 0x56361ea32800 2024-03-24T06:36:20.141 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: starting mds.j at 2024-03-24T06:36:20.141 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 bash[72182]: 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 2024-03-24T06:36:20.141 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:36:19 smithi022 systemd[1]: Started Ceph mds.j for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2941636919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:20 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2941636919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:20 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2941636919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:20.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:20 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:21.306 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:21.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:21 smithi022 ceph-mon[32003]: Deploying daemon mds.d on smithi022 2024-03-24T06:36:21.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:21 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6834/298241489,v1:172.21.15.22:6835/298241489] up:boot 2024-03-24T06:36:21.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:21 smithi022 ceph-mon[32003]: fsmap 3 up:standby 2024-03-24T06:36:21.397 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:21 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:36:21.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:21 smithi143 ceph-mon[36520]: Deploying daemon mds.d on smithi022 2024-03-24T06:36:21.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:21 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6834/298241489,v1:172.21.15.22:6835/298241489] up:boot 2024-03-24T06:36:21.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:21 smithi143 ceph-mon[36520]: fsmap 3 up:standby 2024-03-24T06:36:21.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:21 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:36:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:21 smithi155 ceph-mon[36262]: Deploying daemon mds.d on smithi022 2024-03-24T06:36:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:21 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6834/298241489,v1:172.21.15.22:6835/298241489] up:boot 2024-03-24T06:36:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:21 smithi155 ceph-mon[36262]: fsmap 3 up:standby 2024-03-24T06:36:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:21 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:36:21.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 -- 172.21.15.22:0/1211985753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d8108710 msgr2=0x7f09d810ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:21.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 --2- 172.21.15.22:0/1211985753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d8108710 0x7f09d810ab40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:21.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 -- 172.21.15.22:0/1211985753 shutdown_connections 2024-03-24T06:36:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 --2- 172.21.15.22:0/1211985753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d810b500 0x7f09d81143a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 --2- 172.21.15.22:0/1211985753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b080 0x7f09d8111e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 --2- 172.21.15.22:0/1211985753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d8108710 0x7f09d810ab40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 -- 172.21.15.22:0/1211985753 >> 172.21.15.22:0/1211985753 conn(0x7f09d806e920 msgr2=0x7f09d8070d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:21.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 -- 172.21.15.22:0/1211985753 shutdown_connections 2024-03-24T06:36:21.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.536+0000 7f09dce00640 1 -- 172.21.15.22:0/1211985753 wait complete. 2024-03-24T06:36:21.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.537+0000 7f09dce00640 1 Processor -- start 2024-03-24T06:36:21.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.537+0000 7f09dce00640 1 -- start start 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 0x7f09d81a2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b500 0x7f09d81a97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f09d81a9cf0 con 0x7f09d810b080 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f09d81a9e60 con 0x7f09d8108710 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09dce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09d81aa020 con 0x7f09d810b500 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09d5400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09d5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 0x7f09d81a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:21.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09d5400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60534/0 (socket says 172.21.15.22:60534) 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09d5400640 1 -- 172.21.15.22:0/3893131120 learned_addr learned my addr 172.21.15.22:0/3893131120 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.538+0000 7f09d6800640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b500 0x7f09d81a97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 -- 172.21.15.22:0/3893131120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b500 msgr2=0x7f09d81a97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b500 0x7f09d81a97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 -- 172.21.15.22:0/3893131120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 msgr2=0x7f09d81a2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 0x7f09d81a2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 -- 172.21.15.22:0/3893131120 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c8009560 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5e00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 0x7f09d81a2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09d5400640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c400e9c0 tx=0x7f09c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4009700 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c400cd80 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4010430 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d81aa300 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.539+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d81aa810 con 0x7f09d810b080 2024-03-24T06:36:21.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.540+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d8113560 con 0x7f09d810b080 2024-03-24T06:36:21.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.543+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f09c4003680 con 0x7f09d810b080 2024-03-24T06:36:21.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.545+0000 7f09cea00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 0x7f09bc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:21.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.545+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f09c4014070 con 0x7f09d810b080 2024-03-24T06:36:21.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.545+0000 7f09d5e00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 0x7f09bc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:21.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.546+0000 7f09d5e00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 0x7f09bc0a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:21.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.547+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f09c4097260 con 0x7f09d810b080 2024-03-24T06:36:21.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.790+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09d8117b20 con 0x7f09d810b080 2024-03-24T06:36:21.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.791+0000 7f09cea00640 1 -- 172.21.15.22:0/3893131120 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f09c409a920 con 0x7f09d810b080 2024-03-24T06:36:21.791 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:21.791 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-24T06:36:21.792 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 4 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.794+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 msgr2=0x7f09bc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.794+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 0x7f09bc0a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.794+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 msgr2=0x7f09d81a2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.794+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c400e9c0 tx=0x7f09c400ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 shutdown_connections 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09bc0a20c0 0x7f09bc0a4580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09d810b500 0x7f09d81a97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09d810b080 0x7f09d81a2740 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 --2- 172.21.15.22:0/3893131120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09d8108710 0x7f09d81a2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 >> 172.21.15.22:0/3893131120 conn(0x7f09d806e920 msgr2=0x7f09d806f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 shutdown_connections 2024-03-24T06:36:21.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:21.795+0000 7f09dce00640 1 -- 172.21.15.22:0/3893131120 wait complete. 2024-03-24T06:36:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:22 smithi022 ceph-mon[32003]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:22 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3893131120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:22 smithi143 ceph-mon[36520]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:22 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3893131120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:22 smithi155 ceph-mon[36262]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:22 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3893131120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:23.605 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 systemd[1]: Starting Ceph mds.d for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:23.816 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:23.863 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 podman[72524]: 2024-03-24 06:36:23.509050601 +0000 UTC m=+0.038639361 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:23.863 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 podman[72524]: 2024-03-24 06:36:23.610008871 +0000 UTC m=+0.139597668 container create 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:36:23.863 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 podman[72524]: 2024-03-24 06:36:23.86316246 +0000 UTC m=+0.392751229 container init 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 -- 172.21.15.22:0/2183058219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 msgr2=0x7fa628109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 --2- 172.21.15.22:0/2183058219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa628109690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa620009be0 tx=0x7fa62002bef0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 -- 172.21.15.22:0/2183058219 shutdown_connections 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 --2- 172.21.15.22:0/2183058219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa628109690 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 --2- 172.21.15.22:0/2183058219 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628104910 0x7fa628106d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 --2- 172.21.15.22:0/2183058219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628101fe0 0x7fa6281043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 -- 172.21.15.22:0/2183058219 >> 172.21.15.22:0/2183058219 conn(0x7fa6280fb9f0 msgr2=0x7fa6280fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:24.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 -- 172.21.15.22:0/2183058219 shutdown_connections 2024-03-24T06:36:24.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.043+0000 7fa62d400640 1 -- 172.21.15.22:0/2183058219 wait complete. 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.044+0000 7fa62d400640 1 Processor -- start 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 -- start start 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628104910 0x7fa628199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa6281a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa6281a1270 con 0x7fa628101fe0 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa6281a13e0 con 0x7fa628107240 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62d400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6281a15a0 con 0x7fa628104910 2024-03-24T06:36:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa627e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa627e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60576/0 (socket says 172.21.15.22:60576) 2024-03-24T06:36:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa627e00640 1 -- 172.21.15.22:0/1296931043 learned_addr learned my addr 172.21.15.22:0/1296931043 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa61f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628104910 0x7fa628199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.045+0000 7fa62ca00640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa6281a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 -- 172.21.15.22:0/1296931043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628104910 msgr2=0x7fa628199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628104910 0x7fa628199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 -- 172.21.15.22:0/1296931043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 msgr2=0x7fa6281a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa6281a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 -- 172.21.15.22:0/1296931043 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa620009840 con 0x7fa628101fe0 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa62ca00640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa6281a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa627e00640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa61800d950 tx=0x7fa61800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618009c30 con 0x7fa628101fe0 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa61800de80 con 0x7fa628101fe0 2024-03-24T06:36:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61800f700 con 0x7fa628101fe0 2024-03-24T06:36:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.046+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6281a1880 con 0x7fa628101fe0 2024-03-24T06:36:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.047+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6281a1d40 con 0x7fa628101fe0 2024-03-24T06:36:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.048+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa628108d30 con 0x7fa628101fe0 2024-03-24T06:36:24.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.052+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa618010460 con 0x7fa628101fe0 2024-03-24T06:36:24.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.053+0000 7fa625600640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 0x7fa6080a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:24.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.053+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6180cd3c0 con 0x7fa628101fe0 2024-03-24T06:36:24.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.053+0000 7fa61f400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 0x7fa6080a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:24.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.054+0000 7fa61f400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 0x7fa6080a4580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa62819ac90 tx=0x7fa610009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:24.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.056+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa618096790 con 0x7fa628101fe0 2024-03-24T06:36:24.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 podman[72524]: 2024-03-24 06:36:23.874873286 +0000 UTC m=+0.404462054 container start 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T06:36:24.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 ceph-mds[72559]: -- 172.21.15.22:0/751671859 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56131f2c41a0 con 0x56131f29ac00 2024-03-24T06:36:24.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:23 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d[72539]: starting mds.d at 2024-03-24T06:36:24.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:24 smithi022 bash[72524]: 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f 2024-03-24T06:36:24.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:36:24 smithi022 systemd[1]: Started Ceph mds.d for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:24.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.312+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa62810cfc0 con 0x7fa628101fe0 2024-03-24T06:36:24.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.313+0000 7fa625600640 1 -- 172.21.15.22:0/1296931043 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fa618099e50 con 0x7fa628101fe0 2024-03-24T06:36:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:24.313 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-24T06:36:24.314 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 5 2024-03-24T06:36:24.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 msgr2=0x7fa6080a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:24.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 0x7fa6080a4580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa62819ac90 tx=0x7fa610009260 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 msgr2=0x7fa628199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa61800d950 tx=0x7fa61800dd50 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 shutdown_connections 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6080a20c0 0x7fa6080a4580 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa628107240 0x7fa6281a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa628104910 0x7fa628199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 --2- 172.21.15.22:0/1296931043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa628101fe0 0x7fa628199780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.316+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 >> 172.21.15.22:0/1296931043 conn(0x7fa6280fb9f0 msgr2=0x7fa628101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.317+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 shutdown_connections 2024-03-24T06:36:24.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:24.317+0000 7fa62d400640 1 -- 172.21.15.22:0/1296931043 wait complete. 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:24 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:24.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:24.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:24 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:24 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:25 smithi143 ceph-mon[36520]: Deploying daemon mds.e on smithi143 2024-03-24T06:36:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:25 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6836/4113284449,v1:172.21.15.22:6837/4113284449] up:boot 2024-03-24T06:36:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:25 smithi143 ceph-mon[36520]: fsmap 4 up:standby 2024-03-24T06:36:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:25 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:36:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:25 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1296931043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:25 smithi155 ceph-mon[36262]: Deploying daemon mds.e on smithi143 2024-03-24T06:36:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:25 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6836/4113284449,v1:172.21.15.22:6837/4113284449] up:boot 2024-03-24T06:36:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:25 smithi155 ceph-mon[36262]: fsmap 4 up:standby 2024-03-24T06:36:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:25 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:36:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:25 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1296931043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:25.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:25 smithi022 ceph-mon[32003]: Deploying daemon mds.e on smithi143 2024-03-24T06:36:25.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:25 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6836/4113284449,v1:172.21.15.22:6837/4113284449] up:boot 2024-03-24T06:36:25.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:25 smithi022 ceph-mon[32003]: fsmap 4 up:standby 2024-03-24T06:36:25.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:25 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:36:25.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:25 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1296931043' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:26.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:26 smithi143 ceph-mon[36520]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:26.428 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:26 smithi022 ceph-mon[32003]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:26 smithi155 ceph-mon[36262]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 -- 172.21.15.22:0/857233650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103950 msgr2=0x7f7bf8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 --2- 172.21.15.22:0/857233650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103950 0x7f7bf8103d70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009900 tx=0x7f7be802bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 -- 172.21.15.22:0/857233650 shutdown_connections 2024-03-24T06:36:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 --2- 172.21.15.22:0/857233650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8105500 0x7f7bf810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 --2- 172.21.15.22:0/857233650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8104b40 0x7f7bf8104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 --2- 172.21.15.22:0/857233650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103950 0x7f7bf8103d70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 -- 172.21.15.22:0/857233650 >> 172.21.15.22:0/857233650 conn(0x7f7bf80feec0 msgr2=0x7f7bf8101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 -- 172.21.15.22:0/857233650 shutdown_connections 2024-03-24T06:36:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.571+0000 7f7bff600640 1 -- 172.21.15.22:0/857233650 wait complete. 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.572+0000 7f7bff600640 1 Processor -- start 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.572+0000 7f7bff600640 1 -- start start 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103950 0x7f7bf81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8104b40 0x7f7bf8199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81a14e0 con 0x7f7bf8103950 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81a1650 con 0x7f7bf8105500 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bff600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81a1810 con 0x7f7bf8104b40 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfcc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103950 0x7f7bf81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46858/0 (socket says 172.21.15.22:46858) 2024-03-24T06:36:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 -- 172.21.15.22:0/3755135274 learned_addr learned my addr 172.21.15.22:0/3755135274 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7befe00640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8104b40 0x7f7bf8199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 -- 172.21.15.22:0/3755135274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8104b40 msgr2=0x7f7bf8199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8104b40 0x7f7bf8199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 -- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103950 msgr2=0x7f7bf81999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103950 0x7f7bf81999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.573+0000 7f7bfd600640 1 -- 172.21.15.22:0/3755135274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be8009560 con 0x7f7bf8105500 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bfd600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000eb10 tx=0x7f7bf000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0004b20 con 0x7f7bf8105500 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf0004c80 con 0x7f7bf8105500 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0010650 con 0x7f7bf8105500 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf81a1af0 con 0x7f7bf8105500 2024-03-24T06:36:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.574+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf81a1fb0 con 0x7f7bf8105500 2024-03-24T06:36:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.575+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf810ac70 con 0x7f7bf8105500 2024-03-24T06:36:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.577+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7bf0005020 con 0x7f7bf8105500 2024-03-24T06:36:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.578+0000 7f7bf6000640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 0x7f7bdc0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.579+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7bf0014070 con 0x7f7bf8105500 2024-03-24T06:36:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.579+0000 7f7bfcc00640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 0x7f7bdc0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.579+0000 7f7bfcc00640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 0x7f7bdc0a4600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7be802c240 tx=0x7f7be802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.582+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7bf00971d0 con 0x7f7bf8105500 2024-03-24T06:36:26.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.822+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bf802d230 con 0x7f7bf8105500 2024-03-24T06:36:26.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.823+0000 7f7bf6000640 1 -- 172.21.15.22:0/3755135274 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f7bf009a890 con 0x7f7bf8105500 2024-03-24T06:36:26.824 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:26.824 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-24T06:36:26.824 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 5 2024-03-24T06:36:26.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.826+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 msgr2=0x7f7bdc0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.826+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 0x7f7bdc0a4600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7be802c240 tx=0x7f7be802c7e0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.826+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 msgr2=0x7f7bf81a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.826+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000eb10 tx=0x7f7bf000c2d0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 shutdown_connections 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7bdc0a2140 0x7f7bdc0a4600 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8105500 0x7f7bf81a0fa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8104b40 0x7f7bf8199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 --2- 172.21.15.22:0/3755135274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103950 0x7f7bf81999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 >> 172.21.15.22:0/3755135274 conn(0x7f7bf80feec0 msgr2=0x7f7bf8107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 shutdown_connections 2024-03-24T06:36:26.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:26.827+0000 7f7bff600640 1 -- 172.21.15.22:0/3755135274 wait complete. 2024-03-24T06:36:27.178 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:36:26 smithi143 systemd[1]: Started Ceph mds.e for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3755135274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:27 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3755135274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:27.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:27.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:27 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3755135274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:27 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: Deploying daemon mds.f on smithi155 2024-03-24T06:36:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6834/1050031463,v1:172.21.15.143:6835/1050031463] up:boot 2024-03-24T06:36:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: fsmap 5 up:standby 2024-03-24T06:36:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:28 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: Deploying daemon mds.f on smithi155 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6834/1050031463,v1:172.21.15.143:6835/1050031463] up:boot 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: fsmap 5 up:standby 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:28 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: Deploying daemon mds.f on smithi155 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6834/1050031463,v1:172.21.15.143:6835/1050031463] up:boot 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: fsmap 5 up:standby 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:28 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:36:28.846 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 -- 172.21.15.22:0/2559345602 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78107b70 msgr2=0x7fbc78107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/2559345602 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78107b70 0x7fbc78107f90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009900 tx=0x7fbc6802bc50 comp rx=0 tx=0).stop 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 -- 172.21.15.22:0/2559345602 shutdown_connections 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/2559345602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78109720 0x7fbc7810ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/2559345602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc781091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/2559345602 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78107b70 0x7fbc78107f90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 -- 172.21.15.22:0/2559345602 >> 172.21.15.22:0/2559345602 conn(0x7fbc78076b20 msgr2=0x7fbc78078f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:29.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 -- 172.21.15.22:0/2559345602 shutdown_connections 2024-03-24T06:36:29.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.082+0000 7fbc7ca00640 1 -- 172.21.15.22:0/2559345602 wait complete. 2024-03-24T06:36:29.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.083+0000 7fbc7ca00640 1 Processor -- start 2024-03-24T06:36:29.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.083+0000 7fbc7ca00640 1 -- start start 2024-03-24T06:36:29.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.083+0000 7fbc7ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc7ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc7ca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78109720 0x7fbc781a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc7ca00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbc781a56e0 con 0x7fbc78109720 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc7ca00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbc781a5850 con 0x7fbc78108d60 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc7ca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbc781a5a10 con 0x7fbc78107b70 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc77200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78109720 0x7fbc781a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc75e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc76800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc75e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46870/0 (socket says 172.21.15.22:46870) 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc76800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41420/0 (socket says 172.21.15.22:41420) 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc75e00640 1 -- 172.21.15.22:0/1495594875 learned_addr learned my addr 172.21.15.22:0/1495594875 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.084+0000 7fbc76800640 1 -- 172.21.15.22:0/1495594875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 msgr2=0x7fbc7819e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc76800640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc76800640 1 -- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78109720 msgr2=0x7fbc781a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:29.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc76800640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78109720 0x7fbc781a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc76800640 1 -- 172.21.15.22:0/1495594875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc68009560 con 0x7fbc78107b70 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc75e00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc76800640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbc68036040 tx=0x7fbc680040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc680044c0 con 0x7fbc78107b70 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc680052d0 con 0x7fbc78107b70 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68003e20 con 0x7fbc78107b70 2024-03-24T06:36:29.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc781a5c90 con 0x7fbc78107b70 2024-03-24T06:36:29.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.085+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc781a6170 con 0x7fbc78107b70 2024-03-24T06:36:29.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.086+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7810ee90 con 0x7fbc78107b70 2024-03-24T06:36:29.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.088+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fbc68005440 con 0x7fbc78107b70 2024-03-24T06:36:29.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.089+0000 7fbc6f400640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 0x7fbc5c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:29.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.090+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbc680f0060 con 0x7fbc78107b70 2024-03-24T06:36:29.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.090+0000 7fbc75e00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 0x7fbc5c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:29.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.091+0000 7fbc75e00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 0x7fbc5c0a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbc7819f100 tx=0x7fbc6400b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:29.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.094+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc680b9430 con 0x7fbc78107b70 2024-03-24T06:36:29.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.334+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc781137e0 con 0x7fbc78107b70 2024-03-24T06:36:29.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.335+0000 7fbc6f400640 1 -- 172.21.15.22:0/1495594875 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7fbc680bcaf0 con 0x7fbc78107b70 2024-03-24T06:36:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:29.335 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-24T06:36:29.335 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 6 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.338+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 msgr2=0x7fbc5c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.338+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 0x7fbc5c0a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbc7819f100 tx=0x7fbc6400b480 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.338+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 msgr2=0x7fbc7819dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.338+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbc68036040 tx=0x7fbc680040c0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 shutdown_connections 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbc5c0a20c0 0x7fbc5c0a4580 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbc78109720 0x7fbc781a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc78108d60 0x7fbc7819e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 --2- 172.21.15.22:0/1495594875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc78107b70 0x7fbc7819dbf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 >> 172.21.15.22:0/1495594875 conn(0x7fbc78076b20 msgr2=0x7fbc7810c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 shutdown_connections 2024-03-24T06:36:29.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:29.339+0000 7fbc7ca00640 1 -- 172.21.15.22:0/1495594875 wait complete. 2024-03-24T06:36:29.497 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 systemd[1]: Starting Ceph mds.f for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:29.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 podman[67261]: 2024-03-24 06:36:29.400409921 +0000 UTC m=+0.018202532 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:29.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 podman[67261]: 2024-03-24 06:36:29.512897459 +0000 UTC m=+0.130690099 container create a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T06:36:29.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 podman[67261]: 2024-03-24 06:36:29.765422552 +0000 UTC m=+0.383215142 container init a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T06:36:30.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 podman[67261]: 2024-03-24 06:36:29.768226684 +0000 UTC m=+0.386019277 container start a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T06:36:30.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 ceph-mds[67280]: -- 172.21.15.155:0/4147148152 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d18fe101a0 con 0x55d18fde6800 2024-03-24T06:36:30.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: starting mds.f at 2024-03-24T06:36:30.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 bash[67261]: a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 2024-03-24T06:36:30.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:36:29 smithi155 systemd[1]: Started Ceph mds.f for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1495594875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:30.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:30.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:30 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1495594875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:30 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1495594875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:30 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:31.356 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:31 smithi155 ceph-mon[36262]: Deploying daemon mds.c on smithi155 2024-03-24T06:36:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:31 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:boot 2024-03-24T06:36:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:31 smithi155 ceph-mon[36262]: fsmap 6 up:standby 2024-03-24T06:36:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:31 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:36:31.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 -- 172.21.15.22:0/4010394446 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c105310 msgr2=0x7f7d4c10bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:31.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 --2- 172.21.15.22:0/4010394446 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c105310 0x7f7d4c10bba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009ca0 tx=0x7f7d4402c100 comp rx=0 tx=0).stop 2024-03-24T06:36:31.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 -- 172.21.15.22:0/4010394446 shutdown_connections 2024-03-24T06:36:31.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 --2- 172.21.15.22:0/4010394446 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c105310 0x7f7d4c10bba0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 --2- 172.21.15.22:0/4010394446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c104950 0x7f7d4c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 --2- 172.21.15.22:0/4010394446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c103760 0x7f7d4c103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 -- 172.21.15.22:0/4010394446 >> 172.21.15.22:0/4010394446 conn(0x7f7d4c0fecf0 msgr2=0x7f7d4c101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 -- 172.21.15.22:0/4010394446 shutdown_connections 2024-03-24T06:36:31.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.592+0000 7f7d50c00640 1 -- 172.21.15.22:0/4010394446 wait complete. 2024-03-24T06:36:31.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.593+0000 7f7d50c00640 1 Processor -- start 2024-03-24T06:36:31.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 -- start start 2024-03-24T06:36:31.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:31.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c104950 0x7f7d4c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:31.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c105310 0x7f7d4c1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c1a12a0 con 0x7f7d4c103760 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c1a1410 con 0x7f7d4c105310 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d50c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c1a15d0 con 0x7f7d4c104950 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d4b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d4b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33788/0 (socket says 172.21.15.22:33788) 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.594+0000 7f7d4b400640 1 -- 172.21.15.22:0/450465907 learned_addr learned my addr 172.21.15.22:0/450465907 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4aa00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c104950 0x7f7d4c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4be00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c105310 0x7f7d4c1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 -- 172.21.15.22:0/450465907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c104950 msgr2=0x7f7d4c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c104950 0x7f7d4c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 -- 172.21.15.22:0/450465907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c105310 msgr2=0x7f7d4c1a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c105310 0x7f7d4c1a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 -- 172.21.15.22:0/450465907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d44009900 con 0x7f7d4c103760 2024-03-24T06:36:31.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d4b400640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00e9c0 tx=0x7f7d3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c009700 con 0x7f7d4c103760 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d3c00cd80 con 0x7f7d4c103760 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4c1a18b0 con 0x7f7d4c103760 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.595+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c010430 con 0x7f7d4c103760 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.596+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4c1a1e10 con 0x7f7d4c103760 2024-03-24T06:36:31.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.597+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4c10aa80 con 0x7f7d4c103760 2024-03-24T06:36:31.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.599+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7d3c003680 con 0x7f7d4c103760 2024-03-24T06:36:31.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.601+0000 7f7d43e00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 0x7f7d300a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:31.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.602+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7d3c014070 con 0x7f7d4c103760 2024-03-24T06:36:31.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.602+0000 7f7d4aa00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 0x7f7d300a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:31.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.602+0000 7f7d4aa00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 0x7f7d300a4600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c19acc0 tx=0x7f7d38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:31.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.606+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7d3c097030 con 0x7f7d4c103760 2024-03-24T06:36:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:31 smithi022 ceph-mon[32003]: Deploying daemon mds.c on smithi155 2024-03-24T06:36:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:31 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:boot 2024-03-24T06:36:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:31 smithi022 ceph-mon[32003]: fsmap 6 up:standby 2024-03-24T06:36:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:31 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:36:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:31 smithi143 ceph-mon[36520]: Deploying daemon mds.c on smithi155 2024-03-24T06:36:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:31 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:boot 2024-03-24T06:36:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:31 smithi143 ceph-mon[36520]: fsmap 6 up:standby 2024-03-24T06:36:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:31 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:36:31.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.849+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d4c10f140 con 0x7f7d4c103760 2024-03-24T06:36:31.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.850+0000 7f7d43e00640 1 -- 172.21.15.22:0/450465907 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f7d3c09a6f0 con 0x7f7d4c103760 2024-03-24T06:36:31.850 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:31.851 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-24T06:36:31.851 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 7 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 msgr2=0x7f7d300a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 0x7f7d300a4600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c19acc0 tx=0x7f7d38005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 msgr2=0x7f7d4c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00e9c0 tx=0x7f7d3c00ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 shutdown_connections 2024-03-24T06:36:31.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d300a2140 0x7f7d300a4600 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c105310 0x7f7d4c1a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d4c104950 0x7f7d4c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 --2- 172.21.15.22:0/450465907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d4c103760 0x7f7d4c1997b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 >> 172.21.15.22:0/450465907 conn(0x7f7d4c0fecf0 msgr2=0x7f7d4c107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 shutdown_connections 2024-03-24T06:36:31.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:31.854+0000 7f7d50c00640 1 -- 172.21.15.22:0/450465907 wait complete. 2024-03-24T06:36:32.472 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:32 smithi155 ceph-mon[36262]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:32.472 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:32 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/450465907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:32.472 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 systemd[1]: Starting Ceph mds.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:32 smithi022 ceph-mon[32003]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:32 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/450465907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:32 smithi143 ceph-mon[36520]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:32 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/450465907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:32.733 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 podman[67568]: 2024-03-24 06:36:32.375686219 +0000 UTC m=+0.017420794 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:32.733 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 podman[67568]: 2024-03-24 06:36:32.615685167 +0000 UTC m=+0.257419743 container create b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:36:33.027 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 podman[67568]: 2024-03-24 06:36:32.851727243 +0000 UTC m=+0.493461815 container init b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T06:36:33.027 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 podman[67568]: 2024-03-24 06:36:32.854321855 +0000 UTC m=+0.496056405 container start b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:36:33.027 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 ceph-mds[67587]: -- 172.21.15.155:0/1829347402 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c0af6c1a0 con 0x559c0af42800 2024-03-24T06:36:33.027 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: starting mds.c at 2024-03-24T06:36:33.028 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 bash[67568]: b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 2024-03-24T06:36:33.028 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:36:32 smithi155 systemd[1]: Started Ceph mds.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:33.877 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 -- 172.21.15.22:0/1055721867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0105310 msgr2=0x7f92f010bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 --2- 172.21.15.22:0/1055721867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0105310 0x7f92f010bba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009ca0 tx=0x7f92e802c0f0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 -- 172.21.15.22:0/1055721867 shutdown_connections 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 --2- 172.21.15.22:0/1055721867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0105310 0x7f92f010bba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 --2- 172.21.15.22:0/1055721867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 0x7f92f0104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 --2- 172.21.15.22:0/1055721867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0103760 0x7f92f0103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 -- 172.21.15.22:0/1055721867 >> 172.21.15.22:0/1055721867 conn(0x7f92f00fecf0 msgr2=0x7f92f0101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:34.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 -- 172.21.15.22:0/1055721867 shutdown_connections 2024-03-24T06:36:34.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.113+0000 7f92f5800640 1 -- 172.21.15.22:0/1055721867 wait complete. 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.114+0000 7f92f5800640 1 Processor -- start 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.114+0000 7f92f5800640 1 -- start start 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 0x7f92f0199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0105310 0x7f92f01a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f92f01a1260 con 0x7f92f0103760 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92f01a13d0 con 0x7f92f0104950 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f5800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f92f01a1590 con 0x7f92f0105310 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33792/0 (socket says 172.21.15.22:33792) 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 -- 172.21.15.22:0/1947389280 learned_addr learned my addr 172.21.15.22:0/1947389280 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92ef400640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 0x7f92f0199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:34.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92f4e00640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0105310 0x7f92f01a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 -- 172.21.15.22:0/1947389280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0105310 msgr2=0x7f92f01a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0105310 0x7f92f01a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 -- 172.21.15.22:0/1947389280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 msgr2=0x7f92f0199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 0x7f92f0199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.115+0000 7f92efe00640 1 -- 172.21.15.22:0/1947389280 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc009490 con 0x7f92f0103760 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92efe00640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92e000e9c0 tx=0x7f92e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0009700 con 0x7f92f0103760 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e8009900 con 0x7f92f0103760 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92e000cd80 con 0x7f92f0103760 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e0010430 con 0x7f92f0103760 2024-03-24T06:36:34.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.116+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f01a1c40 con 0x7f92f0103760 2024-03-24T06:36:34.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.118+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f010aa80 con 0x7f92f0103760 2024-03-24T06:36:34.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.119+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f92e0003680 con 0x7f92f0103760 2024-03-24T06:36:34.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.121+0000 7f92ecc00640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 0x7f92d40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:34.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.121+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f92e0014070 con 0x7f92f0103760 2024-03-24T06:36:34.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.121+0000 7f92ef400640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 0x7f92d40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:34.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.122+0000 7f92ef400640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 0x7f92d40a4580 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f92f019ac80 tx=0x7f92dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:34.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.125+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f92e0097020 con 0x7f92f0103760 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: Deploying daemon mds.h on smithi143 2024-03-24T06:36:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:boot 2024-03-24T06:36:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: fsmap 7 up:standby 2024-03-24T06:36:34.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:34 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:36:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.368+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92f010f140 con 0x7f92f0103760 2024-03-24T06:36:34.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.369+0000 7f92ecc00640 1 -- 172.21.15.22:0/1947389280 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f92e009a6e0 con 0x7f92f0103760 2024-03-24T06:36:34.369 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:34.369 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-24T06:36:34.369 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 8 2024-03-24T06:36:34.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.372+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 msgr2=0x7f92d40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.372+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 0x7f92d40a4580 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f92f019ac80 tx=0x7f92dc005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.372+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 msgr2=0x7f92f0199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.372+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92e000e9c0 tx=0x7f92e000ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 shutdown_connections 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f92d40a20c0 0x7f92d40a4580 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92f0105310 0x7f92f01a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f0104950 0x7f92f0199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 --2- 172.21.15.22:0/1947389280 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92f0103760 0x7f92f0199770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 >> 172.21.15.22:0/1947389280 conn(0x7f92f00fecf0 msgr2=0x7f92f0107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 shutdown_connections 2024-03-24T06:36:34.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:34.373+0000 7f92f5800640 1 -- 172.21.15.22:0/1947389280 wait complete. 2024-03-24T06:36:34.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.388 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: Deploying daemon mds.h on smithi143 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:boot 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: fsmap 7 up:standby 2024-03-24T06:36:34.389 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:34 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:36:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: Deploying daemon mds.h on smithi143 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:boot 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: fsmap 7 up:standby 2024-03-24T06:36:34.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:34 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:36:35.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:35 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1947389280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:35.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:35 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1947389280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:35.429 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 systemd[1]: Starting Ceph mds.h for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:35.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:35 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1947389280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:35.816 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 podman[66280]: 2024-03-24 06:36:35.403848887 +0000 UTC m=+0.017665699 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:35.816 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 podman[66280]: 2024-03-24 06:36:35.522919096 +0000 UTC m=+0.136735915 container create 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T06:36:36.178 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 podman[66280]: 2024-03-24 06:36:35.817411252 +0000 UTC m=+0.431228071 container init 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:36:36.178 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 podman[66280]: 2024-03-24 06:36:35.820072715 +0000 UTC m=+0.433889514 container start 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:36:36.178 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 ceph-mds[66298]: -- 172.21.15.143:0/1362025180 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd6a1861a0 con 0x55bd6a15c800 2024-03-24T06:36:36.179 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: starting mds.h at 2024-03-24T06:36:36.179 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 bash[66280]: 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a 2024-03-24T06:36:36.179 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:36:35 smithi143 systemd[1]: Started Ceph mds.h for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:36.395 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:36.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 -- 172.21.15.22:0/2364184199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4341007b0 msgr2=0x7fd434100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:36.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 --2- 172.21.15.22:0/2364184199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4341007b0 0x7fd434100c10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009ca0 tx=0x7fd41c02c190 comp rx=0 tx=0).stop 2024-03-24T06:36:36.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 -- 172.21.15.22:0/2364184199 shutdown_connections 2024-03-24T06:36:36.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 --2- 172.21.15.22:0/2364184199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd434101150 0x7fd4341015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 --2- 172.21.15.22:0/2364184199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4341007b0 0x7fd434100c10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 --2- 172.21.15.22:0/2364184199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4340ffe50 0x7fd434100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 -- 172.21.15.22:0/2364184199 >> 172.21.15.22:0/2364184199 conn(0x7fd4340fb9f0 msgr2=0x7fd4340fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:36.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 -- 172.21.15.22:0/2364184199 shutdown_connections 2024-03-24T06:36:36.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.632+0000 7fd433e00640 1 -- 172.21.15.22:0/2364184199 wait complete. 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 Processor -- start 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 -- start start 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4341007b0 0x7fd434199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd434101150 0x7fd4341a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341a1300 con 0x7fd4341007b0 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341a1470 con 0x7fd434101150 2024-03-24T06:36:36.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.633+0000 7fd433e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341a1630 con 0x7fd4340ffe50 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54674/0 (socket says 172.21.15.22:54674) 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 -- 172.21.15.22:0/1793851171 learned_addr learned my addr 172.21.15.22:0/1793851171 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432000640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4341007b0 0x7fd434199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd433400640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd434101150 0x7fd4341a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 -- 172.21.15.22:0/1793851171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd434101150 msgr2=0x7fd4341a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd434101150 0x7fd4341a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 -- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4341007b0 msgr2=0x7fd434199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4341007b0 0x7fd434199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 -- 172.21.15.22:0/1793851171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd424009490 con 0x7fd4340ffe50 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.634+0000 7fd432a00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd42400ea60 tx=0x7fd42400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.635+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42400cd20 con 0x7fd4340ffe50 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.635+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd42400ce80 con 0x7fd4340ffe50 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.635+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42c009490 con 0x7fd4340ffe50 2024-03-24T06:36:36.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.635+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424010690 con 0x7fd4340ffe50 2024-03-24T06:36:36.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.635+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c009900 con 0x7fd4340ffe50 2024-03-24T06:36:36.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.636+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43410aba0 con 0x7fd4340ffe50 2024-03-24T06:36:36.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.638+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd424003ee0 con 0x7fd4340ffe50 2024-03-24T06:36:36.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.639+0000 7fd42b400640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 0x7fd4140a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:36.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.639+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd424014070 con 0x7fd4340ffe50 2024-03-24T06:36:36.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.639+0000 7fd432000640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 0x7fd4140a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:36.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.640+0000 7fd432000640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 0x7fd4140a4600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd41c02c7a0 tx=0x7fd41c0324e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:36.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.643+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd4240970f0 con 0x7fd4340ffe50 2024-03-24T06:36:36.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.886+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd43410f140 con 0x7fd4340ffe50 2024-03-24T06:36:36.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.887+0000 7fd42b400640 1 -- 172.21.15.22:0/1793851171 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fd42409a7b0 con 0x7fd4340ffe50 2024-03-24T06:36:36.887 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:36.887 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-24T06:36:36.887 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 9 2024-03-24T06:36:36.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 msgr2=0x7fd4140a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:36.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 0x7fd4140a4600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd41c02c7a0 tx=0x7fd41c0324e0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 msgr2=0x7fd434199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:36.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd42400ea60 tx=0x7fd42400ee60 comp rx=0 tx=0).stop 2024-03-24T06:36:36.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 shutdown_connections 2024-03-24T06:36:36.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd4140a2140 0x7fd4140a4600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd434101150 0x7fd4341a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd4341007b0 0x7fd434199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 --2- 172.21.15.22:0/1793851171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd4340ffe50 0x7fd434199810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 >> 172.21.15.22:0/1793851171 conn(0x7fd4340fb9f0 msgr2=0x7fd4340fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 shutdown_connections 2024-03-24T06:36:36.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:36.890+0000 7fd433e00640 1 -- 172.21.15.22:0/1793851171 wait complete. 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:37.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6836/988781970,v1:172.21.15.143:6837/988781970] up:boot 2024-03-24T06:36:37.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: fsmap 8 up:standby 2024-03-24T06:36:37.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:36:37.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1793851171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:37.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:37.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6836/988781970,v1:172.21.15.143:6837/988781970] up:boot 2024-03-24T06:36:37.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: fsmap 8 up:standby 2024-03-24T06:36:37.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:36:37.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1793851171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:37.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:37.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6836/988781970,v1:172.21.15.143:6837/988781970] up:boot 2024-03-24T06:36:37.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: fsmap 8 up:standby 2024-03-24T06:36:37.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:36:37.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1793851171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:38.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:37 smithi155 ceph-mon[36262]: Deploying daemon mds.i on smithi155 2024-03-24T06:36:38.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:37 smithi022 ceph-mon[32003]: Deploying daemon mds.i on smithi155 2024-03-24T06:36:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:37 smithi143 ceph-mon[36520]: Deploying daemon mds.i on smithi155 2024-03-24T06:36:38.527 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 systemd[1]: Starting Ceph mds.i for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:38.888 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 podman[67879]: 2024-03-24 06:36:38.53642365 +0000 UTC m=+0.017936214 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:38.889 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 podman[67879]: 2024-03-24 06:36:38.654224822 +0000 UTC m=+0.135737412 container create 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T06:36:38.889 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 podman[67879]: 2024-03-24 06:36:38.889920505 +0000 UTC m=+0.371433056 container init 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:36:38.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:39.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 -- 172.21.15.22:0/3445237440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 msgr2=0x7f55b0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:39.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 --2- 172.21.15.22:0/3445237440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0100a50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009900 tx=0x7f55a002bc40 comp rx=0 tx=0).stop 2024-03-24T06:36:39.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 -- 172.21.15.22:0/3445237440 shutdown_connections 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 --2- 172.21.15.22:0/3445237440 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b01021e0 0x7f55b0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 --2- 172.21.15.22:0/3445237440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b0101820 0x7f55b0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 --2- 172.21.15.22:0/3445237440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0100a50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 -- 172.21.15.22:0/3445237440 >> 172.21.15.22:0/3445237440 conn(0x7f55b00fbba0 msgr2=0x7f55b00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 -- 172.21.15.22:0/3445237440 shutdown_connections 2024-03-24T06:36:39.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.146+0000 7f55b4e00640 1 -- 172.21.15.22:0/3445237440 wait complete. 2024-03-24T06:36:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.147+0000 7f55b4e00640 1 Processor -- start 2024-03-24T06:36:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.147+0000 7f55b4e00640 1 -- start start 2024-03-24T06:36:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b0101820 0x7f55b019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b01021e0 0x7f55b01a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f55b01a1540 con 0x7f55b0101820 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f55b01a16b0 con 0x7f55b0100630 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55b4e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f55b01a1870 con 0x7f55b01021e0 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b0101820 0x7f55b019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58102/0 (socket says 172.21.15.22:58102) 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 -- 172.21.15.22:0/877039741 learned_addr learned my addr 172.21.15.22:0/877039741 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ae800640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b01021e0 0x7f55b01a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 -- 172.21.15.22:0/877039741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b01021e0 msgr2=0x7f55b01a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b01021e0 0x7f55b01a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 -- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b0101820 msgr2=0x7f55b019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b0101820 0x7f55b019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.148+0000 7f55ade00640 1 -- 172.21.15.22:0/877039741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a0009560 con 0x7f55b0100630 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55ade00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a00043d0 con 0x7f55b0100630 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55a00052d0 con 0x7f55b0100630 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b01a1af0 con 0x7f55b0100630 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0003d30 con 0x7f55b0100630 2024-03-24T06:36:39.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.149+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b01a2000 con 0x7f55b0100630 2024-03-24T06:36:39.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.151+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b010c060 con 0x7f55b0100630 2024-03-24T06:36:39.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.152+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f55a0005440 con 0x7f55b0100630 2024-03-24T06:36:39.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.153+0000 7f55a6a00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 0x7f55940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:39.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.154+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55a00f0170 con 0x7f55b0100630 2024-03-24T06:36:39.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.154+0000 7f55ad400640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 0x7f55940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:39.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.155+0000 7f55ad400640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 0x7f55940a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f55b019af60 tx=0x7f559c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.158+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f55a00b9660 con 0x7f55b0100630 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:38 smithi155 ceph-mon[36262]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 podman[67879]: 2024-03-24 06:36:38.892623366 +0000 UTC m=+0.374135926 container start 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 ceph-mds[67898]: -- 172.21.15.155:0/2561289313 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627c03161a0 con 0x5627c02ecc00 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: starting mds.i at 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 bash[67879]: 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c 2024-03-24T06:36:39.277 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:36:38 smithi155 systemd[1]: Started Ceph mds.i for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:39.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:38 smithi022 ceph-mon[32003]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:39.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.410+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55b010c270 con 0x7f55b0100630 2024-03-24T06:36:39.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.411+0000 7f55a6a00640 1 -- 172.21.15.22:0/877039741 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f55a00bcd20 con 0x7f55b0100630 2024-03-24T06:36:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T06:36:39.410 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 10 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 msgr2=0x7f55940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 0x7f55940a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f55b019af60 tx=0x7f559c005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 msgr2=0x7f55b0199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a0005e80 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 shutdown_connections 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f55940a20c0 0x7f55940a4580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b01021e0 0x7f55b01a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55b0101820 0x7f55b019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 --2- 172.21.15.22:0/877039741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b0100630 0x7f55b0199ac0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 >> 172.21.15.22:0/877039741 conn(0x7f55b00fbba0 msgr2=0x7f55b00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:39.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 shutdown_connections 2024-03-24T06:36:39.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:39.414+0000 7f55b4e00640 1 -- 172.21.15.22:0/877039741 wait complete. 2024-03-24T06:36:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:38 smithi143 ceph-mon[36520]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:40.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:boot 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: fsmap 9 up:standby 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: Deploying daemon mds.g on smithi022 2024-03-24T06:36:40.267 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:39 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/877039741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:boot 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: fsmap 9 up:standby 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:40.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:40.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: Deploying daemon mds.g on smithi022 2024-03-24T06:36:40.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:39 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/877039741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:boot 2024-03-24T06:36:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: fsmap 9 up:standby 2024-03-24T06:36:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:36:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: Deploying daemon mds.g on smithi022 2024-03-24T06:36:40.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:39 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/877039741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:41.434 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:41.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.675+0000 7f3440a00640 1 -- 172.21.15.22:0/1596386159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c10b340 msgr2=0x7f343c10d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.675+0000 7f3440a00640 1 --2- 172.21.15.22:0/1596386159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c10b340 0x7f343c10d750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f342c007380 tx=0x7f342c02b6e0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 -- 172.21.15.22:0/1596386159 shutdown_connections 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 --2- 172.21.15.22:0/1596386159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c10dde0 0x7f343c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 --2- 172.21.15.22:0/1596386159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c10b340 0x7f343c10d750 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f342c007380 tx=0x7f342c02b6e0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 --2- 172.21.15.22:0/1596386159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c108980 0x7f343c10ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 -- 172.21.15.22:0/1596386159 >> 172.21.15.22:0/1596386159 conn(0x7f343c06e580 msgr2=0x7f343c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:41.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 -- 172.21.15.22:0/1596386159 shutdown_connections 2024-03-24T06:36:41.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.676+0000 7f3440a00640 1 -- 172.21.15.22:0/1596386159 wait complete. 2024-03-24T06:36:41.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 Processor -- start 2024-03-24T06:36:41.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 -- start start 2024-03-24T06:36:41.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 0x7f343c1a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c10dde0 0x7f343c1a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1ae6a0 con 0x7f343c108980 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1ae810 con 0x7f343c10dde0 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.678+0000 7f3440a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1ae9d0 con 0x7f343c1abd70 2024-03-24T06:36:41.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.679+0000 7f343a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:41.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.679+0000 7f343a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54712/0 (socket says 172.21.15.22:54712) 2024-03-24T06:36:41.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.679+0000 7f343a800640 1 -- 172.21.15.22:0/1615059269 learned_addr learned my addr 172.21.15.22:0/1615059269 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:41.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f3439e00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 0x7f343c1a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:41.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f343a800640 1 -- 172.21.15.22:0/1615059269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c10dde0 msgr2=0x7f343c1a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f343a800640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c10dde0 0x7f343c1a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f343a800640 1 -- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 msgr2=0x7f343c1a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f343a800640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 0x7f343c1a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.680+0000 7f343a800640 1 -- 172.21.15.22:0/1615059269 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c00ae30 con 0x7f343c1abd70 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f343a800640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f343400ccb0 tx=0x7f343400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3439e00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 0x7f343c1a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343400f450 con 0x7f343c1abd70 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f343400fa90 con 0x7f343c1abd70 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434018c00 con 0x7f343c1abd70 2024-03-24T06:36:41.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c1aecb0 con 0x7f343c1abd70 2024-03-24T06:36:41.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.681+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c1af1c0 con 0x7f343c1abd70 2024-03-24T06:36:41.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.684+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343c112d40 con 0x7f343c1abd70 2024-03-24T06:36:41.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.688+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f343400fc00 con 0x7f343c1abd70 2024-03-24T06:36:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.690+0000 7f3432a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 0x7f34200a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.690+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f34340cdc10 con 0x7f343c1abd70 2024-03-24T06:36:41.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.691+0000 7f3439e00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 0x7f34200a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:41.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.692+0000 7f3439e00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 0x7f34200a4600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f34240058c0 tx=0x7f3424005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:41.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.693+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3434097080 con 0x7f343c1abd70 2024-03-24T06:36:41.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.941+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f343c1af470 con 0x7f343c1abd70 2024-03-24T06:36:41.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.942+0000 7f3432a00640 1 -- 172.21.15.22:0/1615059269 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f343409a740 con 0x7f343c1abd70 2024-03-24T06:36:41.941 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:41.941 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T06:36:41.942 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 10 2024-03-24T06:36:41.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 msgr2=0x7f34200a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:41.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 0x7f34200a4600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f34240058c0 tx=0x7f3424005850 comp rx=0 tx=0).stop 2024-03-24T06:36:41.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 msgr2=0x7f343c1ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:41.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f343400ccb0 tx=0x7f343400a620 comp rx=0 tx=0).stop 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 shutdown_connections 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34200a2140 0x7f34200a4600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343c1abd70 0x7f343c1ae160 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c10dde0 0x7f343c1a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 --2- 172.21.15.22:0/1615059269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343c108980 0x7f343c1a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 >> 172.21.15.22:0/1615059269 conn(0x7f343c06e580 msgr2=0x7f343c109500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 shutdown_connections 2024-03-24T06:36:41.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:41.945+0000 7f3440a00640 1 -- 172.21.15.22:0/1615059269 wait complete. 2024-03-24T06:36:42.271 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:42 smithi022 ceph-mon[32003]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:42.271 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1615059269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:42 smithi143 ceph-mon[36520]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:42 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1615059269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:42.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:42 smithi155 ceph-mon[36262]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:42.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:42 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1615059269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:42.635 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:42 smithi022 systemd[1]: Starting Ceph mds.g for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:42.899 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:42 smithi022 podman[73170]: 2024-03-24 06:36:42.540412536 +0000 UTC m=+0.037424292 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:42.899 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:42 smithi022 podman[73170]: 2024-03-24 06:36:42.668951586 +0000 UTC m=+0.165963379 container create e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:42 smithi022 podman[73170]: 2024-03-24 06:36:42.989736588 +0000 UTC m=+0.486748388 container init e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:43 smithi022 podman[73170]: 2024-03-24 06:36:42.999918739 +0000 UTC m=+0.496930505 container start e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:43 smithi022 ceph-mds[73189]: -- 172.21.15.22:0/1541058937 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ad25341a0 con 0x555ad250b000 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:43 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g[73185]: starting mds.g at 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:43 smithi022 bash[73170]: e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 2024-03-24T06:36:43.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:36:43 smithi022 systemd[1]: Started Ceph mds.g for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:43.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:44.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 -- 172.21.15.22:0/1116904717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec0760d0 msgr2=0x7f8eec0764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:44.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 --2- 172.21.15.22:0/1116904717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec0760d0 0x7f8eec0764f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009900 tx=0x7f8ee002bc50 comp rx=0 tx=0).stop 2024-03-24T06:36:44.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 -- 172.21.15.22:0/1116904717 shutdown_connections 2024-03-24T06:36:44.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 --2- 172.21.15.22:0/1116904717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec078550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 --2- 172.21.15.22:0/1116904717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec077310 0x7f8eec077770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 --2- 172.21.15.22:0/1116904717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec0760d0 0x7f8eec0764f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 -- 172.21.15.22:0/1116904717 >> 172.21.15.22:0/1116904717 conn(0x7f8eec0feec0 msgr2=0x7f8eec101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 -- 172.21.15.22:0/1116904717 shutdown_connections 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.177+0000 7f8ef4600640 1 -- 172.21.15.22:0/1116904717 wait complete. 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 Processor -- start 2024-03-24T06:36:44.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 -- start start 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec1a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec1a5880 con 0x7f8eec077310 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec1a59f0 con 0x7f8eec0760d0 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.179+0000 7f8ef4600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8eec1a5bb0 con 0x7f8eec077cb0 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58152/0 (socket says 172.21.15.22:58152) 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1c00640 1 -- 172.21.15.22:0/2086200687 learned_addr learned my addr 172.21.15.22:0/2086200687 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45450/0 (socket says 172.21.15.22:45450) 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef2600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec1a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:44.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 -- 172.21.15.22:0/2086200687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 msgr2=0x7f8eec1a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec1a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 -- 172.21.15.22:0/2086200687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 msgr2=0x7f8eec19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 -- 172.21.15.22:0/2086200687 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee0009560 con 0x7f8eec077310 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef2600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec1a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1c00640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.180+0000 7f8ef1200640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800e9f0 tx=0x7f8ed800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.181+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800ccc0 con 0x7f8eec077310 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.181+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ed800ce20 con 0x7f8eec077310 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.181+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eec1a5e90 con 0x7f8eec077310 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.181+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8018920 con 0x7f8eec077310 2024-03-24T06:36:44.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.181+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eec1a62b0 con 0x7f8eec077310 2024-03-24T06:36:44.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.182+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eec110450 con 0x7f8eec077310 2024-03-24T06:36:44.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.185+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8ed8018a80 con 0x7f8eec077310 2024-03-24T06:36:44.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.188+0000 7f8eea000640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 0x7f8ed40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:44.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.188+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8ed8014070 con 0x7f8eec077310 2024-03-24T06:36:44.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.188+0000 7f8ef1c00640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 0x7f8ed40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:44.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.189+0000 7f8ef1c00640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 0x7f8ed40a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee002c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:44.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.193+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ed8053280 con 0x7f8eec077310 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6838/2318828115,v1:172.21.15.22:6839/2318828115] up:boot 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: fsmap 10 up:standby 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:44.289 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:44 smithi022 ceph-mon[32003]: Deploying daemon mds.a on smithi022 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6838/2318828115,v1:172.21.15.22:6839/2318828115] up:boot 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: fsmap 10 up:standby 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:44.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:44.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:44.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:44 smithi143 ceph-mon[36520]: Deploying daemon mds.a on smithi022 2024-03-24T06:36:44.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.440+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8eec113950 con 0x7f8eec077310 2024-03-24T06:36:44.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.441+0000 7f8eea000640 1 -- 172.21.15.22:0/2086200687 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f8ed809ab60 con 0x7f8eec077310 2024-03-24T06:36:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:44.440 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T06:36:44.440 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 11 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.443+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 msgr2=0x7f8ed40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.443+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 0x7f8ed40a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee002c800 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 msgr2=0x7f8eec19e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800e9f0 tx=0x7f8ed800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 shutdown_connections 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ed40a20c0 0x7f8ed40a4580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8eec077cb0 0x7f8eec1a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8eec077310 0x7f8eec19e2e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 --2- 172.21.15.22:0/2086200687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8eec0760d0 0x7f8eec19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 >> 172.21.15.22:0/2086200687 conn(0x7f8eec0feec0 msgr2=0x7f8eec1005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 shutdown_connections 2024-03-24T06:36:44.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:44.444+0000 7f8ef4600640 1 -- 172.21.15.22:0/2086200687 wait complete. 2024-03-24T06:36:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6838/2318828115,v1:172.21.15.22:6839/2318828115] up:boot 2024-03-24T06:36:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: fsmap 10 up:standby 2024-03-24T06:36:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:44.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:44 smithi155 ceph-mon[36262]: Deploying daemon mds.a on smithi022 2024-03-24T06:36:45.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:45 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2086200687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:45.465 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:45 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2086200687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:45.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:45 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2086200687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:46.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:46 smithi022 ceph-mon[32003]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:46 smithi143 ceph-mon[36520]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:46.461 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:46 smithi155 ceph-mon[36262]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 -- 172.21.15.22:0/2044885636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 msgr2=0x7f36281045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 --2- 172.21.15.22:0/2044885636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f36281045c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3618009900 tx=0x7f361802bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 -- 172.21.15.22:0/2044885636 shutdown_connections 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 --2- 172.21.15.22:0/2044885636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 0x7f3628109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 --2- 172.21.15.22:0/2044885636 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 0x7f3628106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 --2- 172.21.15.22:0/2044885636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f36281045c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 -- 172.21.15.22:0/2044885636 >> 172.21.15.22:0/2044885636 conn(0x7f36280fbbc0 msgr2=0x7f36280fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:46.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 -- 172.21.15.22:0/2044885636 shutdown_connections 2024-03-24T06:36:46.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.667+0000 7f362e600640 1 -- 172.21.15.22:0/2044885636 wait complete. 2024-03-24T06:36:46.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.668+0000 7f362e600640 1 Processor -- start 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 -- start start 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 0x7f3628199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 0x7f36281a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f36281a1500 con 0x7f3628104b00 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f36281a1670 con 0x7f36281021d0 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f362e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f36281a1830 con 0x7f3628107430 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f3627400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f3627400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60874/0 (socket says 172.21.15.22:60874) 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f3627400640 1 -- 172.21.15.22:0/1186566712 learned_addr learned my addr 172.21.15.22:0/1186566712 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f3626a00640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 0x7f3628199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.669+0000 7f3627e00640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 0x7f36281a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 -- 172.21.15.22:0/1186566712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 msgr2=0x7f36281a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:46.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 0x7f36281a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 -- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 msgr2=0x7f3628199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:46.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 0x7f3628199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 -- 172.21.15.22:0/1186566712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618009560 con 0x7f36281021d0 2024-03-24T06:36:46.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f3627400640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f361802c240 tx=0x7f3618004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618004440 con 0x7f36281021d0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36180052d0 con 0x7f36281021d0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618003e40 con 0x7f36281021d0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.670+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36281a1ab0 con 0x7f36281021d0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.671+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36281a1f70 con 0x7f36281021d0 2024-03-24T06:36:46.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.672+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3628108f80 con 0x7f36281021d0 2024-03-24T06:36:46.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.673+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3618005440 con 0x7f36281021d0 2024-03-24T06:36:46.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.675+0000 7f361fe00640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 0x7f360c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:46.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.675+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36180f0190 con 0x7f36281021d0 2024-03-24T06:36:46.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.675+0000 7f3626a00640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 0x7f360c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:46.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.676+0000 7f3626a00640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 0x7f360c0a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f362819af20 tx=0x7f3614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:46.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.682+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36180b9680 con 0x7f36281021d0 2024-03-24T06:36:46.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.938+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f362802d130 con 0x7f36281021d0 2024-03-24T06:36:46.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.939+0000 7f361fe00640 1 -- 172.21.15.22:0/1186566712 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f36180bcd40 con 0x7f36281021d0 2024-03-24T06:36:46.939 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:46.939 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T06:36:46.939 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 11 2024-03-24T06:36:46.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.942+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 msgr2=0x7f360c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.942+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 0x7f360c0a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f362819af20 tx=0x7f3614005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.942+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 msgr2=0x7f3628199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.942+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f361802c240 tx=0x7f3618004040 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 shutdown_connections 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f360c0a20c0 0x7f360c0a4580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3628107430 0x7f36281a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3628104b00 0x7f3628199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 --2- 172.21.15.22:0/1186566712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36281021d0 0x7f3628199a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 >> 172.21.15.22:0/1186566712 conn(0x7f36280fbbc0 msgr2=0x7f36281018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 shutdown_connections 2024-03-24T06:36:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:46.943+0000 7f362e600640 1 -- 172.21.15.22:0/1186566712 wait complete. 2024-03-24T06:36:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1186566712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:47.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1186566712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:47.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1186566712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:47.649 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:36:47 smithi022 systemd[1]: Started Ceph mds.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: Deploying daemon mds.b on smithi143 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:boot 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: fsmap 11 up:standby 2024-03-24T06:36:48.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:48 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: Deploying daemon mds.b on smithi143 2024-03-24T06:36:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:boot 2024-03-24T06:36:48.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: fsmap 11 up:standby 2024-03-24T06:36:48.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:48 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: Deploying daemon mds.b on smithi143 2024-03-24T06:36:48.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:boot 2024-03-24T06:36:48.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: fsmap 11 up:standby 2024-03-24T06:36:48.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:48 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:36:48.962 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:49.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 -- 172.21.15.22:0/478922674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444102920 msgr2=0x7f0444102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:49.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 --2- 172.21.15.22:0/478922674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444102920 0x7f0444102da0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0430009900 tx=0x7f043002bc40 comp rx=0 tx=0).stop 2024-03-24T06:36:49.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 -- 172.21.15.22:0/478922674 shutdown_connections 2024-03-24T06:36:49.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 --2- 172.21.15.22:0/478922674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04441032e0 0x7f0444109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 --2- 172.21.15.22:0/478922674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444102920 0x7f0444102da0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 --2- 172.21.15.22:0/478922674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0444101730 0x7f0444101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 -- 172.21.15.22:0/478922674 >> 172.21.15.22:0/478922674 conn(0x7f04440fccc0 msgr2=0x7f04440ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 -- 172.21.15.22:0/478922674 shutdown_connections 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.200+0000 7f044ba00640 1 -- 172.21.15.22:0/478922674 wait complete. 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.202+0000 7f044ba00640 1 Processor -- start 2024-03-24T06:36:49.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.202+0000 7f044ba00640 1 -- start start 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.202+0000 7f044ba00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.202+0000 7f044ba00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0444102920 0x7f0444199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f044ba00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04441032e0 0x7f04441a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f044ba00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f04441a14a0 con 0x7f0444102920 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f044ba00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f04441a1610 con 0x7f0444101730 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f044ba00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f04441a17d0 con 0x7f04441032e0 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60896/0 (socket says 172.21.15.22:60896) 2024-03-24T06:36:49.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 -- 172.21.15.22:0/129272143 learned_addr learned my addr 172.21.15.22:0/129272143 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0443e00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0444102920 0x7f0444199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449a00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04441032e0 0x7f04441a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 -- 172.21.15.22:0/129272143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04441032e0 msgr2=0x7f04441a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04441032e0 0x7f04441a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 -- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0444102920 msgr2=0x7f0444199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0444102920 0x7f0444199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.203+0000 7f0449000640 1 -- 172.21.15.22:0/129272143 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0430009560 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f0449000640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f043800d950 tx=0x7f043800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438009c30 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0438009d90 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04441a1ab0 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043800f850 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.204+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441a1fe0 con 0x7f0444101730 2024-03-24T06:36:49.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.205+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044410d120 con 0x7f0444101730 2024-03-24T06:36:49.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.207+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f043800f9b0 con 0x7f0444101730 2024-03-24T06:36:49.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.209+0000 7f0441600640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 0x7f042c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:49.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.209+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f04380cd470 con 0x7f0444101730 2024-03-24T06:36:49.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.209+0000 7f0443e00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 0x7f042c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:49.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.210+0000 7f0443e00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 0x7f042c0a4600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0430009230 tx=0x7f0430005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:49.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.213+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0438096840 con 0x7f0444101730 2024-03-24T06:36:49.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.459+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04441a2230 con 0x7f0444101730 2024-03-24T06:36:49.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.460+0000 7f0441600640 1 -- 172.21.15.22:0/129272143 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f0438099f00 con 0x7f0444101730 2024-03-24T06:36:49.461 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:49.461 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24601,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-24T06:36:49.461 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 12 2024-03-24T06:36:49.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.463+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 msgr2=0x7f042c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.463+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 0x7f042c0a4600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0430009230 tx=0x7f0430005fd0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.463+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 msgr2=0x7f04441999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.463+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f043800d950 tx=0x7f043800de10 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 shutdown_connections 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f042c0a2140 0x7f042c0a4600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04441032e0 0x7f04441a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0444102920 0x7f0444199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 --2- 172.21.15.22:0/129272143 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0444101730 0x7f04441999b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 >> 172.21.15.22:0/129272143 conn(0x7f04440fccc0 msgr2=0x7f0444073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 shutdown_connections 2024-03-24T06:36:49.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:49.464+0000 7f044ba00640 1 -- 172.21.15.22:0/129272143 wait complete. 2024-03-24T06:36:49.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/129272143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:49.880 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:49 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/129272143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:49.880 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:49 smithi143 systemd[1]: Starting Ceph mds.b for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:36:49.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:49 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/129272143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:50.132 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:49 smithi143 podman[66598]: 2024-03-24 06:36:49.784296656 +0000 UTC m=+0.017555542 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:36:50.132 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:49 smithi143 podman[66598]: 2024-03-24 06:36:49.906421702 +0000 UTC m=+0.139680593 container create 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:36:50.132 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 podman[66598]: 2024-03-24 06:36:50.13301197 +0000 UTC m=+0.366270857 container init 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T06:36:50.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 podman[66598]: 2024-03-24 06:36:50.135972331 +0000 UTC m=+0.369231212 container start 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True) 2024-03-24T06:36:50.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mds[66617]: -- 172.21.15.143:0/4210334817 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d5f04681a0 con 0x55d5f043ec00 2024-03-24T06:36:50.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: starting mds.b at 2024-03-24T06:36:50.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 bash[66598]: 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 2024-03-24T06:36:50.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:36:50 smithi143 systemd[1]: Started Ceph mds.b for 7bca48fe-e9a7-11ee-95cd-87774f69a715. 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:50.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:51.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 -- 172.21.15.22:0/2375506746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0101fe0 msgr2=0x7f16d01043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 --2- 172.21.15.22:0/2375506746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0101fe0 0x7f16d01043d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009900 tx=0x7f16c002bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 -- 172.21.15.22:0/2375506746 shutdown_connections 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 --2- 172.21.15.22:0/2375506746 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0107240 0x7f16d0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 --2- 172.21.15.22:0/2375506746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0104910 0x7f16d0106d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 --2- 172.21.15.22:0/2375506746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0101fe0 0x7f16d01043d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 -- 172.21.15.22:0/2375506746 >> 172.21.15.22:0/2375506746 conn(0x7f16d00fb9f0 msgr2=0x7f16d00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 -- 172.21.15.22:0/2375506746 shutdown_connections 2024-03-24T06:36:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.686+0000 7f16d4a00640 1 -- 172.21.15.22:0/2375506746 wait complete. 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 Processor -- start 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 -- start start 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 0x7f16d01997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 0x7f16d0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0107240 0x7f16d01a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f16d01a12e0 con 0x7f16d0101fe0 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f16d01a1450 con 0x7f16d0107240 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.687+0000 7f16d4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f16d01a1610 con 0x7f16d0104910 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16cde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 0x7f16d0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 0x7f16d01997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16cde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 0x7f16d0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54764/0 (socket says 172.21.15.22:54764) 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16cde00640 1 -- 172.21.15.22:0/307009385 learned_addr learned my addr 172.21.15.22:0/307009385 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16cf200640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0107240 0x7f16d01a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 -- 172.21.15.22:0/307009385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 msgr2=0x7f16d0199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 0x7f16d0199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 -- 172.21.15.22:0/307009385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0107240 msgr2=0x7f16d01a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0107240 0x7f16d01a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 -- 172.21.15.22:0/307009385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c0009560 con 0x7f16d0101fe0 2024-03-24T06:36:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.688+0000 7f16ce800640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 0x7f16d01997f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f16c0000c00 tx=0x7f16c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.689+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0004aa0 con 0x7f16d0101fe0 2024-03-24T06:36:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.689+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c00052d0 con 0x7f16d0101fe0 2024-03-24T06:36:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.689+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01a1890 con 0x7f16d0101fe0 2024-03-24T06:36:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.689+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0003e80 con 0x7f16d0101fe0 2024-03-24T06:36:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.689+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d01a1d70 con 0x7f16d0101fe0 2024-03-24T06:36:51.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.690+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d0108d90 con 0x7f16d0101fe0 2024-03-24T06:36:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.693+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f16c0005440 con 0x7f16d0101fe0 2024-03-24T06:36:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.695+0000 7f16c7400640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 0x7f16b40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.695+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f16c00f0250 con 0x7f16d0101fe0 2024-03-24T06:36:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.695+0000 7f16cde00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 0x7f16b40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.696+0000 7f16cde00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 0x7f16b40a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f16d019ad00 tx=0x7f16bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.697+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f16c00b9620 con 0x7f16d0101fe0 2024-03-24T06:36:51.772 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:51 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:boot 2024-03-24T06:36:51.772 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:51 smithi143 ceph-mon[36520]: fsmap 12 up:standby 2024-03-24T06:36:51.772 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:51 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:36:51.772 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:51 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:51.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:51 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:boot 2024-03-24T06:36:51.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:51 smithi155 ceph-mon[36262]: fsmap 12 up:standby 2024-03-24T06:36:51.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:51 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:36:51.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:51 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:51.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:51 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:boot 2024-03-24T06:36:51.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:51 smithi022 ceph-mon[32003]: fsmap 12 up:standby 2024-03-24T06:36:51.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:51 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:36:51.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:51 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:51.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.942+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16d002d130 con 0x7f16d0101fe0 2024-03-24T06:36:51.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.943+0000 7f16c7400640 1 -- 172.21.15.22:0/307009385 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f16c00bcce0 con 0x7f16d0101fe0 2024-03-24T06:36:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:51.944 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24601,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-24T06:36:51.944 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 13 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 msgr2=0x7f16b40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 0x7f16b40a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f16d019ad00 tx=0x7f16bc005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 msgr2=0x7f16d01997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 0x7f16d01997f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f16c0000c00 tx=0x7f16c0004270 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 shutdown_connections 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f16b40a20c0 0x7f16b40a4580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16d0107240 0x7f16d01a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f16d0104910 0x7f16d0199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 --2- 172.21.15.22:0/307009385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f16d0101fe0 0x7f16d01997f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 >> 172.21.15.22:0/307009385 conn(0x7f16d00fb9f0 msgr2=0x7f16d0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:51.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 shutdown_connections 2024-03-24T06:36:51.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:51.946+0000 7f16d4a00640 1 -- 172.21.15.22:0/307009385 wait complete. 2024-03-24T06:36:51.966 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:36:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 -- 172.21.15.22:0/3150354518 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4104b40 msgr2=0x7fecc4104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 --2- 172.21.15.22:0/3150354518 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4104b40 0x7fecc4104fc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecb0009900 tx=0x7fecb002bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 -- 172.21.15.22:0/3150354518 shutdown_connections 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 --2- 172.21.15.22:0/3150354518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4105500 0x7fecc410bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 --2- 172.21.15.22:0/3150354518 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4104b40 0x7fecc4104fc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 --2- 172.21.15.22:0/3150354518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4103950 0x7fecc4103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 -- 172.21.15.22:0/3150354518 >> 172.21.15.22:0/3150354518 conn(0x7fecc40fef00 msgr2=0x7fecc4101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 -- 172.21.15.22:0/3150354518 shutdown_connections 2024-03-24T06:36:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.175+0000 7feccb200640 1 -- 172.21.15.22:0/3150354518 wait complete. 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 Processor -- start 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 -- start start 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 0x7fecc41999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4104b40 0x7fecc4199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 0x7fecc41a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fecc41a14e0 con 0x7fecc4103950 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fecc41a1650 con 0x7fecc4104b40 2024-03-24T06:36:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7feccb200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fecc41a1810 con 0x7fecc4105500 2024-03-24T06:36:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 0x7fecc41999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7fecc9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 0x7fecc41a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 0x7fecc41999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51650/0 (socket says 172.21.15.22:51650) 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7fecc3e00640 1 -- 172.21.15.22:0/575286765 learned_addr learned my addr 172.21.15.22:0/575286765 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.177+0000 7fecc3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4104b40 0x7fecc4199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 -- 172.21.15.22:0/575286765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4104b40 msgr2=0x7fecc4199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4104b40 0x7fecc4199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 -- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 msgr2=0x7fecc41999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 0x7fecc41999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 -- 172.21.15.22:0/575286765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc009840 con 0x7fecc4105500 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc9200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 0x7fecc41a0fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00ec50 tx=0x7fecbc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.178+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc003b70 con 0x7fecc4105500 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.179+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecbc00ce70 con 0x7fecc4105500 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.179+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb0009560 con 0x7fecc4105500 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.179+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc010830 con 0x7fecc4105500 2024-03-24T06:36:52.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.179+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc41a1df0 con 0x7fecc4105500 2024-03-24T06:36:52.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.180+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc410ac70 con 0x7fecc4105500 2024-03-24T06:36:52.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.182+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fecbc004020 con 0x7fecc4105500 2024-03-24T06:36:52.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.184+0000 7fecc0c00640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 0x7feca80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.184+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fecbc014070 con 0x7fecc4105500 2024-03-24T06:36:52.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.184+0000 7fecc3e00640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 0x7feca80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.185+0000 7fecc3e00640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 0x7feca80a4580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fecb40058c0 tx=0x7fecb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:52.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.188+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fecbc097380 con 0x7fecc4105500 2024-03-24T06:36:52.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.397+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecc402d160 con 0x7fecc4105500 2024-03-24T06:36:52.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.400+0000 7fecc0c00640 1 -- 172.21.15.22:0/575286765 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18465 (secure 0 0 0) 0x7fecbc09aa40 con 0x7fecc4105500 2024-03-24T06:36:52.400 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:52.401 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":81,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:35:58.557028+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 msgr2=0x7feca80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 0x7feca80a4580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fecb40058c0 tx=0x7fecb4005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 msgr2=0x7fecc41a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 0x7fecc41a0fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00ec50 tx=0x7fecbc00c6a0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 shutdown_connections 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feca80a20c0 0x7feca80a4580 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecc4105500 0x7fecc41a0fa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fecc4104b40 0x7fecc4199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 --2- 172.21.15.22:0/575286765 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecc4103950 0x7fecc41999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 >> 172.21.15.22:0/575286765 conn(0x7fecc40fef00 msgr2=0x7fecc4107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:52.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 shutdown_connections 2024-03-24T06:36:52.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.403+0000 7feccb200640 1 -- 172.21.15.22:0/575286765 wait complete. 2024-03-24T06:36:52.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T06:36:52.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:36:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 -- 172.21.15.22:0/1391350642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3881095d0 msgr2=0x7ff38810ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1391350642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3881095d0 0x7ff38810ba40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff380009be0 tx=0x7ff38002bef0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 -- 172.21.15.22:0/1391350642 shutdown_connections 2024-03-24T06:36:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1391350642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3881095d0 0x7ff38810ba40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1391350642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3880745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1391350642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff388075710 0x7ff388073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 -- 172.21.15.22:0/1391350642 >> 172.21.15.22:0/1391350642 conn(0x7ff3880fcc90 msgr2=0x7ff3880ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:52.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 -- 172.21.15.22:0/1391350642 shutdown_connections 2024-03-24T06:36:52.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.648+0000 7ff38fc00640 1 -- 172.21.15.22:0/1391350642 wait complete. 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.649+0000 7ff38fc00640 1 Processor -- start 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.649+0000 7ff38fc00640 1 -- start start 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3881999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 0x7ff388199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 0x7ff3881a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff3881a14b0 con 0x7ff388074130 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff3881a1620 con 0x7ff3881095d0 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38fc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff3881a17e0 con 0x7ff388075710 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3881999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3881999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51680/0 (socket says 172.21.15.22:51680) 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38d200640 1 -- 172.21.15.22:0/1767417242 learned_addr learned my addr 172.21.15.22:0/1767417242 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 0x7ff388199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38dc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 0x7ff3881a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 -- 172.21.15.22:0/1767417242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 msgr2=0x7ff3881a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 0x7ff3881a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 -- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 msgr2=0x7ff3881999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3881999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff387e00640 1 -- 172.21.15.22:0/1767417242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff380009840 con 0x7ff388075710 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.650+0000 7ff38dc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 0x7ff3881a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff387e00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 0x7ff388199f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e790 tx=0x7ff37400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400ccc0 con 0x7ff388075710 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37400eea0 con 0x7ff388075710 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3881a1ac0 con 0x7ff388075710 2024-03-24T06:36:52.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374018730 con 0x7ff388075710 2024-03-24T06:36:52.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.651+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3881a2020 con 0x7ff388075710 2024-03-24T06:36:52.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.652+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff388108350 con 0x7ff388075710 2024-03-24T06:36:52.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.654+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff374010430 con 0x7ff388075710 2024-03-24T06:36:52.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.655+0000 7ff385600640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 0x7ff3700a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:52.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.655+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff374014070 con 0x7ff388075710 2024-03-24T06:36:52.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.655+0000 7ff38d200640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 0x7ff3700a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:52.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.656+0000 7ff38d200640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 0x7ff3700a4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009a40 tx=0x7ff37c0091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:52.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.659+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff374053280 con 0x7ff388075710 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/307009385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/575286765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:52 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.868+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff38802d1d0 con 0x7ff388075710 2024-03-24T06:36:52.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.869+0000 7ff385600640 1 -- 172.21.15.22:0/1767417242 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7ff374096e30 con 0x7ff388075710 2024-03-24T06:36:52.868 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:36:52.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 msgr2=0x7ff3700a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 0x7ff3700a4600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009a40 tx=0x7ff37c0091c0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 msgr2=0x7ff388199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:52.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 0x7ff388199f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e790 tx=0x7ff37400ec50 comp rx=0 tx=0).stop 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 shutdown_connections 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff3700a2140 0x7ff3700a4600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3881095d0 0x7ff3881a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff388075710 0x7ff388199f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 --2- 172.21.15.22:0/1767417242 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff388074130 0x7ff3881999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 >> 172.21.15.22:0/1767417242 conn(0x7ff3880fcc90 msgr2=0x7ff388105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.871+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 shutdown_connections 2024-03-24T06:36:52.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:52.872+0000 7ff38fc00640 1 -- 172.21.15.22:0/1767417242 wait complete. 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/307009385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/575286765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.888 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:52 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.892 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/307009385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/575286765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:52.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:52 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:53.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 -- 172.21.15.22:0/930405775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c101730 msgr2=0x7f8f0c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 --2- 172.21.15.22:0/930405775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c101730 0x7f8f0c101b50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009900 tx=0x7f8f0002bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 -- 172.21.15.22:0/930405775 shutdown_connections 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 --2- 172.21.15.22:0/930405775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 --2- 172.21.15.22:0/930405775 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c102920 0x7f8f0c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 --2- 172.21.15.22:0/930405775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c101730 0x7f8f0c101b50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 -- 172.21.15.22:0/930405775 >> 172.21.15.22:0/930405775 conn(0x7f8f0c0fccc0 msgr2=0x7f8f0c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 -- 172.21.15.22:0/930405775 shutdown_connections 2024-03-24T06:36:53.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.098+0000 7f8f13600640 1 -- 172.21.15.22:0/930405775 wait complete. 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.099+0000 7f8f13600640 1 Processor -- start 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 -- start start 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 0x7f8f0c197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c19f2c0 con 0x7f8f0c101730 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c19f430 con 0x7f8f0c1032e0 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f13600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c19f5f0 con 0x7f8f0c102920 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f10c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f10c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51694/0 (socket says 172.21.15.22:51694) 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f10c00640 1 -- 172.21.15.22:0/3988655942 learned_addr learned my addr 172.21.15.22:0/3988655942 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f0be00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 0x7f8f0c197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.100+0000 7f8f11600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 -- 172.21.15.22:0/3988655942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 msgr2=0x7f8f0c197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 0x7f8f0c197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 -- 172.21.15.22:0/3988655942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 msgr2=0x7f8f0c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 -- 172.21.15.22:0/3988655942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f00009560 con 0x7f8f0c101730 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f0be00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 0x7f8f0c197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f11600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:53.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f10c00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005ab0 tx=0x7f8f00005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f000043d0 con 0x7f8f0c101730 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f000052d0 con 0x7f8f0c101730 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.101+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c19f870 con 0x7f8f0c101730 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.102+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00003d40 con 0x7f8f0c101730 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.102+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c19fdd0 con 0x7f8f0c101730 2024-03-24T06:36:53.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.103+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f0c108a50 con 0x7f8f0c101730 2024-03-24T06:36:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.105+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8f00005440 con 0x7f8f0c101730 2024-03-24T06:36:53.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.106+0000 7f8f09600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 0x7f8ef40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.106+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8f000ef940 con 0x7f8f0c101730 2024-03-24T06:36:53.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.107+0000 7f8f0be00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 0x7f8ef40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.107+0000 7f8f0be00640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 0x7f8ef40a4600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c198ce0 tx=0x7f8ef8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:53.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.112+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f000b8d10 con 0x7f8f0c101730 2024-03-24T06:36:53.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.343+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f8f0c02d160 con 0x7f8f0c101730 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.344+0000 7f8f09600640 1 -- 172.21.15.22:0/3988655942 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f8f000bc3d0 con 0x7f8f0c101730 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout:[ 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "poolnum": 1, 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "poolname": ".mgr" 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:36:53.344 INFO:teuthology.orchestra.run.smithi022.stdout:] 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.346+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 msgr2=0x7f8ef40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.346+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 0x7f8ef40a4600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c198ce0 tx=0x7f8ef8005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.346+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 msgr2=0x7f8f0c1977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.346+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005ab0 tx=0x7f8f00005e50 comp rx=0 tx=0).stop 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 shutdown_connections 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ef40a2140 0x7f8ef40a4600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0c1032e0 0x7f8f0c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0c102920 0x7f8f0c197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 --2- 172.21.15.22:0/3988655942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0c101730 0x7f8f0c1977d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 >> 172.21.15.22:0/3988655942 conn(0x7f8f0c0fccc0 msgr2=0x7f8f0c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:53.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 shutdown_connections 2024-03-24T06:36:53.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.347+0000 7f8f13600640 1 -- 172.21.15.22:0/3988655942 wait complete. 2024-03-24T06:36:53.368 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T06:36:53.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.568+0000 7f871aa00640 1 -- 172.21.15.22:0/345915852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 msgr2=0x7f8714109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.568+0000 7f871aa00640 1 --2- 172.21.15.22:0/345915852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f8714109b60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f870c009ca0 tx=0x7f870c02c100 comp rx=0 tx=0).stop 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 -- 172.21.15.22:0/345915852 shutdown_connections 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 --2- 172.21.15.22:0/345915852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f8714109b60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 --2- 172.21.15.22:0/345915852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 0x7f8714102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 --2- 172.21.15.22:0/345915852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 -- 172.21.15.22:0/345915852 >> 172.21.15.22:0/345915852 conn(0x7f87140fccb0 msgr2=0x7f87140ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 -- 172.21.15.22:0/345915852 shutdown_connections 2024-03-24T06:36:53.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.569+0000 7f871aa00640 1 -- 172.21.15.22:0/345915852 wait complete. 2024-03-24T06:36:53.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 Processor -- start 2024-03-24T06:36:53.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 -- start start 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 0x7f8714199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f87141a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f87141a1440 con 0x7f87141032d0 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f87141a15b0 con 0x7f8714101720 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f871aa00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f87141a1770 con 0x7f8714102910 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f8718a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f87141a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.570+0000 7f8713e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60962/0 (socket says 172.21.15.22:60962) 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8718a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f87141a0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51718/0 (socket says 172.21.15.22:51718) 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 -- 172.21.15.22:0/2248896658 learned_addr learned my addr 172.21.15.22:0/2248896658 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713400640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 0x7f8714199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 -- 172.21.15.22:0/2248896658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 msgr2=0x7f8714199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 0x7f8714199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 -- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 msgr2=0x7f87141a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f87141a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 -- 172.21.15.22:0/2248896658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c009900 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.571+0000 7f8713e00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f870400e9c0 tx=0x7f870400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.572+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704009700 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.572+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870400cd80 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.572+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704010430 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.572+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87141a1a50 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.572+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141a1fb0 con 0x7f8714101720 2024-03-24T06:36:53.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.573+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871410d100 con 0x7f8714101720 2024-03-24T06:36:53.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.575+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8704003680 con 0x7f8714101720 2024-03-24T06:36:53.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.576+0000 7f8710c00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 0x7f86f80a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:53.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.576+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8704014070 con 0x7f8714101720 2024-03-24T06:36:53.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.576+0000 7f8713400640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 0x7f86f80a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:53.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.577+0000 7f8713400640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 0x7f86f80a4600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f871419ae60 tx=0x7f8700005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:53.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.580+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8704096f80 con 0x7f8714101720 2024-03-24T06:36:53.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:53 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1767417242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:53.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:53 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3988655942' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T06:36:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:53 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1767417242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:53 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3988655942' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T06:36:53.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:53.801+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f871410d2c0 con 0x7f8714101720 2024-03-24T06:36:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:53 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1767417242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:36:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:53 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3988655942' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T06:36:54.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.700+0000 7f8710c00640 1 -- 172.21.15.22:0/2248896658 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f870409a640 con 0x7f8714101720 2024-03-24T06:36:54.701 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_metadata' created 2024-03-24T06:36:54.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.704+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 msgr2=0x7f86f80a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:54.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.705+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 0x7f86f80a4600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f871419ae60 tx=0x7f8700005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:54.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.705+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 msgr2=0x7f8714199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:54.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.705+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f870400e9c0 tx=0x7f870400ee80 comp rx=0 tx=0).stop 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 shutdown_connections 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86f80a2140 0x7f86f80a4600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87141032d0 0x7f87141a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8714102910 0x7f8714199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 --2- 172.21.15.22:0/2248896658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8714101720 0x7f8714199950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.706+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 >> 172.21.15.22:0/2248896658 conn(0x7f87140fccb0 msgr2=0x7f8714073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:54.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.707+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 shutdown_connections 2024-03-24T06:36:54.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.709+0000 7f871aa00640 1 -- 172.21.15.22:0/2248896658 wait complete. 2024-03-24T06:36:54.731 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:54.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:54.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2248896658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:54.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:54.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:54 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 -- 172.21.15.22:0/4177401170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d601095d0 msgr2=0x7f5d6010ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4177401170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d601095d0 0x7f5d6010ba60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009be0 tx=0x7f5d5802bf80 comp rx=0 tx=0).stop 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 -- 172.21.15.22:0/4177401170 shutdown_connections 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4177401170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d601095d0 0x7f5d6010ba60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4177401170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d600740f0 0x7f5d60074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4177401170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 0x7f5d60073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 -- 172.21.15.22:0/4177401170 >> 172.21.15.22:0/4177401170 conn(0x7f5d600fcc90 msgr2=0x7f5d600ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 -- 172.21.15.22:0/4177401170 shutdown_connections 2024-03-24T06:36:54.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.940+0000 7f5d64e00640 1 -- 172.21.15.22:0/4177401170 wait complete. 2024-03-24T06:36:54.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.941+0000 7f5d64e00640 1 Processor -- start 2024-03-24T06:36:54.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.941+0000 7f5d64e00640 1 -- start start 2024-03-24T06:36:54.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 0x7f5d601999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:54.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 0x7f5d60199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 0x7f5d601a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601a1490 con 0x7f5d600756d0 2024-03-24T06:36:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601a1600 con 0x7f5d601095d0 2024-03-24T06:36:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d64e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601a17c0 con 0x7f5d600740f0 2024-03-24T06:36:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 0x7f5d601999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 0x7f5d601a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 0x7f5d601999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58206/0 (socket says 172.21.15.22:58206) 2024-03-24T06:36:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5de00640 1 -- 172.21.15.22:0/4237709990 learned_addr learned my addr 172.21.15.22:0/4237709990 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5d400640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 0x7f5d60199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 -- 172.21.15.22:0/4237709990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 msgr2=0x7f5d601999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 0x7f5d601999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 -- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 msgr2=0x7f5d60199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 0x7f5d60199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.942+0000 7f5d5e800640 1 -- 172.21.15.22:0/4237709990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d58009840 con 0x7f5d601095d0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d5e800640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 0x7f5d601a0f50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5d5802c4c0 tx=0x7f5d58033c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58033e80 con 0x7f5d601095d0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d601a1a40 con 0x7f5d601095d0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d58003b80 con 0x7f5d601095d0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58034620 con 0x7f5d601095d0 2024-03-24T06:36:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.943+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d601a1f50 con 0x7f5d601095d0 2024-03-24T06:36:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.944+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d60108390 con 0x7f5d601095d0 2024-03-24T06:36:54.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.946+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5d58003cf0 con 0x7f5d601095d0 2024-03-24T06:36:54.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.948+0000 7f5d56a00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 0x7f5d440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.948+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f5d580f0650 con 0x7f5d601095d0 2024-03-24T06:36:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.948+0000 7f5d5de00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 0x7f5d440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.949+0000 7f5d5de00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 0x7f5d440a4580 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009c90 tx=0x7f5d50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:54.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:54.954+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5d580b98b0 con 0x7f5d601095d0 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2248896658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:55.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:55.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:54 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:55.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:55.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2248896658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:55.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T06:36:55.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:54 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:36:55.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.161+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f5d6002d1d0 con 0x7f5d601095d0 2024-03-24T06:36:55.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.714+0000 7f5d56a00640 1 -- 172.21.15.22:0/4237709990 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f5d580bcf70 con 0x7f5d601095d0 2024-03-24T06:36:55.715 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_data' created 2024-03-24T06:36:55.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.720+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 msgr2=0x7f5d440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:55.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.720+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 0x7f5d440a4580 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009c90 tx=0x7f5d50005f90 comp rx=0 tx=0).stop 2024-03-24T06:36:55.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.720+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 msgr2=0x7f5d601a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:55.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.720+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 0x7f5d601a0f50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5d5802c4c0 tx=0x7f5d58033c90 comp rx=0 tx=0).stop 2024-03-24T06:36:55.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 shutdown_connections 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5d440a20c0 0x7f5d440a4580 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d601095d0 0x7f5d601a0f50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d600756d0 0x7f5d60199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 --2- 172.21.15.22:0/4237709990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d600740f0 0x7f5d601999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.722+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 >> 172.21.15.22:0/4237709990 conn(0x7f5d600fcc90 msgr2=0x7f5d60105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.723+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 shutdown_connections 2024-03-24T06:36:55.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.723+0000 7f5d64e00640 1 -- 172.21.15.22:0/4237709990 wait complete. 2024-03-24T06:36:55.754 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T06:36:55.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.951+0000 7f78a7200640 1 -- 172.21.15.22:0/2601159021 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 msgr2=0x7f78a0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.951+0000 7f78a7200640 1 --2- 172.21.15.22:0/2601159021 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0102d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7888009ca0 tx=0x7f788802c0f0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 -- 172.21.15.22:0/2601159021 shutdown_connections 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 --2- 172.21.15.22:0/2601159021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a01032d0 0x7f78a0109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 --2- 172.21.15.22:0/2601159021 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0102d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 --2- 172.21.15.22:0/2601159021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a0101720 0x7f78a0101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 -- 172.21.15.22:0/2601159021 >> 172.21.15.22:0/2601159021 conn(0x7f78a00fcc90 msgr2=0x7f78a00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 -- 172.21.15.22:0/2601159021 shutdown_connections 2024-03-24T06:36:55.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.952+0000 7f78a7200640 1 -- 172.21.15.22:0/2601159021 wait complete. 2024-03-24T06:36:55.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 Processor -- start 2024-03-24T06:36:55.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 -- start start 2024-03-24T06:36:55.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 0x7f78a0074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f78a00747a0 con 0x7f78a0102910 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78a0074910 con 0x7f78a0101720 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78a01a18a0 con 0x7f78a01032d0 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f789fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 0x7f78a0074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f789f400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f789fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:55316/0 (socket says 172.21.15.22:55316) 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f78a5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 0x7f78a0074040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58214/0 (socket says 172.21.15.22:58214) 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.953+0000 7f789fe00640 1 -- 172.21.15.22:0/351407496 learned_addr learned my addr 172.21.15.22:0/351407496 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 -- 172.21.15.22:0/351407496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 msgr2=0x7f78a0074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 0x7f78a0074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 -- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 msgr2=0x7f78a0073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:55.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 -- 172.21.15.22:0/351407496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7888009900 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789f400640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789fe00640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f789000e9f0 tx=0x7f789000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.954+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900096e0 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.955+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789000cd80 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.955+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890010430 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.955+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a01a1b50 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.955+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a01a1fe0 con 0x7f78a0101720 2024-03-24T06:36:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.956+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a0108b10 con 0x7f78a0101720 2024-03-24T06:36:55.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.958+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7890003680 con 0x7f78a0101720 2024-03-24T06:36:55.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.959+0000 7f789cc00640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 0x7f78800a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:55.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.959+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f7890014070 con 0x7f78a0101720 2024-03-24T06:36:55.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.960+0000 7f789f400640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 0x7f78800a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:55.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.960+0000 7f789f400640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 0x7f78800a4600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7888006010 tx=0x7f78880326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:55.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:55.963+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7890096f50 con 0x7f78a0101720 2024-03-24T06:36:56.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:55 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T06:36:56.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:55 smithi155 ceph-mon[36262]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T06:36:56.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:55 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T06:36:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:55 smithi022 ceph-mon[32003]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T06:36:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:55 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T06:36:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:55 smithi143 ceph-mon[36520]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T06:36:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:56.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:56.213+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f78a010d3e0 con 0x7f78a0101720 2024-03-24T06:36:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: pgmap v312: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4237709990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/351407496' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:56 smithi155 ceph-mon[36262]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: pgmap v312: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4237709990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T06:36:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/351407496' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:56 smithi022 ceph-mon[32003]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-24T06:36:57.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.163+0000 7f789cc00640 1 -- 172.21.15.22:0/351407496 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f789009a610 con 0x7f78a0101720 2024-03-24T06:36:57.163 INFO:teuthology.orchestra.run.smithi022.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T06:36:57.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 msgr2=0x7f78800a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 0x7f78800a4600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7888006010 tx=0x7f78880326b0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 msgr2=0x7f78a00754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f789000e9f0 tx=0x7f789000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 shutdown_connections 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f78800a2140 0x7f78800a4600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78a01032d0 0x7f78a0074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f78a0102910 0x7f78a0073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 --2- 172.21.15.22:0/351407496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a0101720 0x7f78a00754c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 >> 172.21.15.22:0/351407496 conn(0x7f78a00fcc90 msgr2=0x7f78a0105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 shutdown_connections 2024-03-24T06:36:57.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.169+0000 7f78a7200640 1 -- 172.21.15.22:0/351407496 wait complete. 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: pgmap v312: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4237709990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/351407496' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T06:36:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:56 smithi143 ceph-mon[36520]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-03-24T06:36:57.197 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-24T06:36:57.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-24T06:36:57.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 -- 172.21.15.22:0/3582008746 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381049a0 msgr2=0x7ff238106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 --2- 172.21.15.22:0/3582008746 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381049a0 0x7ff238106d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff224009900 tx=0x7ff22402bc40 comp rx=0 tx=0).stop 2024-03-24T06:36:57.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 -- 172.21.15.22:0/3582008746 shutdown_connections 2024-03-24T06:36:57.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 --2- 172.21.15.22:0/3582008746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff2381072d0 0x7ff238109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 --2- 172.21.15.22:0/3582008746 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381049a0 0x7ff238106d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 --2- 172.21.15.22:0/3582008746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff238102070 0x7ff238104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 -- 172.21.15.22:0/3582008746 >> 172.21.15.22:0/3582008746 conn(0x7ff2380fba80 msgr2=0x7ff2380fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:57.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 -- 172.21.15.22:0/3582008746 shutdown_connections 2024-03-24T06:36:57.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.390+0000 7ff237200640 1 -- 172.21.15.22:0/3582008746 wait complete. 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.391+0000 7ff237200640 1 Processor -- start 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 -- start start 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 0x7ff238199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 0x7ff2381a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff2381a1370 con 0x7ff2381072d0 2024-03-24T06:36:57.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff2381a14e0 con 0x7ff2381049a0 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff237200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff2381a16a0 con 0x7ff238102070 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff235e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff236800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 0x7ff2381a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff235e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58230/0 (socket says 172.21.15.22:58230) 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff235e00640 1 -- 172.21.15.22:0/1153352877 learned_addr learned my addr 172.21.15.22:0/1153352877 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:57.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.392+0000 7ff235400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 0x7ff238199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 -- 172.21.15.22:0/1153352877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 msgr2=0x7ff238199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 0x7ff238199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 -- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 msgr2=0x7ff2381a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 0x7ff2381a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 -- 172.21.15.22:0/1153352877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff224009560 con 0x7ff238102070 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235400640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 0x7ff238199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff236800640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 0x7ff2381a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:57.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff235e00640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c920 tx=0x7ff22800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.393+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228007c70 con 0x7ff238102070 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.394+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2381a1980 con 0x7ff238102070 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.394+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff228007dd0 con 0x7ff238102070 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.394+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228018860 con 0x7ff238102070 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.394+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2381a1eb0 con 0x7ff238102070 2024-03-24T06:36:57.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.395+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff238108e20 con 0x7ff238102070 2024-03-24T06:36:57.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.397+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff22800fac0 con 0x7ff238102070 2024-03-24T06:36:57.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.398+0000 7ff22ea00640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 0x7ff21c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:57.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.398+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7ff2280cdd90 con 0x7ff238102070 2024-03-24T06:36:57.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.398+0000 7ff235400640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 0x7ff21c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:57.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.399+0000 7ff235400640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 0x7ff21c0a4600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff224005ab0 tx=0x7ff224005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:57.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.401+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff228096e90 con 0x7ff238102070 2024-03-24T06:36:57.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.610+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ff23810d240 con 0x7ff238102070 2024-03-24T06:36:57.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.750+0000 7ff22ea00640 1 -- 172.21.15.22:0/1153352877 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7ff22809a550 con 0x7ff238102070 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 msgr2=0x7ff21c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 0x7ff21c0a4600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff224005ab0 tx=0x7ff224005fd0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 msgr2=0x7ff238199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c920 tx=0x7ff22800cde0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 shutdown_connections 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff21c0a2140 0x7ff21c0a4600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2381072d0 0x7ff2381a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2381049a0 0x7ff238199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 --2- 172.21.15.22:0/1153352877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff238102070 0x7ff238199880 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:57.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 >> 172.21.15.22:0/1153352877 conn(0x7ff2380fba80 msgr2=0x7ff238101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:57.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 shutdown_connections 2024-03-24T06:36:57.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:57.754+0000 7ff237200640 1 -- 172.21.15.22:0/1153352877 wait complete. 2024-03-24T06:36:57.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T06:36:58.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 -- 172.21.15.22:0/882455945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b4101130 msgr2=0x7fd1b41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:58.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 --2- 172.21.15.22:0/882455945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b4101130 0x7fd1b41015b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009900 tx=0x7fd1ac02bc30 comp rx=0 tx=0).stop 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 -- 172.21.15.22:0/882455945 shutdown_connections 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 --2- 172.21.15.22:0/882455945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b4101130 0x7fd1b41015b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 --2- 172.21.15.22:0/882455945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4100790 0x7fd1b4100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 --2- 172.21.15.22:0/882455945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b40ffe30 0x7fd1b4100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 -- 172.21.15.22:0/882455945 >> 172.21.15.22:0/882455945 conn(0x7fd1b40fb9f0 msgr2=0x7fd1b40fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 -- 172.21.15.22:0/882455945 shutdown_connections 2024-03-24T06:36:58.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.002+0000 7fd1b3200640 1 -- 172.21.15.22:0/882455945 wait complete. 2024-03-24T06:36:58.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.003+0000 7fd1b3200640 1 Processor -- start 2024-03-24T06:36:58.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 -- start start 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 0x7fd1b4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4101130 0x7fd1b41a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41a1250 con 0x7fd1b40ffe30 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41a13c0 con 0x7fd1b4101130 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41a1580 con 0x7fd1b4100790 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42700/0 (socket says 172.21.15.22:42700) 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b1e00640 1 -- 172.21.15.22:0/2385887816 learned_addr learned my addr 172.21.15.22:0/2385887816 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.004+0000 7fd1b1400640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 0x7fd1b4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b2800640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4101130 0x7fd1b41a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 -- 172.21.15.22:0/2385887816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4101130 msgr2=0x7fd1b41a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:58.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4101130 0x7fd1b41a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 -- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 msgr2=0x7fd1b41997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 -- 172.21.15.22:0/2385887816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c009490 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1e00640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b1400640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 0x7fd1b4199cf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e9f0 tx=0x7fd19c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00cd80 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac009560 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.005+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd19c0035f0 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.006+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c010690 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.006+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b41a1c30 con 0x7fd1b4100790 2024-03-24T06:36:58.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.007+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b410ab80 con 0x7fd1b4100790 2024-03-24T06:36:58.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.009+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd19c003af0 con 0x7fd1b4100790 2024-03-24T06:36:58.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.010+0000 7fd1aaa00640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 0x7fd1980a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:36:58.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.010+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fd19c014070 con 0x7fd1b4100790 2024-03-24T06:36:58.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.011+0000 7fd1b1e00640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 0x7fd1980a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:36:58.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.011+0000 7fd1b1e00640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 0x7fd1980a4580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40049d0 tx=0x7fd1a4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:36:58.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.012+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd19c097060 con 0x7fd1b4100790 2024-03-24T06:36:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T06:36:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-24T06:36:58.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1153352877' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:57 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-24T06:36:58.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1153352877' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:57 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T06:36:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1153352877' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:57 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T06:36:58.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:36:58.223+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd1b410f250 con 0x7fd1b4100790 2024-03-24T06:36:58.649 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: 2024-03-24T06:36:58.180+0000 7f516d800700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: pgmap v315: 97 pgs: 28 creating+peering, 2 active+clean, 67 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T06:36:59.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:active 2024-03-24T06:36:59.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2385887816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:36:59.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:36:58 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: pgmap v315: 97 pgs: 28 creating+peering, 2 active+clean, 67 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:active 2024-03-24T06:36:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2385887816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:36:59.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:36:58 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: pgmap v315: 97 pgs: 28 creating+peering, 2 active+clean, 67 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] up:active 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:36:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2385887816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:36:59.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:36:58 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T06:37:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.211+0000 7fd1aaa00640 1 -- 172.21.15.22:0/2385887816 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fd19c09a720 con 0x7fd1b4100790 2024-03-24T06:37:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_data_ec' created 2024-03-24T06:37:00.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.218+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 msgr2=0x7fd1980a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:00.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.218+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 0x7fd1980a4580 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40049d0 tx=0x7fd1a4009500 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.218+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 msgr2=0x7fd1b4199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.218+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 0x7fd1b4199cf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e9f0 tx=0x7fd19c00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 shutdown_connections 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd1980a20c0 0x7fd1980a4580 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b4101130 0x7fd1b41a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd1b4100790 0x7fd1b4199cf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 --2- 172.21.15.22:0/2385887816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1b40ffe30 0x7fd1b41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.219+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 >> 172.21.15.22:0/2385887816 conn(0x7fd1b40fb9f0 msgr2=0x7fd1b40fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:00.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.220+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 shutdown_connections 2024-03-24T06:37:00.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.220+0000 7fd1b3200640 1 -- 172.21.15.22:0/2385887816 wait complete. 2024-03-24T06:37:00.247 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 -- 172.21.15.22:0/281239792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd2900ff690 msgr2=0x7fd2900ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 --2- 172.21.15.22:0/281239792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd2900ff690 0x7fd2900ffab0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd280009900 tx=0x7fd28002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 -- 172.21.15.22:0/281239792 shutdown_connections 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 --2- 172.21.15.22:0/281239792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2901010a0 0x7fd290101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 --2- 172.21.15.22:0/281239792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd290100700 0x7fd290100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 --2- 172.21.15.22:0/281239792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd2900ff690 0x7fd2900ffab0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 -- 172.21.15.22:0/281239792 >> 172.21.15.22:0/281239792 conn(0x7fd2900faed0 msgr2=0x7fd2900fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 -- 172.21.15.22:0/281239792 shutdown_connections 2024-03-24T06:37:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.447+0000 7fd296200640 1 -- 172.21.15.22:0/281239792 wait complete. 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 Processor -- start 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 -- start start 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290100700 0x7fd290199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2901010a0 0x7fd2901a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901a14a0 con 0x7fd290100700 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901a1610 con 0x7fd2901010a0 2024-03-24T06:37:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.448+0000 7fd296200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901a17d0 con 0x7fd2900ff690 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290100700 0x7fd290199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58258/0 (socket says 172.21.15.22:58258) 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 -- 172.21.15.22:0/197935519 learned_addr learned my addr 172.21.15.22:0/197935519 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28fe00640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2901010a0 0x7fd2901a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 -- 172.21.15.22:0/197935519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2901010a0 msgr2=0x7fd2901a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2901010a0 0x7fd2901a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 -- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290100700 msgr2=0x7fd290199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290100700 0x7fd290199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 -- 172.21.15.22:0/197935519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c009490 con 0x7fd2900ff690 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.449+0000 7fd28f400640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd28002c6a0 tx=0x7fd2800040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.450+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2800044c0 con 0x7fd2900ff690 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.450+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd280009560 con 0x7fd2900ff690 2024-03-24T06:37:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.450+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2800052d0 con 0x7fd2900ff690 2024-03-24T06:37:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.450+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28003e450 con 0x7fd2900ff690 2024-03-24T06:37:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.450+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2901a1dd0 con 0x7fd2900ff690 2024-03-24T06:37:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.451+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd29010bb50 con 0x7fd2900ff690 2024-03-24T06:37:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.453+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd280005440 con 0x7fd2900ff690 2024-03-24T06:37:00.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.454+0000 7fd287e00640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 0x7fd2740a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:00.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.454+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fd2800f0820 con 0x7fd2900ff690 2024-03-24T06:37:00.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.454+0000 7fd28ea00640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 0x7fd2740a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:00.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.455+0000 7fd28ea00640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 0x7fd2740a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd29019aec0 tx=0x7fd27c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.458+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd2800b9570 con 0x7fd2900ff690 2024-03-24T06:37:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:00 smithi155 ceph-mon[36262]: pgmap v317: 97 pgs: 28 creating+peering, 51 active+clean, 18 unknown; 578 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T06:37:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:00 smithi155 ceph-mon[36262]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T06:37:00.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:00 smithi022 ceph-mon[32003]: pgmap v317: 97 pgs: 28 creating+peering, 51 active+clean, 18 unknown; 578 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T06:37:00.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:00 smithi022 ceph-mon[32003]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T06:37:00.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:00.658+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fd29002d1d0 con 0x7fd2900ff690 2024-03-24T06:37:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:00 smithi143 ceph-mon[36520]: pgmap v317: 97 pgs: 28 creating+peering, 51 active+clean, 18 unknown; 578 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-24T06:37:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:00 smithi143 ceph-mon[36520]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T06:37:01.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.224+0000 7fd287e00640 1 -- 172.21.15.22:0/197935519 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fd2800430c0 con 0x7fd2900ff690 2024-03-24T06:37:01.224 INFO:teuthology.orchestra.run.smithi022.stderr:set pool 4 allow_ec_overwrites to true 2024-03-24T06:37:01.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.229+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 msgr2=0x7fd2740a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:01.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.229+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 0x7fd2740a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd29019aec0 tx=0x7fd27c005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:01.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.229+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 msgr2=0x7fd2901999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:01.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.229+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd28002c6a0 tx=0x7fd2800040c0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 shutdown_connections 2024-03-24T06:37:01.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd2740a20c0 0x7fd2740a4580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2901010a0 0x7fd2901a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290100700 0x7fd290199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 --2- 172.21.15.22:0/197935519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2900ff690 0x7fd2901999b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.230+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 >> 172.21.15.22:0/197935519 conn(0x7fd2900faed0 msgr2=0x7fd2900fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:01.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.232+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 shutdown_connections 2024-03-24T06:37:01.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.233+0000 7fd296200640 1 -- 172.21.15.22:0/197935519 wait complete. 2024-03-24T06:37:01.270 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-24T06:37:01.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 -- 172.21.15.22:0/2585178302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 msgr2=0x7fb720102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:01.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 --2- 172.21.15.22:0/2585178302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720102d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009900 tx=0x7fb70c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:01.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 -- 172.21.15.22:0/2585178302 shutdown_connections 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 --2- 172.21.15.22:0/2585178302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb7201032d0 0x7fb720109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 --2- 172.21.15.22:0/2585178302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720102d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 --2- 172.21.15.22:0/2585178302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb720101720 0x7fb720101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 -- 172.21.15.22:0/2585178302 >> 172.21.15.22:0/2585178302 conn(0x7fb7200fccb0 msgr2=0x7fb7200ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.462+0000 7fb727e00640 1 -- 172.21.15.22:0/2585178302 shutdown_connections 2024-03-24T06:37:01.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.463+0000 7fb727e00640 1 -- 172.21.15.22:0/2585178302 wait complete. 2024-03-24T06:37:01.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 Processor -- start 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 -- start start 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7201032d0 0x7fb7201a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb7201a1500 con 0x7fb720102910 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb7201a1670 con 0x7fb7201032d0 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb727e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb7201a1830 con 0x7fb720101720 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb724a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb725400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb725400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58268/0 (socket says 172.21.15.22:58268) 2024-03-24T06:37:01.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb725400640 1 -- 172.21.15.22:0/3703622803 learned_addr learned my addr 172.21.15.22:0/3703622803 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.464+0000 7fb725e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7201032d0 0x7fb7201a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 -- 172.21.15.22:0/3703622803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7201032d0 msgr2=0x7fb7201a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7201032d0 0x7fb7201a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 -- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 msgr2=0x7fb720199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 -- 172.21.15.22:0/3703622803 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c009560 con 0x7fb720101720 2024-03-24T06:37:01.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb725400640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb71400e9c0 tx=0x7fb71400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.465+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714009700 con 0x7fb720101720 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.466+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7201a1b10 con 0x7fb720101720 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.466+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb71400cd80 con 0x7fb720101720 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.466+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714010430 con 0x7fb720101720 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.466+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7201a2040 con 0x7fb720101720 2024-03-24T06:37:01.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.467+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb72010d130 con 0x7fb720101720 2024-03-24T06:37:01.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.469+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb714003680 con 0x7fb720101720 2024-03-24T06:37:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.470+0000 7fb71e000640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 0x7fb7080a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.470+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fb714014070 con 0x7fb720101720 2024-03-24T06:37:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.470+0000 7fb724a00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 0x7fb7080a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:01.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.471+0000 7fb724a00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 0x7fb7080a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009230 tx=0x7fb70c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.472+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb714097000 con 0x7fb720101720 2024-03-24T06:37:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:01 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T06:37:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:01 smithi155 ceph-mon[36262]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T06:37:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:01 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/197935519' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:01 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:01 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T06:37:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:01 smithi022 ceph-mon[32003]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T06:37:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:01 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/197935519' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:01 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:01 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T06:37:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:01 smithi143 ceph-mon[36520]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T06:37:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:01 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/197935519' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:01 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T06:37:01.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:01.826+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fb7201a21d0 con 0x7fb720101720 2024-03-24T06:37:02.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.226+0000 7fb71e000640 1 -- 172.21.15.22:0/3703622803 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fb71409a6c0 con 0x7fb720101720 2024-03-24T06:37:02.226 INFO:teuthology.orchestra.run.smithi022.stderr:added data pool 4 to fsmap 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 msgr2=0x7fb7080a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 0x7fb7080a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009230 tx=0x7fb70c005fd0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 msgr2=0x7fb720199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb71400e9c0 tx=0x7fb71400ee80 comp rx=0 tx=0).stop 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 shutdown_connections 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb7080a20c0 0x7fb7080a4580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7201032d0 0x7fb7201a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb720102910 0x7fb720199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 --2- 172.21.15.22:0/3703622803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb720101720 0x7fb720199a10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 >> 172.21.15.22:0/3703622803 conn(0x7fb7200fccb0 msgr2=0x7fb720073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:02.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 shutdown_connections 2024-03-24T06:37:02.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.232+0000 7fb727e00640 1 -- 172.21.15.22:0/3703622803 wait complete. 2024-03-24T06:37:02.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 -- 172.21.15.22:0/2180842464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 msgr2=0x7ff9b8100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2180842464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8100bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009900 tx=0x7ff9a802bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 -- 172.21.15.22:0/2180842464 shutdown_connections 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2180842464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 0x7ff9b81015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2180842464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8100bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2180842464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b8100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 -- 172.21.15.22:0/2180842464 >> 172.21.15.22:0/2180842464 conn(0x7ff9b80fb9f0 msgr2=0x7ff9b80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 -- 172.21.15.22:0/2180842464 shutdown_connections 2024-03-24T06:37:02.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.474+0000 7ff9bf200640 1 -- 172.21.15.22:0/2180842464 wait complete. 2024-03-24T06:37:02.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.475+0000 7ff9bf200640 1 Processor -- start 2024-03-24T06:37:02.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 -- start start 2024-03-24T06:37:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b81997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 0x7ff9b81a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b81a12b0 con 0x7ff9b8100790 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b81a1420 con 0x7ff9b80ffe30 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b81a15e0 con 0x7ff9b8101130 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bd400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bd400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42744/0 (socket says 172.21.15.22:42744) 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bd400640 1 -- 172.21.15.22:0/2434598388 learned_addr learned my addr 172.21.15.22:0/2434598388 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.476+0000 7ff9bde00640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b81997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9be800640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 0x7ff9b81a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 -- 172.21.15.22:0/2434598388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 msgr2=0x7ff9b81a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 0x7ff9b81a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 -- 172.21.15.22:0/2434598388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 msgr2=0x7ff9b81997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b81997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 -- 172.21.15.22:0/2434598388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ac009490 con 0x7ff9b8100790 2024-03-24T06:37:02.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bde00640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b81997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:02.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bd400640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9a802c6a0 tx=0x7ff9a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:02.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80044e0 con 0x7ff9b8100790 2024-03-24T06:37:02.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a8004ac0 con 0x7ff9b8100790 2024-03-24T06:37:02.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a8009560 con 0x7ff9b8100790 2024-03-24T06:37:02.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.477+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a803e450 con 0x7ff9b8100790 2024-03-24T06:37:02.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.478+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b81a1c30 con 0x7ff9b8100790 2024-03-24T06:37:02.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.479+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b8068420 con 0x7ff9b8100790 2024-03-24T06:37:02.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.481+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff9a80035f0 con 0x7ff9b8100790 2024-03-24T06:37:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.482+0000 7ff9b2a00640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 0x7ff9a00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.482+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff9a80f0820 con 0x7ff9b8100790 2024-03-24T06:37:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.482+0000 7ff9bde00640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 0x7ff9a00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.483+0000 7ff9bde00640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 0x7ff9a00a4600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac011630 tx=0x7ff9ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:02.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.487+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff9a80b9550 con 0x7ff9b8100790 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: pgmap v320: 129 pgs: 14 creating+peering, 91 active+clean, 24 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3703622803' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:02 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: pgmap v320: 129 pgs: 14 creating+peering, 91 active+clean, 24 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3703622803' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:02 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: pgmap v320: 129 pgs: 14 creating+peering, 91 active+clean, 24 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 10 op/s 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3703622803' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:02 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T06:37:02.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.732+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9b810ab80 con 0x7ff9b8100790 2024-03-24T06:37:02.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.734+0000 7ff9b2a00640 1 -- 172.21.15.22:0/2434598388 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7ff9b810ab80 con 0x7ff9b8100790 2024-03-24T06:37:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:02.734 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:01.829667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24601,"qdb_cluster":[24601]},"id":1}]} 2024-03-24T06:37:02.734 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 17 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.737+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 msgr2=0x7ff9a00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.737+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 0x7ff9a00a4600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac011630 tx=0x7ff9ac005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.737+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 msgr2=0x7ff9b8199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.737+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff9a802c6a0 tx=0x7ff9a80042f0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 shutdown_connections 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff9a00a2140 0x7ff9a00a4600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9b8101130 0x7ff9b81a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff9b8100790 0x7ff9b8199d00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 --2- 172.21.15.22:0/2434598388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b80ffe30 0x7ff9b81997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 >> 172.21.15.22:0/2434598388 conn(0x7ff9b80fb9f0 msgr2=0x7ff9b80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 shutdown_connections 2024-03-24T06:37:02.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.738+0000 7ff9bf200640 1 -- 172.21.15.22:0/2434598388 wait complete. 2024-03-24T06:37:02.759 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 -- 172.21.15.22:0/4075392154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc103760 msgr2=0x7f74cc103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 --2- 172.21.15.22:0/4075392154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc103760 0x7f74cc103b80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009900 tx=0x7f74c002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 -- 172.21.15.22:0/4075392154 shutdown_connections 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 --2- 172.21.15.22:0/4075392154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc105310 0x7f74cc10bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 --2- 172.21.15.22:0/4075392154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc104950 0x7f74cc104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 --2- 172.21.15.22:0/4075392154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc103760 0x7f74cc103b80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 -- 172.21.15.22:0/4075392154 >> 172.21.15.22:0/4075392154 conn(0x7f74cc0fecf0 msgr2=0x7f74cc101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:02.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 -- 172.21.15.22:0/4075392154 shutdown_connections 2024-03-24T06:37:02.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.983+0000 7f74d2e00640 1 -- 172.21.15.22:0/4075392154 wait complete. 2024-03-24T06:37:02.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 Processor -- start 2024-03-24T06:37:02.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 -- start start 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc104950 0x7f74cc199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 0x7f74cc1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1a12b0 con 0x7f74cc103760 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1a1420 con 0x7f74cc105310 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.985+0000 7f74d2e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc1a15e0 con 0x7f74cc104950 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42760/0 (socket says 172.21.15.22:42760) 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 -- 172.21.15.22:0/655075325 learned_addr learned my addr 172.21.15.22:0/655075325 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1000640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc104950 0x7f74cc199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d2400640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 0x7f74cc1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 -- 172.21.15.22:0/655075325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc104950 msgr2=0x7f74cc199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc104950 0x7f74cc199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 -- 172.21.15.22:0/655075325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 msgr2=0x7f74cc1a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 0x7f74cc1a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 -- 172.21.15.22:0/655075325 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b8009490 con 0x7f74cc103760 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d2400640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 0x7f74cc1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:02.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.986+0000 7f74d1a00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f74c0000c00 tx=0x7f74c0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.987+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c00043d0 con 0x7f74cc103760 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.987+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c0009560 con 0x7f74cc103760 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.987+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c00052d0 con 0x7f74cc103760 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.987+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0003da0 con 0x7f74cc103760 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.987+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc1a1be0 con 0x7f74cc103760 2024-03-24T06:37:02.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.988+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74cc068870 con 0x7f74cc103760 2024-03-24T06:37:02.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.990+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f74c0005440 con 0x7f74cc103760 2024-03-24T06:37:02.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.992+0000 7f74ca000640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 0x7f74b40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:02.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.992+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f74c00f0b50 con 0x7f74cc103760 2024-03-24T06:37:02.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.992+0000 7f74d1000640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 0x7f74b40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:02.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.993+0000 7f74d1000640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 0x7f74b40a4580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74cc19acd0 tx=0x7f74b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:02.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:02.995+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f74c00b9880 con 0x7f74cc103760 2024-03-24T06:37:03.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.343+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74cc10ab00 con 0x7f74cc103760 2024-03-24T06:37:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.348+0000 7f74ca000640 1 -- 172.21.15.22:0/655075325 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f74c00bcf40 con 0x7f74cc103760 2024-03-24T06:37:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:03.348 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:03.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.351+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 msgr2=0x7f74b40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.351+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 0x7f74b40a4580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74cc19acd0 tx=0x7f74b8005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.351+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 msgr2=0x7f74cc1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.351+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f74c0000c00 tx=0x7f74c0005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 shutdown_connections 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f74b40a20c0 0x7f74b40a4580 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74cc105310 0x7f74cc1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74cc104950 0x7f74cc199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 --2- 172.21.15.22:0/655075325 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74cc103760 0x7f74cc1997c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 >> 172.21.15.22:0/655075325 conn(0x7f74cc0fecf0 msgr2=0x7f74cc107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 shutdown_connections 2024-03-24T06:37:03.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.352+0000 7f74d2e00640 1 -- 172.21.15.22:0/655075325 wait complete. 2024-03-24T06:37:03.379 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:03 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T06:37:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:03 smithi155 ceph-mon[36262]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T06:37:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:03 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:03 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2434598388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.596+0000 7f7db5200640 1 -- 172.21.15.22:0/1800047729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0101570 msgr2=0x7f7db0101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.596+0000 7f7db5200640 1 --2- 172.21.15.22:0/1800047729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0101570 0x7f7db0101990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009860 tx=0x7f7da002bb70 comp rx=0 tx=0).stop 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 -- 172.21.15.22:0/1800047729 shutdown_connections 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 --2- 172.21.15.22:0/1800047729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0103120 0x7f7db01099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 --2- 172.21.15.22:0/1800047729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 --2- 172.21.15.22:0/1800047729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0101570 0x7f7db0101990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 -- 172.21.15.22:0/1800047729 >> 172.21.15.22:0/1800047729 conn(0x7f7db00fcae0 msgr2=0x7f7db00fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:03.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 -- 172.21.15.22:0/1800047729 shutdown_connections 2024-03-24T06:37:03.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.597+0000 7f7db5200640 1 -- 172.21.15.22:0/1800047729 wait complete. 2024-03-24T06:37:03.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.598+0000 7f7db5200640 1 Processor -- start 2024-03-24T06:37:03.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.598+0000 7f7db5200640 1 -- start start 2024-03-24T06:37:03.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 0x7f7db0199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:03.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:03.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 0x7f7db01a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01a12c0 con 0x7f7db0102760 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01a1430 con 0x7f7db0103120 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7db5200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7db01a15f0 con 0x7f7db0101570 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7da7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 0x7f7db01a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7da7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42778/0 (socket says 172.21.15.22:42778) 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7da7e00640 1 -- 172.21.15.22:0/3610842653 learned_addr learned my addr 172.21.15.22:0/3610842653 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7daf400640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 0x7f7db0199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 -- 172.21.15.22:0/3610842653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 msgr2=0x7f7db0199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 0x7f7db0199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 -- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 msgr2=0x7f7db0199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7dafe00640 1 -- 172.21.15.22:0/3610842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d98009490 con 0x7f7db0103120 2024-03-24T06:37:03.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.599+0000 7f7daf400640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 0x7f7db0199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:03.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7dafe00640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 0x7f7db01a0d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7da800ef00 tx=0x7f7da800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:03.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da8004b20 con 0x7f7db0103120 2024-03-24T06:37:03.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da8003e50 con 0x7f7db0103120 2024-03-24T06:37:03.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da0009490 con 0x7f7db0103120 2024-03-24T06:37:03.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da800b570 con 0x7f7db0103120 2024-03-24T06:37:03.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.600+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db01a1c00 con 0x7f7db0103120 2024-03-24T06:37:03.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.601+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db010cf80 con 0x7f7db0103120 2024-03-24T06:37:03.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.605+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7da800a040 con 0x7f7db0103120 2024-03-24T06:37:03.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.606+0000 7f7dacc00640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 0x7f7d940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:03.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.606+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7da80cddc0 con 0x7f7db0103120 2024-03-24T06:37:03.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.606+0000 7f7daf400640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 0x7f7d940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:03.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.607+0000 7f7daf400640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 0x7f7d940a4580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009830 tx=0x7f7da0034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:03.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.611+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7da8096af0 con 0x7f7db0103120 2024-03-24T06:37:03.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:03 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T06:37:03.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:03 smithi022 ceph-mon[32003]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T06:37:03.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:03 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:03.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:03 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2434598388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:03 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T06:37:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:03 smithi143 ceph-mon[36520]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T06:37:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:03 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:03 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2434598388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:03.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.816+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7db01a20a0 con 0x7f7db0103120 2024-03-24T06:37:03.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.818+0000 7f7dacc00640 1 -- 172.21.15.22:0/3610842653 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f7da809a1b0 con 0x7f7db0103120 2024-03-24T06:37:03.818 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:03.819 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.821+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 msgr2=0x7f7d940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.821+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 0x7f7d940a4580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009830 tx=0x7f7da0034c00 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 msgr2=0x7f7db01a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 0x7f7db01a0d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7da800ef00 tx=0x7f7da800c740 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 shutdown_connections 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7d940a20c0 0x7f7d940a4580 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7db0103120 0x7f7db01a0d80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7db0102760 0x7f7db0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 --2- 172.21.15.22:0/3610842653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7db0101570 0x7f7db0199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 >> 172.21.15.22:0/3610842653 conn(0x7f7db00fcae0 msgr2=0x7f7db0073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 shutdown_connections 2024-03-24T06:37:03.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:03.822+0000 7f7db5200640 1 -- 172.21.15.22:0/3610842653 wait complete. 2024-03-24T06:37:03.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:37:04.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 -- 172.21.15.22:0/1708484294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500104b20 msgr2=0x7f3500104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 --2- 172.21.15.22:0/1708484294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500104b20 0x7f3500104fa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009900 tx=0x7f34ec02bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:04.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 -- 172.21.15.22:0/1708484294 shutdown_connections 2024-03-24T06:37:04.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 --2- 172.21.15.22:0/1708484294 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 0x7f350010bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 --2- 172.21.15.22:0/1708484294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500104b20 0x7f3500104fa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 --2- 172.21.15.22:0/1708484294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500103930 0x7f3500103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 -- 172.21.15.22:0/1708484294 >> 172.21.15.22:0/1708484294 conn(0x7f35000feec0 msgr2=0x7f35001012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:04.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 -- 172.21.15.22:0/1708484294 shutdown_connections 2024-03-24T06:37:04.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.088+0000 7f3506000640 1 -- 172.21.15.22:0/1708484294 wait complete. 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.089+0000 7f3506000640 1 Processor -- start 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 -- start start 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500104b20 0x7f3500199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 0x7f35001a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a14a0 con 0x7f35001054e0 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a1610 con 0x7f3500103930 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f3506000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f35001a17d0 con 0x7f3500104b20 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f34ff400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f34ff400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:55478/0 (socket says 172.21.15.22:55478) 2024-03-24T06:37:04.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f34ff400640 1 -- 172.21.15.22:0/1885831255 learned_addr learned my addr 172.21.15.22:0/1885831255 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.090+0000 7f34ffe00640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 0x7f35001a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34fea00640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500104b20 0x7f3500199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 -- 172.21.15.22:0/1885831255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500104b20 msgr2=0x7f3500199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500104b20 0x7f3500199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 -- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 msgr2=0x7f35001a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 0x7f35001a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 -- 172.21.15.22:0/1885831255 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f0009840 con 0x7f3500103930 2024-03-24T06:37:04.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.091+0000 7f34ff400640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009d20 tx=0x7f34f000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.092+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0009b40 con 0x7f3500103930 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.092+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0010b10 con 0x7f3500103930 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.092+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0018cc0 con 0x7f3500103930 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.092+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ec009560 con 0x7f3500103930 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.092+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35001a1e50 con 0x7f3500103930 2024-03-24T06:37:04.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.093+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350010acd0 con 0x7f3500103930 2024-03-24T06:37:04.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.095+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f34f0010c80 con 0x7f3500103930 2024-03-24T06:37:04.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.096+0000 7f34f7e00640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 0x7f34e40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.096+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f34f0014070 con 0x7f3500103930 2024-03-24T06:37:04.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.097+0000 7f34fea00640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 0x7f34e40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.097+0000 7f34fea00640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 0x7f34e40a4600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34ec000c00 tx=0x7f34ec02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:04.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.098+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f34f0076280 con 0x7f3500103930 2024-03-24T06:37:04.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.350+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f350010f360 con 0x7f3500103930 2024-03-24T06:37:04.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.351+0000 7f34f7e00640 1 -- 172.21.15.22:0/1885831255 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f34f0097670 con 0x7f3500103930 2024-03-24T06:37:04.351 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:04.351 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24496,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24544,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24565,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24584,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:01.829667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24601},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24601,"qdb_cluster":[24601]},"id":1}]} 2024-03-24T06:37:04.351 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 17 2024-03-24T06:37:04.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 msgr2=0x7f34e40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 0x7f34e40a4600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f34ec000c00 tx=0x7f34ec02c7e0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 msgr2=0x7f3500199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009d20 tx=0x7f34f000c6a0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 shutdown_connections 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f34e40a2140 0x7f34e40a4600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35001054e0 0x7f35001a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3500104b20 0x7f3500199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 --2- 172.21.15.22:0/1885831255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3500103930 0x7f3500199a00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 >> 172.21.15.22:0/1885831255 conn(0x7f35000feec0 msgr2=0x7f3500107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:04.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 shutdown_connections 2024-03-24T06:37:04.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.354+0000 7f3506000640 1 -- 172.21.15.22:0/1885831255 wait complete. 2024-03-24T06:37:04.373 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:04.458 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:04 smithi022 ceph-mon[32003]: pgmap v323: 129 pgs: 9 creating+peering, 104 active+clean, 16 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-24T06:37:04.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:04 smithi022 ceph-mon[32003]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:37:04.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:04 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:37:04.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:04 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/655075325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:04 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3610842653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:04 smithi155 ceph-mon[36262]: pgmap v323: 129 pgs: 9 creating+peering, 104 active+clean, 16 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-24T06:37:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:04 smithi155 ceph-mon[36262]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:37:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:04 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:37:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:04 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/655075325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:04 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3610842653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 -- 172.21.15.22:0/4021501922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 msgr2=0x7f8698101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 --2- 172.21.15.22:0/4021501922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f8698101740 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8690009ca0 tx=0x7f869002c100 comp rx=0 tx=0).stop 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 -- 172.21.15.22:0/4021501922 shutdown_connections 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 --2- 172.21.15.22:0/4021501922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f8698101740 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 --2- 172.21.15.22:0/4021501922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8698100920 0x7f8698100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 --2- 172.21.15.22:0/4021501922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86980fffc0 0x7f86981003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.599+0000 7f869f400640 1 -- 172.21.15.22:0/4021501922 >> 172.21.15.22:0/4021501922 conn(0x7f86980fbbc0 msgr2=0x7f86980fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.600+0000 7f869f400640 1 -- 172.21.15.22:0/4021501922 shutdown_connections 2024-03-24T06:37:04.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.600+0000 7f869f400640 1 -- 172.21.15.22:0/4021501922 wait complete. 2024-03-24T06:37:04.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 Processor -- start 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 -- start start 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 0x7f8698199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f86981a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f86981a1450 con 0x7f86981012c0 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f86981a15c0 con 0x7f8698100920 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869f400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f86981a1780 con 0x7f86980fffc0 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f8697e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f86981a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f8697e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:55498/0 (socket says 172.21.15.22:55498) 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f8697e00640 1 -- 172.21.15.22:0/2510616347 learned_addr learned my addr 172.21.15.22:0/2510616347 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.601+0000 7f869ca00640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 0x7f8698199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 -- 172.21.15.22:0/2510616347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 msgr2=0x7f8698199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 0x7f8698199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 -- 172.21.15.22:0/2510616347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 msgr2=0x7f8698199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 -- 172.21.15.22:0/2510616347 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8690009900 con 0x7f86981012c0 2024-03-24T06:37:04.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869ca00640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 0x7f8698199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:04.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f8697e00640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.602+0000 7f869d400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f86981a0f10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8690037040 tx=0x7f8690032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.603+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690034640 con 0x7f86981012c0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.603+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8690003e50 con 0x7f86981012c0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.603+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86981a1a00 con 0x7f86981012c0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.603+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690040450 con 0x7f86981012c0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.603+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86981a1e80 con 0x7f86981012c0 2024-03-24T06:37:04.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.604+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f869810ae50 con 0x7f86981012c0 2024-03-24T06:37:04.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.608+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f869000a040 con 0x7f86981012c0 2024-03-24T06:37:04.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.610+0000 7f8695600640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 0x7f86800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:04.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.611+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f86900f10e0 con 0x7f86981012c0 2024-03-24T06:37:04.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.611+0000 7f869ca00640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 0x7f86800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:04.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.612+0000 7f869ca00640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 0x7f86800a4580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f868c009fc0 tx=0x7f868c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:04.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.612+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86900b9f30 con 0x7f86981012c0 2024-03-24T06:37:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:04 smithi143 ceph-mon[36520]: pgmap v323: 129 pgs: 9 creating+peering, 104 active+clean, 16 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-24T06:37:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:04 smithi143 ceph-mon[36520]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T06:37:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:04 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:37:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:04 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/655075325' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:04 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3610842653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:04.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.843+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f869810a8c0 con 0x7f86981012c0 2024-03-24T06:37:04.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.845+0000 7f8695600640 1 -- 172.21.15.22:0/2510616347 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f8690039090 con 0x7f86981012c0 2024-03-24T06:37:04.845 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:04.845 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 msgr2=0x7f86800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 0x7f86800a4580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f868c009fc0 tx=0x7f868c005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 msgr2=0x7f86981a0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f86981a0f10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8690037040 tx=0x7f8690032fc0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 shutdown_connections 2024-03-24T06:37:04.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f86800a20c0 0x7f86800a4580 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86981012c0 0x7f86981a0f10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8698100920 0x7f8698199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 --2- 172.21.15.22:0/2510616347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86980fffc0 0x7f8698199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 >> 172.21.15.22:0/2510616347 conn(0x7f86980fbbc0 msgr2=0x7f86980fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.847+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 shutdown_connections 2024-03-24T06:37:04.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:04.848+0000 7f869f400640 1 -- 172.21.15.22:0/2510616347 wait complete. 2024-03-24T06:37:04.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T06:36:56.216020+0000', 'modified': '2024-03-24T06:37:01.829667+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24601}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24601': {'gid': 24601, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.22:6841/857116957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6840', 'nonce': 857116957}, {'type': 'v1', 'addr': '172.21.15.22:6841', 'nonce': 857116957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24601, 'qdb_cluster': [24601]} 2024-03-24T06:37:04.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T06:37:04.865 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.073+0000 7fe61dc00640 1 -- 172.21.15.22:0/3821272794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 msgr2=0x7fe618104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.073+0000 7fe61dc00640 1 --2- 172.21.15.22:0/3821272794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618104fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe604009900 tx=0x7fe60402bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 -- 172.21.15.22:0/3821272794 shutdown_connections 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 --2- 172.21.15.22:0/3821272794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe61810bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 --2- 172.21.15.22:0/3821272794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618104fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 --2- 172.21.15.22:0/3821272794 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe618103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 -- 172.21.15.22:0/3821272794 >> 172.21.15.22:0/3821272794 conn(0x7fe6180feec0 msgr2=0x7fe618101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 -- 172.21.15.22:0/3821272794 shutdown_connections 2024-03-24T06:37:05.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.074+0000 7fe61dc00640 1 -- 172.21.15.22:0/3821272794 wait complete. 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 Processor -- start 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 -- start start 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe6181a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe6181a14c0 con 0x7fe618103950 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe6181a1630 con 0x7fe618105500 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.075+0000 7fe61dc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe6181a17f0 con 0x7fe618104b40 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:41416/0 (socket says 172.21.15.22:41416) 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 -- 172.21.15.22:0/2422985501 learned_addr learned my addr 172.21.15.22:0/2422985501 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe617400640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe6181a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 -- 172.21.15.22:0/2422985501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 msgr2=0x7fe618199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 -- 172.21.15.22:0/2422985501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 msgr2=0x7fe6181a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe6181a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 -- 172.21.15.22:0/2422985501 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe608009490 con 0x7fe618103950 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe617400640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe6181a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:05.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.076+0000 7fe616a00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe60800e9c0 tx=0x7fe60800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.077+0000 7fe60f400640 1 -- 172.21.15.22:0/2422985501 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60800cde0 con 0x7fe618103950 2024-03-24T06:37:05.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.077+0000 7fe60f400640 1 -- 172.21.15.22:0/2422985501 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe608010910 con 0x7fe618103950 2024-03-24T06:37:05.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.077+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe604009560 con 0x7fe618103950 2024-03-24T06:37:05.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.077+0000 7fe60f400640 1 -- 172.21.15.22:0/2422985501 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608018a10 con 0x7fe618103950 2024-03-24T06:37:05.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.077+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6181a1e20 con 0x7fe618103950 2024-03-24T06:37:05.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.080+0000 7fe62083a740 1 Processor -- start 2024-03-24T06:37:05.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.080+0000 7fe60f400640 1 -- 172.21.15.22:0/2422985501 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe608010a80 con 0x7fe618103950 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.080+0000 7fe62083a740 1 -- start start 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.080+0000 7fe62083a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea96d60 0x55b56ea97180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.080+0000 7fe62083a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe62083a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56ea9ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe62083a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b56e9fd050 con 0x55b56ea97750 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe62083a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b56e9fa580 con 0x55b56ea9c8c0 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe62083a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b56e9f5d60 con 0x55b56ea96d60 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:41428/0 (socket says 172.21.15.22:41428) 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 -- 172.21.15.22:0/3362970436 learned_addr learned my addr 172.21.15.22:0/3362970436 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:05.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe617400640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea96d60 0x55b56ea97180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 -- 172.21.15.22:0/3362970436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea96d60 msgr2=0x55b56ea97180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea96d60 0x55b56ea97180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 -- 172.21.15.22:0/3362970436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 msgr2=0x55b56ea9ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616000640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56ea9ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56ea9ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616a00640 1 -- 172.21.15.22:0/3362970436 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b56e9ae620 con 0x55b56ea97750 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.081+0000 7fe616000640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56ea9ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe616a00640 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe6080d6430 tx=0x7fe6080d6520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b622f42a891501c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe603e00640 1 -- 172.21.15.22:0/3362970436 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6080ceab0 con 0x55b56ea97750 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe60f400640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 0x7fe5fc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe603e00640 1 -- 172.21.15.22:0/3362970436 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6080d05b0 con 0x55b56ea97750 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe603e00640 1 -- 172.21.15.22:0/3362970436 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6080973f0 con 0x55b56ea97750 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe60f400640 1 -- 172.21.15.22:0/2422985501 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe608014070 con 0x7fe618103950 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.082+0000 7fe617400640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 0x7fe5fc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 -- 172.21.15.22:0/3362970436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 msgr2=0x55b56ea9c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe6080d6430 tx=0x7fe6080d6520 comp rx=0 tx=0).stop 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 -- 172.21.15.22:0/3362970436 shutdown_connections 2024-03-24T06:37:05.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56ea9ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56ea9c250 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 --2- 172.21.15.22:0/3362970436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea96d60 0x55b56ea97180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 -- 172.21.15.22:0/3362970436 >> 172.21.15.22:0/3362970436 conn(0x55b56ea89ac0 msgr2=0x55b56ea89ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe617400640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 0x7fe5fc0a4580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55b56ea97fb0 tx=0x7fe610009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 -- 172.21.15.22:0/3362970436 shutdown_connections 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.083+0000 7fe62083a740 1 -- 172.21.15.22:0/3362970436 wait complete. 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 -- start start 2024-03-24T06:37:05.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 Processor -- start 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b56e99afb0 con 0x55b56ea9c8c0 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b56e99ac50 con 0x55b56ea96d60 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b56e73df20 con 0x55b56ea97750 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.084+0000 7fe62083a740 10 client.0 _ll_register_callbacks cb 0x55b56e706ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:37:05.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:34150/0 (socket says 172.21.15.22:34150) 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616000640 1 -- 172.21.15.22:0/2114825362 learned_addr learned my addr 172.21.15.22:0/2114825362 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52460/0 (socket says 172.21.15.22:52460) 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 -- 172.21.15.22:0/2114825362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 msgr2=0x55b56eaa14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 -- 172.21.15.22:0/2114825362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 msgr2=0x55b56eaa0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 -- 172.21.15.22:0/2114825362 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6181a1e20 con 0x55b56ea9c8c0 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe616a00640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe617400640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa8500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe610011c60 tx=0x7fe610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99afc0b4dc6cbc82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.085+0000 7fe600c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:37:05.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610020620 con 0x55b56ea9c8c0 2024-03-24T06:37:05.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b56e6f56c0 con 0x55b56ea9c8c0 2024-03-24T06:37:05.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe610020780 con 0x55b56ea9c8c0 2024-03-24T06:37:05.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fe61001fe80 con 0x55b56ea9c8c0 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe62083a740 10 client.14577 fetch_fsmap learned FSMap version 17 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55b56e9f43e0 con 0x55b56ea9c8c0 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.086+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b56e6f56c0 con 0x55b56ea9c8c0 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fe61001d970 con 0x55b56ea9c8c0 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 10 client.14577 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 10 client.14577 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:37:05.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 20 client.14577 populate_metadata read hostname 'smithi022' 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eaadc60 0x55b56eaaf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 4 client.14577 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b56e746270 con 0x55b56eaadc60 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.087+0000 7fe616a00640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eaadc60 0x55b56eaaf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.088+0000 7fe616a00640 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eaadc60 0x55b56eaaf0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=848859769533d33e server_cookie=535f1c9a4522b047 in_seq=0 out_seq=0 2024-03-24T06:37:05.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.088+0000 7fe600c00640 10 client.14577 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe600c00640 1 -- 172.21.15.22:0/2114825362 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b56e746270 con 0x55b56eaadc60 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe600c00640 10 client.14577 handle_command_reply: tid=0 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 client.14577 shutdown 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 msgr2=0x55b56eaa8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa8500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe610011c60 tx=0x7fe610004a40 comp rx=0 tx=0).stop 2024-03-24T06:37:05.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 shutdown_connections 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa8500 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eaadc60 0x55b56eaaf0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 --2- 172.21.15.22:0/2114825362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe600c00640 0 client.14577 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.089+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 >> 172.21.15.22:0/2114825362 conn(0x55b56ea89ac0 msgr2=0x55b56ea88cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.090+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 shutdown_connections 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.090+0000 7fe62083a740 1 -- 172.21.15.22:0/2114825362 wait complete. 2024-03-24T06:37:05.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.090+0000 7fe62083a740 20 client.14577 trim_cache size 0 max 16384 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 Processor -- start 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 -- start start 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea96d60 0x55b56ea84890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56eaacbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b56ea8e110 con 0x55b56ea96d60 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b56eaae560 con 0x55b56ea9c8c0 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.110+0000 7fe62083a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b56eaa9180 con 0x55b56ea97750 2024-03-24T06:37:05.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:34156/0 (socket says 172.21.15.22:34156) 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 -- 172.21.15.22:0/1087927776 learned_addr learned my addr 172.21.15.22:0/1087927776 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616a00640 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea96d60 0x55b56ea84890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe617400640 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56eaacbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 -- 172.21.15.22:0/1087927776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 msgr2=0x55b56eaacbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56eaacbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 -- 172.21.15.22:0/1087927776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea96d60 msgr2=0x55b56ea84890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea96d60 0x55b56ea84890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.111+0000 7fe616000640 1 -- 172.21.15.22:0/1087927776 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b56e990900 con 0x55b56ea97750 2024-03-24T06:37:05.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.112+0000 7fe616000640 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe604036040 tx=0x7fe60402ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884d0f43551809b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.112+0000 7fe603400640 1 -- 172.21.15.22:0/1087927776 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60402cc70 con 0x55b56ea97750 2024-03-24T06:37:05.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.112+0000 7fe603400640 1 -- 172.21.15.22:0/1087927776 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe604004040 con 0x55b56ea97750 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.112+0000 7fe603400640 1 -- 172.21.15.22:0/1087927776 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604005180 con 0x55b56ea97750 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- 172.21.15.22:0/1087927776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 msgr2=0x55b56eaa84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe604036040 tx=0x7fe60402ca40 comp rx=0 tx=0).stop 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- 172.21.15.22:0/1087927776 shutdown_connections 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea9c8c0 0x55b56eaacbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea97750 0x55b56eaa84c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 --2- 172.21.15.22:0/1087927776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea96d60 0x55b56ea84890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- 172.21.15.22:0/1087927776 >> 172.21.15.22:0/1087927776 conn(0x55b56ea89ac0 msgr2=0x55b56ea88cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- 172.21.15.22:0/1087927776 shutdown_connections 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- 172.21.15.22:0/1087927776 wait complete. 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 -- start start 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.113+0000 7fe62083a740 1 Processor -- start 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b56eaa7640 con 0x55b56ea97750 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b56eaa77b0 con 0x55b56ea96d60 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b56eaa7970 con 0x55b56ea9c8c0 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe62083a740 10 client.0 _ll_register_callbacks cb 0x7fe610009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:41440/0 (socket says 172.21.15.22:41440) 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 -- 172.21.15.22:0/2524955461 learned_addr learned my addr 172.21.15.22:0/2524955461 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe617400640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616a00640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 -- 172.21.15.22:0/2524955461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea9c8c0 msgr2=0x55b56eaa7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 -- 172.21.15.22:0/2524955461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 msgr2=0x55b56eaa3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.114+0000 7fe616000640 1 -- 172.21.15.22:0/2524955461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61810c690 con 0x55b56ea97750 2024-03-24T06:37:05.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe616000640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe604000c00 tx=0x7fe6040048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817afdb4dd630408 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:05.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe600c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60402cc70 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6080d05b0 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe604005310 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fe604004df0 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe62083a740 10 client.14583 fetch_fsmap learned FSMap version 17 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.115+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55b56e9b6560 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6080d05b0 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fe60403c040 con 0x55b56ea97750 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 10 client.14583 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 10 client.14583 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:37:05.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 20 client.14583 populate_metadata read hostname 'smithi022' 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eb17e60 0x55b56eb16b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 4 client.14583 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "status"}) -- 0x55b56e732610 con 0x55b56eb17e60 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.116+0000 7fe616a00640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eb17e60 0x55b56eb16b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe616a00640 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eb17e60 0x55b56eb16b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf238ce1119ecca server_cookie=35b6dead1acf9f67 in_seq=0 out_seq=0 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe600c00640 10 client.14583 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe600c00640 1 -- 172.21.15.22:0/2524955461 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55b56e732610 con 0x55b56eb17e60 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe600c00640 10 client.14583 handle_command_reply: tid=0 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 client.14583 shutdown 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 msgr2=0x55b56eaa40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe604000c00 tx=0x7fe6040048f0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 shutdown_connections 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b56ea9c8c0 0x55b56eaa7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b56ea97750 0x55b56eaa40e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55b56eb17e60 0x55b56eb16b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 --2- 172.21.15.22:0/2524955461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b56ea96d60 0x55b56eaa3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe600c00640 0 client.14583 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.117+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 >> 172.21.15.22:0/2524955461 conn(0x55b56ea89ac0 msgr2=0x55b56ea8d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 shutdown_connections 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe62083a740 1 -- 172.21.15.22:0/2524955461 wait complete. 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe62083a740 20 client.14583 trim_cache size 0 max 16384 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 0, 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24601, 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:37:05.121 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 7.9534742879999998, 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 17, 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 89, 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 88, 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 17.853963341 2024-03-24T06:37:05.122 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 msgr2=0x7fe5fc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 0x7fe5fc0a4580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55b56ea97fb0 tx=0x7fe610009800 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 msgr2=0x7fe6181999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe60800e9c0 tx=0x7fe60800ee80 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.118+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 shutdown_connections 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5fc0a20c0 0x7fe5fc0a4580 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe618105500 0x7fe6181a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe618104b40 0x7fe618199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 --2- 172.21.15.22:0/2422985501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe618103950 0x7fe6181999d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 >> 172.21.15.22:0/2422985501 conn(0x7fe6180feec0 msgr2=0x7fe618107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 shutdown_connections 2024-03-24T06:37:05.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:05.119+0000 7fe61dc00640 1 -- 172.21.15.22:0/2422985501 wait complete. 2024-03-24T06:37:05.134 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T06:37:05.135 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:05.135 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T06:37:05.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:05 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1885831255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:05 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2510616347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:05 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1885831255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:05 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2510616347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:05.634 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:05.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:05 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1885831255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:05.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:05 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2510616347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:06.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:06 smithi022 ceph-mon[32003]: pgmap v324: 129 pgs: 9 creating+peering, 119 active+clean, 1 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T06:37:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:06 smithi155 ceph-mon[36262]: pgmap v324: 129 pgs: 9 creating+peering, 119 active+clean, 1 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T06:37:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:06 smithi143 ceph-mon[36520]: pgmap v324: 129 pgs: 9 creating+peering, 119 active+clean, 1 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T06:37:08.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 -- 172.21.15.22:0/3613790528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 msgr2=0x7fdf4c074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:08.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 --2- 172.21.15.22:0/3613790528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c074810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdf400098f0 tx=0x7fdf40009c00 comp rx=0 tx=0).stop 2024-03-24T06:37:08.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 -- 172.21.15.22:0/3613790528 shutdown_connections 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 --2- 172.21.15.22:0/3613790528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 --2- 172.21.15.22:0/3613790528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c074810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 --2- 172.21.15.22:0/3613790528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c074d50 0x7fdf4c10e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 -- 172.21.15.22:0/3613790528 >> 172.21.15.22:0/3613790528 conn(0x7fdf4c079350 msgr2=0x7fdf4c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 -- 172.21.15.22:0/3613790528 shutdown_connections 2024-03-24T06:37:08.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.178+0000 7fdf54800700 1 -- 172.21.15.22:0/3613790528 wait complete. 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.179+0000 7fdf54800700 1 Processor -- start 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.179+0000 7fdf54800700 1 -- start start 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c19af30 con 0x7fdf4c074d50 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c19b0a0 con 0x7fdf4c10ec30 2024-03-24T06:37:08.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf54800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c19b260 con 0x7fdf4c074430 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf51400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf51400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:41466/0 (socket says 172.21.15.22:41466) 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf51400700 1 -- 172.21.15.22:0/450878513 learned_addr learned my addr 172.21.15.22:0/450878513 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf51e00700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.180+0000 7fdf52800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:08.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 -- 172.21.15.22:0/450878513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 msgr2=0x7fdf4c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 -- 172.21.15.22:0/450878513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 msgr2=0x7fdf4c19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 -- 172.21.15.22:0/450878513 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf40009520 con 0x7fdf4c074d50 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51e00700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf52800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.181+0000 7fdf51400700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00e9a0 tx=0x7fdf3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:08.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.182+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c009790 con 0x7fdf4c074d50 2024-03-24T06:37:08.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.182+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf3c010430 con 0x7fdf4c074d50 2024-03-24T06:37:08.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.182+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4c19b540 con 0x7fdf4c074d50 2024-03-24T06:37:08.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.182+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c018680 con 0x7fdf4c074d50 2024-03-24T06:37:08.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.182+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4c1a76e0 con 0x7fdf4c074d50 2024-03-24T06:37:08.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.183+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4c075af0 con 0x7fdf4c074d50 2024-03-24T06:37:08.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.187+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fdf3c010910 con 0x7fdf4c074d50 2024-03-24T06:37:08.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.188+0000 7fdf46a00700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 0x7fdf340a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:08.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.189+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdf3c014070 con 0x7fdf4c074d50 2024-03-24T06:37:08.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.189+0000 7fdf51e00700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 0x7fdf340a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:08.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.190+0000 7fdf51e00700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 0x7fdf340a4610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009810 tx=0x7fdf40019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:08.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.191+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf3c097350 con 0x7fdf4c074d50 2024-03-24T06:37:08.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:08 smithi022 ceph-mon[32003]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 148 B/s wr, 0 op/s 2024-03-24T06:37:08.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.516+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdf4c114560 con 0x7fdf4c074d50 2024-03-24T06:37:08.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.517+0000 7fdf46a00700 1 -- 172.21.15.22:0/450878513 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fdf3c026070 con 0x7fdf4c074d50 2024-03-24T06:37:08.517 INFO:teuthology.orchestra.run.smithi022.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 msgr2=0x7fdf340a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 0x7fdf340a4610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009810 tx=0x7fdf40019040 comp rx=0 tx=0).stop 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 msgr2=0x7fdf4c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00e9a0 tx=0x7fdf3c00ecb0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 shutdown_connections 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fdf340a2150 0x7fdf340a4610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf4c10ec30 0x7fdf4c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf4c074430 0x7fdf4c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 --2- 172.21.15.22:0/450878513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdf4c074d50 0x7fdf4c1a0bf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 >> 172.21.15.22:0/450878513 conn(0x7fdf4c079350 msgr2=0x7fdf4c100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 shutdown_connections 2024-03-24T06:37:08.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:08.520+0000 7fdf54800700 1 -- 172.21.15.22:0/450878513 wait complete. 2024-03-24T06:37:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:08 smithi155 ceph-mon[36262]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 148 B/s wr, 0 op/s 2024-03-24T06:37:08.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:08 smithi143 ceph-mon[36520]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 148 B/s wr, 0 op/s 2024-03-24T06:37:09.522 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:09 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/450878513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi022.front.sepia.ceph.com 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T06:37:09.526 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T06:37:09.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:09.527 DEBUG:teuthology.orchestra.run.smithi022:> ip addr 2024-03-24T06:37:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:09 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/450878513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:37:09.585 INFO:teuthology.orchestra.run.smithi022.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 ::1/128 scope host 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9a brd ff:ff:ff:ff:ff:ff 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f0 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9b brd ff:ff:ff:ff:ff:ff 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f1 2024-03-24T06:37:09.586 INFO:teuthology.orchestra.run.smithi022.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e4 brd ff:ff:ff:ff:ff:ff 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f0 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: inet 172.21.15.22/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft 42053sec preferred_lft 42053sec 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e5 brd ff:ff:ff:ff:ff:ff 2024-03-24T06:37:09.587 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f1 2024-03-24T06:37:09.588 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add name ceph-brx type bridge 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr flush dev ceph-brx 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set ceph-brx up 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T06:37:09.588 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:37:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:09 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/450878513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:37:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:37:10.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:10.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:10.148 DEBUG:teuthology.orchestra.run.smithi022:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T06:37:10.230 INFO:teuthology.orchestra.run.smithi022.stdout:1 2024-03-24T06:37:10.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:10.233 DEBUG:teuthology.orchestra.run.smithi022:> route 2024-03-24T06:37:10.294 INFO:teuthology.orchestra.run.smithi022.stdout:Kernel IP routing table 2024-03-24T06:37:10.294 INFO:teuthology.orchestra.run.smithi022.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T06:37:10.294 INFO:teuthology.orchestra.run.smithi022.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T06:37:10.294 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T06:37:10.294 INFO:teuthology.orchestra.run.smithi022.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T06:37:10.295 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:37:10.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:37:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:10 smithi155 ceph-mon[36262]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:10.647 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:10 smithi022 ceph-mon[32003]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:10 smithi143 ceph-mon[36520]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:11.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:11.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:11.025 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T06:37:11.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:11.085 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list-id 2024-03-24T06:37:11.139 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:37:11.139 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:37:11.139 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:11.139 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-24T06:37:11.139 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:37:11.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:37:11.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:11.360 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-24T06:37:11.360 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:37:11.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:37:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:11.716 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:37:11.716 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:37:11.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set brx.0 up 2024-03-24T06:37:11.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T06:37:11.716 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:37:11.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:37:11.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:11.933 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-24T06:37:11.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T06:37:11.933 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:11.986 INFO:teuthology.orchestra.run.smithi022.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T06:37:11.986 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T06:37:11.989 DEBUG:teuthology.orchestra.run.smithi022:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:12.038 DEBUG:teuthology.orchestra.run.smithi022:> sudo modprobe fuse 2024-03-24T06:37:12.116 DEBUG:teuthology.orchestra.run.smithi022:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T06:37:12.156 INFO:teuthology.orchestra.run.smithi022.stdout:/proc 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/dev 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/security 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/shm 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/pts 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/run 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/cgroup 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/pstore 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/bpf 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/ 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/selinux 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/hugepages 2024-03-24T06:37:12.157 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/mqueue 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/debug 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/tracing 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/fuse/connections 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/config 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T06:37:12.158 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/1000 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/ceph 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/47cfffe28d2a8e4e52d93f3d6abe92c5f09685f522cd92e89b30335f2dbdf683/merged 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3683ace59444bfde9c10a5ce30ad4a69b1e7589fa4cb82ea2ddab6c2ca69d0a8/merged 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/0 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3a471cc7e51efba6f58ae125290182f430f3929a1d633155d47ff0178cdf7b05/merged 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/98bddbb076661884447c9503f029b2b50b490ea13320c3a8ab28b37474f91695/merged 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/fc7fc77c50b8fb0aa9ee5dde32a3af0ab75f5056e7f2d44bd4750de05ef02acf/merged 2024-03-24T06:37:12.161 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/abcbe859200859c50dd2d7711a4681f042bf0efc0f74cee5b8259cd68a101e0e/merged 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3907e94e5386a04b0119bdcdfbae8de30e9e6ec0c169a289805ca1b2afb7159c/merged 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/4d83ac292f3bd22059474324dd43fe2c56e41aad618afeefbf4bacac9bf9c565/merged 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/be3d9b67372da9f70c9f49399fa8de00c7f3f1ee461e115f4662261e69bb5c87/merged 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/f84bbe2a2efe9d0035a7d6a4031b6ad4999106a79c4c5024d5f5557880db6a17/merged 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:12.162 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:12.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:12.163 DEBUG:teuthology.orchestra.run.smithi022:> ls /sys/fs/fuse/connections 2024-03-24T06:37:12.259 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T06:37:12.260 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-24T06:37:12.266 DEBUG:teuthology.orchestra.run.smithi022:> sudo modprobe fuse 2024-03-24T06:37:12.348 DEBUG:teuthology.orchestra.run.smithi022:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T06:37:12.370 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:12 smithi022 ceph-mon[32003]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:12.393 INFO:teuthology.orchestra.run.smithi022.stdout:/proc 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/dev 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/security 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/shm 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/pts 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/run 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/cgroup 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/pstore 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/bpf 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/ 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/selinux 2024-03-24T06:37:12.394 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/hugepages 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/mqueue 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/debug 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/tracing 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/fuse/connections 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/config 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/1000 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/ceph 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/47cfffe28d2a8e4e52d93f3d6abe92c5f09685f522cd92e89b30335f2dbdf683/merged 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3683ace59444bfde9c10a5ce30ad4a69b1e7589fa4cb82ea2ddab6c2ca69d0a8/merged 2024-03-24T06:37:12.395 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/0 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3a471cc7e51efba6f58ae125290182f430f3929a1d633155d47ff0178cdf7b05/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/98bddbb076661884447c9503f029b2b50b490ea13320c3a8ab28b37474f91695/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/fc7fc77c50b8fb0aa9ee5dde32a3af0ab75f5056e7f2d44bd4750de05ef02acf/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/abcbe859200859c50dd2d7711a4681f042bf0efc0f74cee5b8259cd68a101e0e/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3907e94e5386a04b0119bdcdfbae8de30e9e6ec0c169a289805ca1b2afb7159c/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/4d83ac292f3bd22059474324dd43fe2c56e41aad618afeefbf4bacac9bf9c565/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/be3d9b67372da9f70c9f49399fa8de00c7f3f1ee461e115f4662261e69bb5c87/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/f84bbe2a2efe9d0035a7d6a4031b6ad4999106a79c4c5024d5f5557880db6a17/merged 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:12.396 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:12.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:12.397 DEBUG:teuthology.orchestra.run.smithi022:> ls /sys/fs/fuse/connections 2024-03-24T06:37:12.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi022.stderr:2024-03-24T06:37:12.483+0000 7f89e440c480 -1 init, newargv = 0x557aa9b35a80 newargc=15 2024-03-24T06:37:12.483 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi022.stderr:ceph-fuse[75698]: starting ceph client 2024-03-24T06:37:12.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi022.stderr:ceph-fuse[75698]: starting fuse 2024-03-24T06:37:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:12 smithi155 ceph-mon[36262]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:12.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:12 smithi143 ceph-mon[36520]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:37:13.591 DEBUG:teuthology.orchestra.run.smithi022:> sudo modprobe fuse 2024-03-24T06:37:13.699 DEBUG:teuthology.orchestra.run.smithi022:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/proc 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/sys 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/dev 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/security 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/shm 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/pts 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/run 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/cgroup 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/pstore 2024-03-24T06:37:13.757 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/bpf 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/ 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/selinux 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/hugepages 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/dev/mqueue 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/debug 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/tracing 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/fs/fuse/connections 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/sys/kernel/config 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T06:37:13.758 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/1000 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/ceph 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/47cfffe28d2a8e4e52d93f3d6abe92c5f09685f522cd92e89b30335f2dbdf683/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3683ace59444bfde9c10a5ce30ad4a69b1e7589fa4cb82ea2ddab6c2ca69d0a8/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/run/user/0 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3a471cc7e51efba6f58ae125290182f430f3929a1d633155d47ff0178cdf7b05/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/98bddbb076661884447c9503f029b2b50b490ea13320c3a8ab28b37474f91695/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/fc7fc77c50b8fb0aa9ee5dde32a3af0ab75f5056e7f2d44bd4750de05ef02acf/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/abcbe859200859c50dd2d7711a4681f042bf0efc0f74cee5b8259cd68a101e0e/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/3907e94e5386a04b0119bdcdfbae8de30e9e6ec0c169a289805ca1b2afb7159c/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/4d83ac292f3bd22059474324dd43fe2c56e41aad618afeefbf4bacac9bf9c565/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/be3d9b67372da9f70c9f49399fa8de00c7f3f1ee461e115f4662261e69bb5c87/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/var/lib/containers/storage/overlay/f84bbe2a2efe9d0035a7d6a4031b6ad4999106a79c4c5024d5f5557880db6a17/merged 2024-03-24T06:37:13.759 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns 2024-03-24T06:37:13.760 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:13.760 INFO:teuthology.orchestra.run.smithi022.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:37:13.760 INFO:teuthology.orchestra.run.smithi022.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:13.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:13.760 DEBUG:teuthology.orchestra.run.smithi022:> ls /sys/fs/fuse/connections 2024-03-24T06:37:13.815 INFO:teuthology.orchestra.run.smithi022.stdout:118 2024-03-24T06:37:13.815 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-24T06:37:13.815 DEBUG:teuthology.orchestra.run.smithi022:> sudo stdin-killer -- python3 -c ' 2024-03-24T06:37:13.815 DEBUG:teuthology.orchestra.run.smithi022:> import glob 2024-03-24T06:37:13.815 DEBUG:teuthology.orchestra.run.smithi022:> import re 2024-03-24T06:37:13.815 DEBUG:teuthology.orchestra.run.smithi022:> import os 2024-03-24T06:37:13.815 DEBUG:teuthology.orchestra.run.smithi022:> import subprocess 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> def _find_admin_socket(client_name): 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> files = glob.glob(asok_path) 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> # Given a non-glob path, it better be there 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> if "*" not in asok_path: 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> assert(len(files) == 1) 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> return files[0] 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> for f in files: 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> contents = proc_f.read() 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> if mountpoint in contents: 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> return f 2024-03-24T06:37:13.816 DEBUG:teuthology.orchestra.run.smithi022:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T06:37:13.817 DEBUG:teuthology.orchestra.run.smithi022:> 2024-03-24T06:37:13.817 DEBUG:teuthology.orchestra.run.smithi022:> print(_find_admin_socket("client.admin")) 2024-03-24T06:37:13.817 DEBUG:teuthology.orchestra.run.smithi022:> ' 2024-03-24T06:37:13.999 INFO:teuthology.orchestra.run.smithi022.stdout:/var/run/ceph/ceph-client.admin.75698.asok 2024-03-24T06:37:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:14.018 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75698.asok 2024-03-24T06:37:14.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:14.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75698.asok status 2024-03-24T06:37:14.256 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "metadata": { 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "entity_id": "admin", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "hostname": "smithi022", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "pid": "75698", 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "root": "/" 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "dentry_count": 0, 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "dentry_pinned_count": 0, 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 14595, 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "inst": { 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "name": { 2024-03-24T06:37:14.257 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "client", 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "num": 14595 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": { 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v1", 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "192.168.0.1:0", 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 1617864950 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": { 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v1", 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "192.168.0.1:0", 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 1617864950 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:37:14.258 INFO:teuthology.orchestra.run.smithi022.stdout: "inst_str": "client.14595 192.168.0.1:0/1617864950", 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "addr_str": "192.168.0.1:0/1617864950", 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "inode_count": 1, 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_epoch": 17, 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_epoch": 89, 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_epoch_barrier": 0, 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "blocklisted": false, 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs" 2024-03-24T06:37:14.259 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:37:14.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:14.274 DEBUG:teuthology.orchestra.run.smithi022:> stat --file-system '--printf=%T 2024-03-24T06:37:14.274 DEBUG:teuthology.orchestra.run.smithi022:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:14.327 INFO:teuthology.orchestra.run.smithi022.stdout:fuseblk 2024-03-24T06:37:14.327 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:14.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:14.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:14.423 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-24T06:37:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:14 smithi155 ceph-mon[36262]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 188 B/s wr, 0 op/s 2024-03-24T06:37:14.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T06:37:14.577 INFO:teuthology.orchestra.run.smithi022.stdout:# file: . 2024-03-24T06:37:14.578 INFO:teuthology.orchestra.run.smithi022.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-24T06:37:14.578 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:14.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:37:14.588 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:37:14.589 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T06:37:14.639 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:14.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:14 smithi022 ceph-mon[32003]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 188 B/s wr, 0 op/s 2024-03-24T06:37:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:14 smithi143 ceph-mon[36520]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 188 B/s wr, 0 op/s 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 -- 172.21.15.22:0/3062014801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 msgr2=0x7f7bf8101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 --2- 172.21.15.22:0/3062014801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8101970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009900 tx=0x7f7bec02bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 -- 172.21.15.22:0/3062014801 shutdown_connections 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 --2- 172.21.15.22:0/3062014801 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8103100 0x7f7bf8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 --2- 172.21.15.22:0/3062014801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8102740 0x7f7bf8102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 --2- 172.21.15.22:0/3062014801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8101970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 -- 172.21.15.22:0/3062014801 >> 172.21.15.22:0/3062014801 conn(0x7f7bf80fcaa0 msgr2=0x7f7bf80fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 -- 172.21.15.22:0/3062014801 shutdown_connections 2024-03-24T06:37:14.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.843+0000 7f7bff000640 1 -- 172.21.15.22:0/3062014801 wait complete. 2024-03-24T06:37:14.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.845+0000 7f7bff000640 1 Processor -- start 2024-03-24T06:37:14.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.845+0000 7f7bff000640 1 -- start start 2024-03-24T06:37:14.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.845+0000 7f7bff000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:14.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.845+0000 7f7bff000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:14.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bff000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 0x7f7bf819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:14.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bff000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf819f150 con 0x7f7bf8102740 2024-03-24T06:37:14.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bff000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf819f2c0 con 0x7f7bf8103100 2024-03-24T06:37:14.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bff000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf819f480 con 0x7f7bf8101550 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfd200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfd200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:41484/0 (socket says 172.21.15.22:41484) 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:34208/0 (socket says 172.21.15.22:34208) 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfd200640 1 -- 172.21.15.22:0/543623164 learned_addr learned my addr 172.21.15.22:0/543623164 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfe600640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 0x7f7bf819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:14.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 -- 172.21.15.22:0/543623164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 msgr2=0x7f7bf819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 0x7f7bf819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 -- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 msgr2=0x7f7bf8197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfdc00640 1 -- 172.21.15.22:0/543623164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf0009840 con 0x7f7bf8101550 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfd200640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.846+0000 7f7bfe600640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 0x7f7bf819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bfdc00640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7bec036040 tx=0x7f7bec005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0043d0 con 0x7f7bf8101550 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bec0052d0 con 0x7f7bf8101550 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bec009560 con 0x7f7bf8101550 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec003b60 con 0x7f7bf8101550 2024-03-24T06:37:14.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.847+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf819faa0 con 0x7f7bf8101550 2024-03-24T06:37:14.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.849+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf8108870 con 0x7f7bf8101550 2024-03-24T06:37:14.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.853+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7bec003cc0 con 0x7f7bf8101550 2024-03-24T06:37:14.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.855+0000 7f7bf6000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 0x7f7be00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:14.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.855+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7bec0f05c0 con 0x7f7bf8101550 2024-03-24T06:37:14.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.855+0000 7f7bfd200640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 0x7f7be00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:14.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.856+0000 7f7bfd200640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 0x7f7be00a4600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8198b70 tx=0x7f7be4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:14.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:14.859+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7bec0b9350 con 0x7f7bf8101550 2024-03-24T06:37:15.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.064+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7bf802d160 con 0x7f7bf8101550 2024-03-24T06:37:15.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.067+0000 7f7bf6000640 1 -- 172.21.15.22:0/543623164 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f7bec0bca10 con 0x7f7bf8101550 2024-03-24T06:37:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:15.067 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:15.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 msgr2=0x7f7be00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 0x7f7be00a4600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8198b70 tx=0x7f7be4005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:15.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 msgr2=0x7f7bf8197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7bec036040 tx=0x7f7bec005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 shutdown_connections 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7be00a2140 0x7f7be00a4600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf8103100 0x7f7bf819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7bf8102740 0x7f7bf8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 --2- 172.21.15.22:0/543623164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8101550 0x7f7bf8197660 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.070+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 >> 172.21.15.22:0/543623164 conn(0x7f7bf80fcaa0 msgr2=0x7f7bf8105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.071+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 shutdown_connections 2024-03-24T06:37:15.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.071+0000 7f7bff000640 1 -- 172.21.15.22:0/543623164 wait complete. 2024-03-24T06:37:15.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:37:15.090 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 -- 172.21.15.22:0/3045099484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8104a60 msgr2=0x7fe6c8106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/3045099484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8104a60 0x7fe6c8106e50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009860 tx=0x7fe6b002bb70 comp rx=0 tx=0).stop 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 -- 172.21.15.22:0/3045099484 shutdown_connections 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/3045099484 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 0x7fe6c8109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/3045099484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8104a60 0x7fe6c8106e50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/3045099484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8102130 0x7fe6c8104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 -- 172.21.15.22:0/3045099484 >> 172.21.15.22:0/3045099484 conn(0x7fe6c80fba80 msgr2=0x7fe6c80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 -- 172.21.15.22:0/3045099484 shutdown_connections 2024-03-24T06:37:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.358+0000 7fe6c7e00640 1 -- 172.21.15.22:0/3045099484 wait complete. 2024-03-24T06:37:15.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 Processor -- start 2024-03-24T06:37:15.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 -- start start 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 0x7fe6c8199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 0x7fe6c81a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c81a13b0 con 0x7fe6c8107390 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c81a1520 con 0x7fe6c8102130 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.359+0000 7fe6c7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c81a16e0 con 0x7fe6c8104a60 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44492/0 (socket says 172.21.15.22:44492) 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 -- 172.21.15.22:0/456423576 learned_addr learned my addr 172.21.15.22:0/456423576 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c7400640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 0x7fe6c81a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6bfe00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 0x7fe6c8199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 -- 172.21.15.22:0/456423576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 msgr2=0x7fe6c8199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 0x7fe6c8199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 -- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 msgr2=0x7fe6c81a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 0x7fe6c81a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 -- 172.21.15.22:0/456423576 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b8009490 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6bfe00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 0x7fe6c8199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.360+0000 7fe6c6a00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800ea60 tx=0x7fe6b800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.361+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800cde0 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.361+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b0009490 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.361+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b80035f0 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.361+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8010690 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.361+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c81a1d10 con 0x7fe6c8102130 2024-03-24T06:37:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.362+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6c8108e80 con 0x7fe6c8102130 2024-03-24T06:37:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.364+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe6b8003af0 con 0x7fe6c8102130 2024-03-24T06:37:15.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.365+0000 7fe6bf400640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 0x7fe6ac0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.366+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe6b8014070 con 0x7fe6c8102130 2024-03-24T06:37:15.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.366+0000 7fe6bfe00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 0x7fe6ac0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.367+0000 7fe6bfe00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 0x7fe6ac0a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0034b10 tx=0x7fe6b0034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.370+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe6b80970c0 con 0x7fe6c8102130 2024-03-24T06:37:15.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:15 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/543623164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:15.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.600+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6c802d160 con 0x7fe6c8102130 2024-03-24T06:37:15.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.602+0000 7fe6bf400640 1 -- 172.21.15.22:0/456423576 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fe6b809a780 con 0x7fe6c8102130 2024-03-24T06:37:15.601 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:37:15.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.604+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 msgr2=0x7fe6ac0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.604+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 0x7fe6ac0a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0034b10 tx=0x7fe6b0034be0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.604+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 msgr2=0x7fe6c81998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.604+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800ea60 tx=0x7fe6b800ef20 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 shutdown_connections 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6ac0a20c0 0x7fe6ac0a4580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8107390 0x7fe6c81a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6c8104a60 0x7fe6c8199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 --2- 172.21.15.22:0/456423576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6c8102130 0x7fe6c81998c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 >> 172.21.15.22:0/456423576 conn(0x7fe6c80fba80 msgr2=0x7fe6c81018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 shutdown_connections 2024-03-24T06:37:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.605+0000 7fe6c7e00640 1 -- 172.21.15.22:0/456423576 wait complete. 2024-03-24T06:37:15.622 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:37:15.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:15 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/543623164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:15.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:15 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/543623164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 -- 172.21.15.22:0/2692671831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868104b20 msgr2=0x7f8868104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 --2- 172.21.15.22:0/2692671831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868104b20 0x7f8868104fa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8858009ca0 tx=0x7f885802c100 comp rx=0 tx=0).stop 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 -- 172.21.15.22:0/2692671831 shutdown_connections 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 --2- 172.21.15.22:0/2692671831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f886810bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 --2- 172.21.15.22:0/2692671831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868104b20 0x7f8868104fa0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 --2- 172.21.15.22:0/2692671831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868103930 0x7f8868103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 -- 172.21.15.22:0/2692671831 >> 172.21.15.22:0/2692671831 conn(0x7f88680feec0 msgr2=0x7f88681012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 -- 172.21.15.22:0/2692671831 shutdown_connections 2024-03-24T06:37:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.834+0000 7f886f800640 1 -- 172.21.15.22:0/2692671831 wait complete. 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.835+0000 7f886f800640 1 Processor -- start 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.835+0000 7f886f800640 1 -- start start 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.835+0000 7f886f800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886f800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868104b20 0x7f8868199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886f800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f88681a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886f800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f88681a14c0 con 0x7f8868103930 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886f800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f88681a1630 con 0x7f88681054e0 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886f800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f88681a17f0 con 0x7f8868104b20 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51628/0 (socket says 172.21.15.22:51628) 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 -- 172.21.15.22:0/1434366262 learned_addr learned my addr 172.21.15.22:0/1434366262 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886d800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f88681a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f8867e00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868104b20 0x7f8868199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 -- 172.21.15.22:0/1434366262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868104b20 msgr2=0x7f8868199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868104b20 0x7f8868199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 -- 172.21.15.22:0/1434366262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 msgr2=0x7f88681a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f88681a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886ce00640 1 -- 172.21.15.22:0/1434366262 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8858009900 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.836+0000 7f886d800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f88681a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f886ce00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f885000e9c0 tx=0x7f885000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88500096e0 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f885000cd80 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850010430 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88681a1ad0 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.837+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88681a1f60 con 0x7f8868103930 2024-03-24T06:37:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.839+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886810ac50 con 0x7f8868103930 2024-03-24T06:37:15.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.841+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8850003680 con 0x7f8868103930 2024-03-24T06:37:15.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.842+0000 7f8865600640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 0x7f884c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:15.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.843+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8850014070 con 0x7f8868103930 2024-03-24T06:37:15.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.843+0000 7f8867e00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 0x7f884c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:15.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.844+0000 7f8867e00640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 0x7f884c0a4580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f885802c710 tx=0x7f88580324e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:15.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:15.845+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8850097060 con 0x7f8868103930 2024-03-24T06:37:16.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.048+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f886810f5e0 con 0x7f8868103930 2024-03-24T06:37:16.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.049+0000 7f8865600640 1 -- 172.21.15.22:0/1434366262 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f886810f5e0 con 0x7f8868103930 2024-03-24T06:37:16.049 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.051+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 msgr2=0x7f884c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.051+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 0x7f884c0a4580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f885802c710 tx=0x7f88580324e0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.051+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 msgr2=0x7f88681999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.051+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f885000e9c0 tx=0x7f885000ee80 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 shutdown_connections 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f884c0a20c0 0x7f884c0a4580 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88681054e0 0x7f88681a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8868104b20 0x7f8868199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 --2- 172.21.15.22:0/1434366262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8868103930 0x7f88681999d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 >> 172.21.15.22:0/1434366262 conn(0x7f88680feec0 msgr2=0x7f8868107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 shutdown_connections 2024-03-24T06:37:16.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.052+0000 7f886f800640 1 -- 172.21.15.22:0/1434366262 wait complete. 2024-03-24T06:37:16.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:37:16.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 -- 172.21.15.22:0/3470539025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe090101800 msgr2=0x7fe090101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 --2- 172.21.15.22:0/3470539025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe090101800 0x7fe090101c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009900 tx=0x7fe07c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:37:16.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 -- 172.21.15.22:0/3470539025 shutdown_connections 2024-03-24T06:37:16.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 --2- 172.21.15.22:0/3470539025 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe0901021c0 0x7fe090108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 --2- 172.21.15.22:0/3470539025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe090101800 0x7fe090101c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 --2- 172.21.15.22:0/3470539025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090100610 0x7fe090100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 -- 172.21.15.22:0/3470539025 >> 172.21.15.22:0/3470539025 conn(0x7fe0900fbb80 msgr2=0x7fe0900fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 -- 172.21.15.22:0/3470539025 shutdown_connections 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.282+0000 7fe097000640 1 -- 172.21.15.22:0/3470539025 wait complete. 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.284+0000 7fe097000640 1 Processor -- start 2024-03-24T06:37:16.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.284+0000 7fe097000640 1 -- start start 2024-03-24T06:37:16.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.284+0000 7fe097000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe090100610 0x7fe0901966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.284+0000 7fe097000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe097000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 0x7fe09019dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe097000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe09019e190 con 0x7fe090100610 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe097000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe090069ce0 con 0x7fe0901021c0 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe097000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe090069ea0 con 0x7fe090101800 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe08f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe08f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39812/0 (socket says 172.21.15.22:39812) 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe08f400640 1 -- 172.21.15.22:0/4131729502 learned_addr learned my addr 172.21.15.22:0/4131729502 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe095000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 0x7fe09019dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.285+0000 7fe08fe00640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe090100610 0x7fe0901966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 -- 172.21.15.22:0/4131729502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 msgr2=0x7fe09019dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 0x7fe09019dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 -- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe090100610 msgr2=0x7fe0901966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe090100610 0x7fe0901966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 -- 172.21.15.22:0/4131729502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe080009490 con 0x7fe090101800 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe095000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 0x7fe09019dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:16.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.286+0000 7fe08f400640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c6b0 tx=0x7fe07c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:16.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.288+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0044e0 con 0x7fe090101800 2024-03-24T06:37:16.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.288+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07c004ac0 con 0x7fe090101800 2024-03-24T06:37:16.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.288+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c03e450 con 0x7fe090101800 2024-03-24T06:37:16.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.288+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07c009560 con 0x7fe090101800 2024-03-24T06:37:16.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.289+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09006a4f0 con 0x7fe090101800 2024-03-24T06:37:16.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.290+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe090068950 con 0x7fe090101800 2024-03-24T06:37:16.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.292+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe07c0035f0 con 0x7fe090101800 2024-03-24T06:37:16.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.294+0000 7fe08cc00640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 0x7fe0740a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.294+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe07c0f07b0 con 0x7fe090101800 2024-03-24T06:37:16.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.294+0000 7fe08fe00640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 0x7fe0740a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.295+0000 7fe08fe00640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 0x7fe0740a4600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe090101660 tx=0x7fe080005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:16.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.298+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe07c0b94e0 con 0x7fe090101800 2024-03-24T06:37:16.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:16 smithi022 ceph-mon[32003]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:16.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:16 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/456423576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:16.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:16 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1434366262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:16.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.505+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe09010c220 con 0x7fe090101800 2024-03-24T06:37:16.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.506+0000 7fe08cc00640 1 -- 172.21.15.22:0/4131729502 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fe07c043020 con 0x7fe090101800 2024-03-24T06:37:16.506 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:37:16.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.509+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 msgr2=0x7fe0740a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.509+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 0x7fe0740a4600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe090101660 tx=0x7fe080005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.509+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 msgr2=0x7fe090196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.509+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c6b0 tx=0x7fe07c0042f0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 shutdown_connections 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0740a2140 0x7fe0740a4600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0901021c0 0x7fe09019dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090101800 0x7fe090196be0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 --2- 172.21.15.22:0/4131729502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe090100610 0x7fe0901966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 >> 172.21.15.22:0/4131729502 conn(0x7fe0900fbb80 msgr2=0x7fe090104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 shutdown_connections 2024-03-24T06:37:16.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.510+0000 7fe097000640 1 -- 172.21.15.22:0/4131729502 wait complete. 2024-03-24T06:37:16.529 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:37:16.579 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:16 smithi143 ceph-mon[36520]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:16.579 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:16 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/456423576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:16.579 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:16 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1434366262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 -- 172.21.15.22:0/3270298539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a0105180 msgr2=0x7fd9a0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 --2- 172.21.15.22:0/3270298539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a0105180 0x7fd9a0105600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd998009ca0 tx=0x7fd99802c100 comp rx=0 tx=0).stop 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 -- 172.21.15.22:0/3270298539 shutdown_connections 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 --2- 172.21.15.22:0/3270298539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a0105180 0x7fd9a0105600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 --2- 172.21.15.22:0/3270298539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 0x7fd9a0104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 --2- 172.21.15.22:0/3270298539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a00ffe60 0x7fd9a0102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 -- 172.21.15.22:0/3270298539 >> 172.21.15.22:0/3270298539 conn(0x7fd9a00fba80 msgr2=0x7fd9a00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:16.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.735+0000 7fd9a6000640 1 -- 172.21.15.22:0/3270298539 shutdown_connections 2024-03-24T06:37:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.736+0000 7fd9a6000640 1 -- 172.21.15.22:0/3270298539 wait complete. 2024-03-24T06:37:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.736+0000 7fd9a6000640 1 Processor -- start 2024-03-24T06:37:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.736+0000 7fd9a6000640 1 -- start start 2024-03-24T06:37:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 0x7fd9a019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0105180 0x7fd9a01a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01a5700 con 0x7fd9a00ffe60 2024-03-24T06:37:16.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01a5870 con 0x7fd9a0105180 2024-03-24T06:37:16.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a6000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a01a5a30 con 0x7fd9a01027d0 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a4c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a4c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51670/0 (socket says 172.21.15.22:51670) 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a4c00640 1 -- 172.21.15.22:0/312150250 learned_addr learned my addr 172.21.15.22:0/312150250 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd99fe00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 0x7fd9a019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.737+0000 7fd9a5600640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0105180 0x7fd9a01a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 -- 172.21.15.22:0/312150250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 msgr2=0x7fd9a019e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 0x7fd9a019e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 -- 172.21.15.22:0/312150250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0105180 msgr2=0x7fd9a01a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0105180 0x7fd9a01a5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 -- 172.21.15.22:0/312150250 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c009490 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a4c00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e9f0 tx=0x7fd99400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994009700 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99400cd80 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994018920 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd998009900 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.738+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a01a6090 con 0x7fd9a00ffe60 2024-03-24T06:37:16.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.740+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a00fd570 con 0x7fd9a00ffe60 2024-03-24T06:37:16.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.742+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd994018a80 con 0x7fd9a00ffe60 2024-03-24T06:37:16.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.744+0000 7fd99d600640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 0x7fd9880a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:16.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.744+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd994014070 con 0x7fd9a00ffe60 2024-03-24T06:37:16.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.745+0000 7fd99fe00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 0x7fd9880a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:16.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.745+0000 7fd99fe00640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 0x7fd9880a4580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009d60 tx=0x7fd98c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:16.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:16.747+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd994053280 con 0x7fd9a00ffe60 2024-03-24T06:37:16.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:16 smithi155 ceph-mon[36262]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:16.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:16 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/456423576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:16.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:16 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1434366262' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.046+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9a002d130 con 0x7fd9a00ffe60 2024-03-24T06:37:17.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.047+0000 7fd99d600640 1 -- 172.21.15.22:0/312150250 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fd994097090 con 0x7fd9a00ffe60 2024-03-24T06:37:17.047 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 msgr2=0x7fd9880a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 0x7fd9880a4580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009d60 tx=0x7fd98c005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 msgr2=0x7fd9a019dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e9f0 tx=0x7fd99400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 shutdown_connections 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd9880a20c0 0x7fd9880a4580 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9a0105180 0x7fd9a01a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9a01027d0 0x7fd9a019e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 --2- 172.21.15.22:0/312150250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9a00ffe60 0x7fd9a019dc20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 >> 172.21.15.22:0/312150250 conn(0x7fd9a00fba80 msgr2=0x7fd9a01007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:17.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 shutdown_connections 2024-03-24T06:37:17.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.050+0000 7fd9a6000640 1 -- 172.21.15.22:0/312150250 wait complete. 2024-03-24T06:37:17.076 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 -- 172.21.15.22:0/1736588259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc103930 msgr2=0x7f1ecc103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/1736588259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc103930 0x7f1ecc103d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009900 tx=0x7f1ebc02bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 -- 172.21.15.22:0/1736588259 shutdown_connections 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/1736588259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc1054e0 0x7f1ecc10bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/1736588259 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/1736588259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc103930 0x7f1ecc103d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 -- 172.21.15.22:0/1736588259 >> 172.21.15.22:0/1736588259 conn(0x7f1ecc0feec0 msgr2=0x7f1ecc1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 -- 172.21.15.22:0/1736588259 shutdown_connections 2024-03-24T06:37:17.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.292+0000 7f1ed2c00640 1 -- 172.21.15.22:0/1736588259 wait complete. 2024-03-24T06:37:17.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.293+0000 7f1ed2c00640 1 Processor -- start 2024-03-24T06:37:17.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.293+0000 7f1ed2c00640 1 -- start start 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.293+0000 7f1ed2c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc103930 0x7f1ecc199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed2c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed2c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc1054e0 0x7f1ecc1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed2c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc1a1510 con 0x7f1ecc104b20 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed2c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc1a1680 con 0x7f1ecc103930 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed2c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1ecc1a1840 con 0x7f1ecc1054e0 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc103930 0x7f1ecc199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51690/0 (socket says 172.21.15.22:51690) 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 -- 172.21.15.22:0/2970683508 learned_addr learned my addr 172.21.15.22:0/2970683508 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:17.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ed0c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc1054e0 0x7f1ecc1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 -- 172.21.15.22:0/2970683508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc1054e0 msgr2=0x7f1ecc1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc1054e0 0x7f1ecc1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 -- 172.21.15.22:0/2970683508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc103930 msgr2=0x7f1ecc199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc103930 0x7f1ecc199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.294+0000 7f1ecb400640 1 -- 172.21.15.22:0/2970683508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc009560 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ecb400640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9f0 tx=0x7f1eb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8009720 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb800cd80 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ecc1a1b20 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8010690 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.295+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ecc1a1fe0 con 0x7f1ecc104b20 2024-03-24T06:37:17.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.297+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ecc10aca0 con 0x7f1ecc104b20 2024-03-24T06:37:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.302+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1eb80046c0 con 0x7f1ecc104b20 2024-03-24T06:37:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.303+0000 7f1ec8c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 0x7f1eb00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.303+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1eb8014070 con 0x7f1ecc104b20 2024-03-24T06:37:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.303+0000 7f1ecbe00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 0x7f1eb00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.304+0000 7f1ecbe00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 0x7f1eb00a4580 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc02c240 tx=0x7f1ebc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:17.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.311+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1eb8096fe0 con 0x7f1ecc104b20 2024-03-24T06:37:17.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:17 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4131729502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:17 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/312150250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.579+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1ecc10f370 con 0x7f1ecc104b20 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.580+0000 7f1ec8c00640 1 -- 172.21.15.22:0/2970683508 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7f1eb809a6a0 con 0x7f1ecc104b20 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/896325545 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2695260292 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:17.580 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2447624484 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1811358368 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/3441957019 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1555230471 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:17.581 INFO:teuthology.orchestra.run.smithi022.stderr:listed 10 entries 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.582+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 msgr2=0x7f1eb00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.582+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 0x7f1eb00a4580 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc02c240 tx=0x7f1ebc02c7e0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 msgr2=0x7f1ecc199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800e9f0 tx=0x7f1eb800edf0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 shutdown_connections 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1eb00a20c0 0x7f1eb00a4580 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ecc1054e0 0x7f1ecc1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ecc104b20 0x7f1ecc199f60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 --2- 172.21.15.22:0/2970683508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ecc103930 0x7f1ecc199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 >> 172.21.15.22:0/2970683508 conn(0x7f1ecc0feec0 msgr2=0x7f1ecc107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 shutdown_connections 2024-03-24T06:37:17.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.583+0000 7f1ed2c00640 1 -- 172.21.15.22:0/2970683508 wait complete. 2024-03-24T06:37:17.606 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:37:17.606 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T06:37:17.655 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:17.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:17 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4131729502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:17 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/312150250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:17 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4131729502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:17 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/312150250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 -- 172.21.15.22:0/2796447146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574103950 msgr2=0x7fc574103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 --2- 172.21.15.22:0/2796447146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574103950 0x7fc574103d70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc564009900 tx=0x7fc56402bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 -- 172.21.15.22:0/2796447146 shutdown_connections 2024-03-24T06:37:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 --2- 172.21.15.22:0/2796447146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574105500 0x7fc57410bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 --2- 172.21.15.22:0/2796447146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574104b40 0x7fc574104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 --2- 172.21.15.22:0/2796447146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574103950 0x7fc574103d70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 -- 172.21.15.22:0/2796447146 >> 172.21.15.22:0/2796447146 conn(0x7fc5740fef00 msgr2=0x7fc574101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 -- 172.21.15.22:0/2796447146 shutdown_connections 2024-03-24T06:37:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.914+0000 7fc579000640 1 -- 172.21.15.22:0/2796447146 wait complete. 2024-03-24T06:37:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.915+0000 7fc579000640 1 Processor -- start 2024-03-24T06:37:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.915+0000 7fc579000640 1 -- start start 2024-03-24T06:37:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574104b40 0x7fc574199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 0x7fc5741a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc5741a14c0 con 0x7fc574105500 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc5741a1630 con 0x7fc574103950 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc579000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc5741a17f0 con 0x7fc574104b40 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc572800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 0x7fc5741a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44596/0 (socket says 172.21.15.22:44596) 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 -- 172.21.15.22:0/1748469128 learned_addr learned my addr 172.21.15.22:0/1748469128 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc569400640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574104b40 0x7fc574199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 -- 172.21.15.22:0/1748469128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574104b40 msgr2=0x7fc574199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574104b40 0x7fc574199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 -- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 msgr2=0x7fc5741a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 0x7fc5741a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:17.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc571e00640 1 -- 172.21.15.22:0/1748469128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc564009560 con 0x7fc574103950 2024-03-24T06:37:17.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.916+0000 7fc572800640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 0x7fc5741a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:17.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc571e00640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5640043d0 con 0x7fc574103950 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5640052d0 con 0x7fc574103950 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5741a1a70 con 0x7fc574103950 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564003d30 con 0x7fc574103950 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.917+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741a1f80 con 0x7fc574103950 2024-03-24T06:37:17.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.918+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57410ac70 con 0x7fc574103950 2024-03-24T06:37:17.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.920+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fc564005440 con 0x7fc574103950 2024-03-24T06:37:17.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.923+0000 7fc56b400640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 0x7fc5540a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:17.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.923+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc5640f0ae0 con 0x7fc574103950 2024-03-24T06:37:17.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.923+0000 7fc569400640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 0x7fc5540a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:17.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.923+0000 7fc569400640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 0x7fc5540a4580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc57419aee0 tx=0x7fc55c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:17.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:17.924+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5640b9810 con 0x7fc574103950 2024-03-24T06:37:18.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.127+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc57410f610 con 0x7fc574103950 2024-03-24T06:37:18.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.129+0000 7fc56b400640 1 -- 172.21.15.22:0/1748469128 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7fc5640bced0 con 0x7fc574103950 2024-03-24T06:37:18.129 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:18.129 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:18.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 msgr2=0x7fc5540a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 0x7fc5540a4580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc57419aee0 tx=0x7fc55c005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:18.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 msgr2=0x7fc5741999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:18.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 shutdown_connections 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5540a20c0 0x7fc5540a4580 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc574105500 0x7fc5741a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc574104b40 0x7fc574199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 --2- 172.21.15.22:0/1748469128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc574103950 0x7fc5741999d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 >> 172.21.15.22:0/1748469128 conn(0x7fc5740fef00 msgr2=0x7fc574107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 shutdown_connections 2024-03-24T06:37:18.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.132+0000 7fc579000640 1 -- 172.21.15.22:0/1748469128 wait complete. 2024-03-24T06:37:18.151 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:37:18.151 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 -- 172.21.15.22:0/3891715796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c60108f20 msgr2=0x7f2c601093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 --2- 172.21.15.22:0/3891715796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c60108f20 0x7f2c601093a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009900 tx=0x7f2c4c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 -- 172.21.15.22:0/3891715796 shutdown_connections 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 --2- 172.21.15.22:0/3891715796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c601098e0 0x7f2c60110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 --2- 172.21.15.22:0/3891715796 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c60108f20 0x7f2c601093a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 --2- 172.21.15.22:0/3891715796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 0x7f2c60108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.378+0000 7f2c65200640 1 -- 172.21.15.22:0/3891715796 >> 172.21.15.22:0/3891715796 conn(0x7f2c60076cd0 msgr2=0x7f2c60079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.379+0000 7f2c65200640 1 -- 172.21.15.22:0/3891715796 shutdown_connections 2024-03-24T06:37:18.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.379+0000 7f2c65200640 1 -- 172.21.15.22:0/3891715796 wait complete. 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 Processor -- start 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 -- start start 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 0x7f2c6019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 0x7f2c6019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601a58e0 con 0x7f2c601098e0 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601a5a50 con 0x7f2c60108f20 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c65200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2c601a5c10 con 0x7f2c60107d30 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c5e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c5d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 0x7f2c6019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c5e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51728/0 (socket says 172.21.15.22:51728) 2024-03-24T06:37:18.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c5e800640 1 -- 172.21.15.22:0/1944887715 learned_addr learned my addr 172.21.15.22:0/1944887715 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.380+0000 7f2c5de00640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 0x7f2c6019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 -- 172.21.15.22:0/1944887715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 msgr2=0x7f2c6019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 0x7f2c6019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 -- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 msgr2=0x7f2c601a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 -- 172.21.15.22:0/1944887715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c009560 con 0x7f2c60108f20 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5e800640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:18.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.381+0000 7f2c5d400640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 0x7f2c6019e330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c02c180 tx=0x7f2c4c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.382+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0043d0 con 0x7f2c60108f20 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.382+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c0052d0 con 0x7f2c60108f20 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.382+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c003d40 con 0x7f2c60108f20 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.382+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c601a5e90 con 0x7f2c60108f20 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.382+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c601a62c0 con 0x7f2c60108f20 2024-03-24T06:37:18.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.383+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6010f050 con 0x7f2c60108f20 2024-03-24T06:37:18.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.387+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2c4c005440 con 0x7f2c60108f20 2024-03-24T06:37:18.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.388+0000 7f2c56a00640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 0x7f2c440a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.389+0000 7f2c5de00640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 0x7f2c440a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.389+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2c4c0f0790 con 0x7f2c60108f20 2024-03-24T06:37:18.389 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.389+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c4c0bcff0 con 0x7f2c60108f20 2024-03-24T06:37:18.389 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.389+0000 7f2c5de00640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 0x7f2c440a4600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2c60108d80 tx=0x7f2c50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:18.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:18 smithi022 ceph-mon[32003]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:18.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:18 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2970683508' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:18.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:18 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1748469128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:18.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.595+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2c60113910 con 0x7f2c60108f20 2024-03-24T06:37:18.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.596+0000 7f2c56a00640 1 -- 172.21.15.22:0/1944887715 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f2c4c043020 con 0x7f2c60108f20 2024-03-24T06:37:18.596 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 msgr2=0x7f2c440a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 0x7f2c440a4600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2c60108d80 tx=0x7f2c50005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 msgr2=0x7f2c6019e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 0x7f2c6019e330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c02c180 tx=0x7f2c4c005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 shutdown_connections 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2c440a2140 0x7f2c440a4600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2c601098e0 0x7f2c601a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c60108f20 0x7f2c6019e330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 --2- 172.21.15.22:0/1944887715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c60107d30 0x7f2c6019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 >> 172.21.15.22:0/1944887715 conn(0x7f2c60076cd0 msgr2=0x7f2c6010c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:18.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 shutdown_connections 2024-03-24T06:37:18.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.599+0000 7f2c65200640 1 -- 172.21.15.22:0/1944887715 wait complete. 2024-03-24T06:37:18.623 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:37:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:18 smithi143 ceph-mon[36520]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:18 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2970683508' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:18 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1748469128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:18 smithi155 ceph-mon[36262]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-24T06:37:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:18 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2970683508' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:18 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1748469128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:18.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 -- 172.21.15.22:0/2902932856 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 msgr2=0x7ff0d0101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/2902932856 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d0101a30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009900 tx=0x7ff0c002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:18.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 -- 172.21.15.22:0/2902932856 shutdown_connections 2024-03-24T06:37:18.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/2902932856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/2902932856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/2902932856 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d0101a30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 -- 172.21.15.22:0/2902932856 >> 172.21.15.22:0/2902932856 conn(0x7ff0d00fcb80 msgr2=0x7ff0d00fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:18.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 -- 172.21.15.22:0/2902932856 shutdown_connections 2024-03-24T06:37:18.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.858+0000 7ff0d4a00640 1 -- 172.21.15.22:0/2902932856 wait complete. 2024-03-24T06:37:18.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.859+0000 7ff0d4a00640 1 Processor -- start 2024-03-24T06:37:18.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.859+0000 7ff0d4a00640 1 -- start start 2024-03-24T06:37:18.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d00747a0 con 0x7ff0d0101610 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d0074910 con 0x7ff0d0102800 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0d4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d01a1710 con 0x7ff0d01031c0 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0ce800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0ce800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51746/0 (socket says 172.21.15.22:51746) 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0ce800640 1 -- 172.21.15.22:0/1506700361 learned_addr learned my addr 172.21.15.22:0/1506700361 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0c7e00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.860+0000 7ff0cf200640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 -- 172.21.15.22:0/1506700361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 msgr2=0x7ff0d0074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 -- 172.21.15.22:0/1506700361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 msgr2=0x7ff0d0073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 -- 172.21.15.22:0/1506700361 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b8009490 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0cf200640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0c7e00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0ce800640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000c00 tx=0x7ff0c0005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c00043d0 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0c00052d0 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0003d70 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.861+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c0009560 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.862+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d01a1ce0 con 0x7ff0d0101610 2024-03-24T06:37:18.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.863+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d01089a0 con 0x7ff0d0101610 2024-03-24T06:37:18.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.865+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff0c0005440 con 0x7ff0d0101610 2024-03-24T06:37:18.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.866+0000 7ff0c7400640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 0x7ff0b40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:18.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.866+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff0c00f06e0 con 0x7ff0d0101610 2024-03-24T06:37:18.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.866+0000 7ff0c7e00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 0x7ff0b40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:18.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.867+0000 7ff0c7e00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 0x7ff0b40a4580 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0075230 tx=0x7ff0b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:18.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:18.870+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff0c00b9410 con 0x7ff0d0101610 2024-03-24T06:37:19.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.100+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff0d010d020 con 0x7ff0d0101610 2024-03-24T06:37:19.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.101+0000 7ff0c7400640 1 -- 172.21.15.22:0/1506700361 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ff0c00bcad0 con 0x7ff0d0101610 2024-03-24T06:37:19.101 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:37:19.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 msgr2=0x7ff0b40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 0x7ff0b40a4580 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0075230 tx=0x7ff0b8005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:19.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 msgr2=0x7ff0d00754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0000c00 tx=0x7ff0c0005e50 comp rx=0 tx=0).stop 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 shutdown_connections 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff0b40a20c0 0x7ff0b40a4580 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0d01031c0 0x7ff0d0074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.103+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d0102800 0x7ff0d0073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.104+0000 7ff0d4a00640 1 --2- 172.21.15.22:0/1506700361 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff0d0101610 0x7ff0d00754c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.104+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 >> 172.21.15.22:0/1506700361 conn(0x7ff0d00fcb80 msgr2=0x7ff0d0105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.104+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 shutdown_connections 2024-03-24T06:37:19.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.104+0000 7ff0d4a00640 1 -- 172.21.15.22:0/1506700361 wait complete. 2024-03-24T06:37:19.123 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:37:19.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 -- 172.21.15.22:0/929639698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70108e20 msgr2=0x7fed701092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 --2- 172.21.15.22:0/929639698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70108e20 0x7fed701092a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009860 tx=0x7fed5c02bb70 comp rx=0 tx=0).stop 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 -- 172.21.15.22:0/929639698 shutdown_connections 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 --2- 172.21.15.22:0/929639698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed701097e0 0x7fed70110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 --2- 172.21.15.22:0/929639698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70108e20 0x7fed701092a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 --2- 172.21.15.22:0/929639698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed70107c30 0x7fed70108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 -- 172.21.15.22:0/929639698 >> 172.21.15.22:0/929639698 conn(0x7fed70076be0 msgr2=0x7fed70079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 -- 172.21.15.22:0/929639698 shutdown_connections 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.359+0000 7fed77000640 1 -- 172.21.15.22:0/929639698 wait complete. 2024-03-24T06:37:19.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.361+0000 7fed77000640 1 Processor -- start 2024-03-24T06:37:19.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.361+0000 7fed77000640 1 -- start start 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.361+0000 7fed77000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed77000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed70108e20 0x7fed7019e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed77000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 0x7fed701a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed77000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fed701a5740 con 0x7fed70108e20 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed77000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fed701a58b0 con 0x7fed70107c30 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed77000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fed701a5a70 con 0x7fed701097e0 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed70108e20 0x7fed7019e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed76600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 0x7fed701a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44650/0 (socket says 172.21.15.22:44650) 2024-03-24T06:37:19.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 -- 172.21.15.22:0/3671583284 learned_addr learned my addr 172.21.15.22:0/3671583284 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:19.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed76600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 0x7fed701a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39900/0 (socket says 172.21.15.22:39900) 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 -- 172.21.15.22:0/3671583284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 msgr2=0x7fed701a5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 0x7fed701a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 -- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed70108e20 msgr2=0x7fed7019e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed70108e20 0x7fed7019e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.362+0000 7fed75c00640 1 -- 172.21.15.22:0/3671583284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c009490 con 0x7fed70107c30 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed75c00640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6800ccc0 con 0x7fed70107c30 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed6800ce20 con 0x7fed70107c30 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed701a5d50 con 0x7fed70107c30 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68010430 con 0x7fed70107c30 2024-03-24T06:37:19.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.363+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed701a61b0 con 0x7fed70107c30 2024-03-24T06:37:19.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.365+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed7010ef50 con 0x7fed70107c30 2024-03-24T06:37:19.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.368+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fed68003fb0 con 0x7fed70107c30 2024-03-24T06:37:19.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.369+0000 7fed6e000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 0x7fed540a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.369+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed68014070 con 0x7fed70107c30 2024-03-24T06:37:19.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.369+0000 7fed75200640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 0x7fed540a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.370+0000 7fed75200640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 0x7fed540a4580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009830 tx=0x7fed5c034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:19.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.374+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed68096ec0 con 0x7fed70107c30 2024-03-24T06:37:19.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:19 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1944887715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:19 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1506700361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.576+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed70113610 con 0x7fed70107c30 2024-03-24T06:37:19.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.577+0000 7fed6e000640 1 -- 172.21.15.22:0/3671583284 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fed6809a580 con 0x7fed70107c30 2024-03-24T06:37:19.576 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:37:19.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 msgr2=0x7fed540a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 0x7fed540a4580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009830 tx=0x7fed5c034be0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 msgr2=0x7fed7019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 shutdown_connections 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fed540a20c0 0x7fed540a4580 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed701097e0 0x7fed701a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fed70108e20 0x7fed7019e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 --2- 172.21.15.22:0/3671583284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70107c30 0x7fed7019dd40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 >> 172.21.15.22:0/3671583284 conn(0x7fed70076be0 msgr2=0x7fed7010c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.579+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 shutdown_connections 2024-03-24T06:37:19.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.580+0000 7fed77000640 1 -- 172.21.15.22:0/3671583284 wait complete. 2024-03-24T06:37:19.599 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:37:19.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:19 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1944887715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:19 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1506700361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:19 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1944887715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:19 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1506700361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:37:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 -- 172.21.15.22:0/4044033959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe84100500 msgr2=0x7efe84100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 --2- 172.21.15.22:0/4044033959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe84100500 0x7efe84100920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe74009ca0 tx=0x7efe7402c100 comp rx=0 tx=0).stop 2024-03-24T06:37:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 -- 172.21.15.22:0/4044033959 shutdown_connections 2024-03-24T06:37:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 --2- 172.21.15.22:0/4044033959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841020b0 0x7efe84108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 --2- 172.21.15.22:0/4044033959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe841016f0 0x7efe84101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 --2- 172.21.15.22:0/4044033959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe84100500 0x7efe84100920 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 -- 172.21.15.22:0/4044033959 >> 172.21.15.22:0/4044033959 conn(0x7efe840fba90 msgr2=0x7efe840fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 -- 172.21.15.22:0/4044033959 shutdown_connections 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.819+0000 7efe83200640 1 -- 172.21.15.22:0/4044033959 wait complete. 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 Processor -- start 2024-03-24T06:37:19.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 -- start start 2024-03-24T06:37:19.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe84100500 0x7efe8406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efe8419f1b0 con 0x7efe841020b0 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efe8419f320 con 0x7efe84100500 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.821+0000 7efe83200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efe8419f4e0 con 0x7efe841016f0 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe79400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51778/0 (socket says 172.21.15.22:51778) 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 -- 172.21.15.22:0/3734744437 learned_addr learned my addr 172.21.15.22:0/3734744437 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe79400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39922/0 (socket says 172.21.15.22:39922) 2024-03-24T06:37:19.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe81e00640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe84100500 0x7efe8406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 -- 172.21.15.22:0/3734744437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 msgr2=0x7efe8419c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 -- 172.21.15.22:0/3734744437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe84100500 msgr2=0x7efe8406a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe84100500 0x7efe8406a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe82800640 1 -- 172.21.15.22:0/3734744437 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe74009900 con 0x7efe841020b0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.822+0000 7efe79400640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe82800640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00e970 tx=0x7efe7c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c009720 con 0x7efe841020b0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7c005500 con 0x7efe841020b0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8419f7c0 con 0x7efe841020b0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c010430 con 0x7efe841020b0 2024-03-24T06:37:19.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.823+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8419fc80 con 0x7efe841020b0 2024-03-24T06:37:19.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.825+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe84107820 con 0x7efe841020b0 2024-03-24T06:37:19.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.827+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7efe7c005020 con 0x7efe841020b0 2024-03-24T06:37:19.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.829+0000 7efe7b400640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:19.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.829+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efe7c014070 con 0x7efe841020b0 2024-03-24T06:37:19.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.829+0000 7efe81e00640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:19.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.830+0000 7efe81e00640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 0x7efe680a4580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe7402c640 tx=0x7efe74009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:19.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:19.834+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efe7c097000 con 0x7efe841020b0 2024-03-24T06:37:20.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.035+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe8402d130 con 0x7efe841020b0 2024-03-24T06:37:20.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.035+0000 7efe7b400640 1 -- 172.21.15.22:0/3734744437 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7efe7c09a6c0 con 0x7efe841020b0 2024-03-24T06:37:20.035 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:37:20.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 msgr2=0x7efe680a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 0x7efe680a4580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe7402c640 tx=0x7efe74009800 comp rx=0 tx=0).stop 2024-03-24T06:37:20.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 msgr2=0x7efe8419ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00e970 tx=0x7efe7c00ed70 comp rx=0 tx=0).stop 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 shutdown_connections 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efe841020b0 0x7efe8419ea20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe841016f0 0x7efe8419c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.038+0000 7efe83200640 1 --2- 172.21.15.22:0/3734744437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe84100500 0x7efe8406a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.039+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 >> 172.21.15.22:0/3734744437 conn(0x7efe840fba90 msgr2=0x7efe841049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.039+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 shutdown_connections 2024-03-24T06:37:20.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.039+0000 7efe83200640 1 -- 172.21.15.22:0/3734744437 wait complete. 2024-03-24T06:37:20.059 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 -- 172.21.15.22:0/3830628105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048104a00 msgr2=0x7f0048104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 --2- 172.21.15.22:0/3830628105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048104a00 0x7f0048104e80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0034009900 tx=0x7f003402bc50 comp rx=0 tx=0).stop 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 -- 172.21.15.22:0/3830628105 shutdown_connections 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 --2- 172.21.15.22:0/3830628105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00481053c0 0x7f004810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 --2- 172.21.15.22:0/3830628105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048104a00 0x7f0048104e80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 --2- 172.21.15.22:0/3830628105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0048103810 0x7f0048103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 -- 172.21.15.22:0/3830628105 >> 172.21.15.22:0/3830628105 conn(0x7f00480fed80 msgr2=0x7f00481011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 -- 172.21.15.22:0/3830628105 shutdown_connections 2024-03-24T06:37:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.277+0000 7f0047200640 1 -- 172.21.15.22:0/3830628105 wait complete. 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 Processor -- start 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 -- start start 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048103810 0x7f00481998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00481053c0 0x7f00481a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f00481a13d0 con 0x7f0048103810 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f00481a1540 con 0x7f0048104a00 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.278+0000 7f0047200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f00481a1700 con 0x7f00481053c0 2024-03-24T06:37:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0046800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00481053c0 0x7f00481a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44688/0 (socket says 172.21.15.22:44688) 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 -- 172.21.15.22:0/3002588917 learned_addr learned my addr 172.21.15.22:0/3002588917 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045e00640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048103810 0x7f00481998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 -- 172.21.15.22:0/3002588917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00481053c0 msgr2=0x7f00481a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00481053c0 0x7f00481a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 -- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048103810 msgr2=0x7f00481998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048103810 0x7f00481998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.279+0000 7f0045400640 1 -- 172.21.15.22:0/3002588917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0034009560 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f0045400640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f003402c6c0 tx=0x7f00340042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340044e0 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0034004ac0 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003403e450 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481a1980 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.280+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481a1e90 con 0x7f0048104a00 2024-03-24T06:37:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.281+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004810ab30 con 0x7f0048104a00 2024-03-24T06:37:20.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.283+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0034003680 con 0x7f0048104a00 2024-03-24T06:37:20.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.285+0000 7f003ea00640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 0x7f002c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:20.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.285+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f00340f0830 con 0x7f0048104a00 2024-03-24T06:37:20.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.285+0000 7f0045e00640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 0x7f002c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.286+0000 7f0045e00640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 0x7f002c0a4580 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0048104860 tx=0x7f0038005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:20.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.289+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00340b9680 con 0x7f0048104a00 2024-03-24T06:37:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:20 smithi022 ceph-mon[32003]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:20 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3671583284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:20 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3734744437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:20.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.540+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f004802d130 con 0x7f0048104a00 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.541+0000 7f003ea00640 1 -- 172.21.15.22:0/3002588917 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7f0034043080 con 0x7f0048104a00 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/896325545 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2695260292 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2447624484 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1811358368 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/3441957019 2024-03-25T06:28:20.396453+0000 2024-03-24T06:37:20.541 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1555230471 2024-03-25T06:28:54.941425+0000 2024-03-24T06:37:20.542 INFO:teuthology.orchestra.run.smithi022.stderr:listed 10 entries 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 msgr2=0x7f002c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 0x7f002c0a4580 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0048104860 tx=0x7f0038005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 msgr2=0x7f0048199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f003402c6c0 tx=0x7f00340042f0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 shutdown_connections 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f002c0a20c0 0x7f002c0a4580 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00481053c0 0x7f00481a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0048104a00 0x7f0048199e20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 --2- 172.21.15.22:0/3002588917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0048103810 0x7f00481998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:20.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 >> 172.21.15.22:0/3002588917 conn(0x7f00480fed80 msgr2=0x7f0048107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:20.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 shutdown_connections 2024-03-24T06:37:20.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:20.544+0000 7f0047200640 1 -- 172.21.15.22:0/3002588917 wait complete. 2024-03-24T06:37:20.561 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi022.front.sepia.ceph.com... 2024-03-24T06:37:20.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:20.562 DEBUG:teuthology.orchestra.run.smithi022:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-24T06:37:20.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi022.stderr:ceph-fuse[75698]: fuse finished with error 0 and tester_r 0 2024-03-24T06:37:20.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T06:37:20.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:20 smithi143 ceph-mon[36520]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:20.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:20 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3671583284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:20.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:20 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3734744437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:20.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:20 smithi155 ceph-mon[36262]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:20.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:20 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3671583284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:37:20.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:20 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3734744437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:37:21.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:21 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3002588917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:21 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3002588917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:21 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3002588917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:37:22.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:22 smithi022 ceph-mon[32003]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:22 smithi143 ceph-mon[36520]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:22.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:22 smithi155 ceph-mon[36262]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-24T06:37:24.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:24 smithi022 ceph-mon[32003]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T06:37:24.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:24 smithi143 ceph-mon[36520]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T06:37:24.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:24 smithi155 ceph-mon[36262]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T06:37:26.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:26 smithi022 ceph-mon[32003]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:26.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:26 smithi143 ceph-mon[36520]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:26.753 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:37:26.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:37:26.753 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-24T06:37:26.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:26 smithi155 ceph-mon[36262]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:26.828 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:27.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 -- 172.21.15.22:0/4141855410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 msgr2=0x7fe54010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 --2- 172.21.15.22:0/4141855410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe54010b8b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe538009be0 tx=0x7fe53802bf80 comp rx=0 tx=0).stop 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 -- 172.21.15.22:0/4141855410 shutdown_connections 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 --2- 172.21.15.22:0/4141855410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe54010b8b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 --2- 172.21.15.22:0/4141855410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400740f0 0x7fe540074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 --2- 172.21.15.22:0/4141855410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400756d0 0x7fe540073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.062+0000 7fe544a00640 1 -- 172.21.15.22:0/4141855410 >> 172.21.15.22:0/4141855410 conn(0x7fe5400fcae0 msgr2=0x7fe5400fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.063+0000 7fe544a00640 1 -- 172.21.15.22:0/4141855410 shutdown_connections 2024-03-24T06:37:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.063+0000 7fe544a00640 1 -- 172.21.15.22:0/4141855410 wait complete. 2024-03-24T06:37:27.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.064+0000 7fe544a00640 1 Processor -- start 2024-03-24T06:37:27.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 -- start start 2024-03-24T06:37:27.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 0x7fe540199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe5401a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe5401a1270 con 0x7fe5400756d0 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe5401a13e0 con 0x7fe540109420 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.065+0000 7fe544a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5401a15a0 con 0x7fe5400740f0 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe537e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 0x7fe540199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe537e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39926/0 (socket says 172.21.15.22:39926) 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe537e00640 1 -- 172.21.15.22:0/529640127 learned_addr learned my addr 172.21.15.22:0/529640127 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:27.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53fe00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe5401a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 -- 172.21.15.22:0/529640127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 msgr2=0x7fe5401a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe5401a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 -- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 msgr2=0x7fe540199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe53f400640 1 -- 172.21.15.22:0/529640127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe528009560 con 0x7fe5400740f0 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.066+0000 7fe537e00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:27.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe53f400640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 0x7fe540199780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe53000e9f0 tx=0x7fe53000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53000ccc0 con 0x7fe5400740f0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe53000ce20 con 0x7fe5400740f0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530004a00 con 0x7fe5400740f0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe538009840 con 0x7fe5400740f0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.067+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5401a1bb0 con 0x7fe5400740f0 2024-03-24T06:37:27.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.068+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5401081e0 con 0x7fe5400740f0 2024-03-24T06:37:27.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.070+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe5300050b0 con 0x7fe5400740f0 2024-03-24T06:37:27.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.072+0000 7fe53cc00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 0x7fe5240a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.072+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe530014070 con 0x7fe5400740f0 2024-03-24T06:37:27.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.072+0000 7fe537e00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 0x7fe5240a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.073+0000 7fe537e00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 0x7fe5240a4600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe54019ac90 tx=0x7fe528009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:27.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.076+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe530096b90 con 0x7fe5400740f0 2024-03-24T06:37:27.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.278+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe54010f1d0 con 0x7fe5400740f0 2024-03-24T06:37:27.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.281+0000 7fe53cc00640 1 -- 172.21.15.22:0/529640127 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7fe53009a250 con 0x7fe5400740f0 2024-03-24T06:37:27.281 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:27.281 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 msgr2=0x7fe5240a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 0x7fe5240a4600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe54019ac90 tx=0x7fe528009260 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 msgr2=0x7fe540199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 0x7fe540199780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe53000e9f0 tx=0x7fe53000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 shutdown_connections 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe5240a2140 0x7fe5240a4600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe540109420 0x7fe5401a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5400756d0 0x7fe540199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 --2- 172.21.15.22:0/529640127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5400740f0 0x7fe540199780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 >> 172.21.15.22:0/529640127 conn(0x7fe5400fcae0 msgr2=0x7fe5401053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 shutdown_connections 2024-03-24T06:37:27.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.284+0000 7fe544a00640 1 -- 172.21.15.22:0/529640127 wait complete. 2024-03-24T06:37:27.306 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.533+0000 7f3beb200640 1 -- 172.21.15.22:0/2053992724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec102010 msgr2=0x7f3bec1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.533+0000 7f3beb200640 1 --2- 172.21.15.22:0/2053992724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec102010 0x7f3bec1088a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009ca0 tx=0x7f3be402c100 comp rx=0 tx=0).stop 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 -- 172.21.15.22:0/2053992724 shutdown_connections 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 --2- 172.21.15.22:0/2053992724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec102010 0x7f3bec1088a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 --2- 172.21.15.22:0/2053992724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec101650 0x7f3bec101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 --2- 172.21.15.22:0/2053992724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 0x7f3bec100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 -- 172.21.15.22:0/2053992724 >> 172.21.15.22:0/2053992724 conn(0x7f3bec0fb9f0 msgr2=0x7f3bec0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 -- 172.21.15.22:0/2053992724 shutdown_connections 2024-03-24T06:37:27.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.534+0000 7f3beb200640 1 -- 172.21.15.22:0/2053992724 wait complete. 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 Processor -- start 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 -- start start 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 0x7f3bec197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec102010 0x7f3bec19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec19f150 con 0x7f3bec101650 2024-03-24T06:37:27.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec19f2c0 con 0x7f3bec102010 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.535+0000 7f3beb200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec19f480 con 0x7f3bec100460 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39934/0 (socket says 172.21.15.22:39934) 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 -- 172.21.15.22:0/1487380284 learned_addr learned my addr 172.21.15.22:0/1487380284 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9e00640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 0x7f3bec197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3bea800640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec102010 0x7f3bec19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 -- 172.21.15.22:0/1487380284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 msgr2=0x7f3bec197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 0x7f3bec197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 -- 172.21.15.22:0/1487380284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec102010 msgr2=0x7f3bec19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec102010 0x7f3bec19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.536+0000 7f3be9400640 1 -- 172.21.15.22:0/1487380284 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be4009900 con 0x7f3bec101650 2024-03-24T06:37:27.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3be9400640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d950 tx=0x7f3bd400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4009c30 con 0x7f3bec101650 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bec19f760 con 0x7f3bec101650 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd4009d90 con 0x7f3bec101650 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400f7e0 con 0x7f3bec101650 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.537+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bec19fc70 con 0x7f3bec101650 2024-03-24T06:37:27.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.538+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bec10be40 con 0x7f3bec101650 2024-03-24T06:37:27.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.540+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3bd4010ad0 con 0x7f3bec101650 2024-03-24T06:37:27.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.541+0000 7f3be2a00640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 0x7f3bd00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:27.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.542+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3bd40cdac0 con 0x7f3bec101650 2024-03-24T06:37:27.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.542+0000 7f3be9e00640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 0x7f3bd00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:27.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.543+0000 7f3be9e00640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 0x7f3bd00a4580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc005fd0 tx=0x7f3bdc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:27.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.546+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3bd40968a0 con 0x7f3bec101650 2024-03-24T06:37:27.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:27 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/529640127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:27 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/529640127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:27.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.747+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bec10c000 con 0x7f3bec101650 2024-03-24T06:37:27.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.749+0000 7f3be2a00640 1 -- 172.21.15.22:0/1487380284 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f3bd4096030 con 0x7f3bec101650 2024-03-24T06:37:27.750 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:27.750 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:27.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.752+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 msgr2=0x7f3bd00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.752+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 0x7f3bd00a4580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc005fd0 tx=0x7f3bdc00b410 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.752+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 msgr2=0x7f3bec197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.752+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d950 tx=0x7f3bd400de10 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.752+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 shutdown_connections 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3bd00a20c0 0x7f3bd00a4580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec102010 0x7f3bec19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bec101650 0x7f3bec197ba0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 --2- 172.21.15.22:0/1487380284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bec100460 0x7f3bec197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 >> 172.21.15.22:0/1487380284 conn(0x7f3bec0fb9f0 msgr2=0x7f3bec0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 shutdown_connections 2024-03-24T06:37:27.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:27.753+0000 7f3beb200640 1 -- 172.21.15.22:0/1487380284 wait complete. 2024-03-24T06:37:27.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T06:37:27.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:27 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/529640127' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 -- 172.21.15.22:0/2818775110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5064105180 msgr2=0x7f5064105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 --2- 172.21.15.22:0/2818775110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5064105180 0x7f5064105600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f505c009be0 tx=0x7f505c02bef0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 -- 172.21.15.22:0/2818775110 shutdown_connections 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 --2- 172.21.15.22:0/2818775110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5064105180 0x7f5064105600 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 --2- 172.21.15.22:0/2818775110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50641027d0 0x7f5064104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.003+0000 7f5063e00640 1 --2- 172.21.15.22:0/2818775110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50640ffe60 0x7f5064102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.004+0000 7f5063e00640 1 -- 172.21.15.22:0/2818775110 >> 172.21.15.22:0/2818775110 conn(0x7f50640fba80 msgr2=0x7f50640fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.004+0000 7f5063e00640 1 -- 172.21.15.22:0/2818775110 shutdown_connections 2024-03-24T06:37:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.004+0000 7f5063e00640 1 -- 172.21.15.22:0/2818775110 wait complete. 2024-03-24T06:37:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.005+0000 7f5063e00640 1 Processor -- start 2024-03-24T06:37:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.005+0000 7f5063e00640 1 -- start start 2024-03-24T06:37:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.005+0000 7f5063e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.005+0000 7f5063e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 0x7f506419e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.005+0000 7f5063e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064105180 0x7f50641a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5063e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f50641a56c0 con 0x7f50640ffe60 2024-03-24T06:37:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5063e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f50641a5830 con 0x7f5064105180 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5063e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f50641a59f0 con 0x7f50641027d0 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5062a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5062a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39952/0 (socket says 172.21.15.22:39952) 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5062a00640 1 -- 172.21.15.22:0/1927695086 learned_addr learned my addr 172.21.15.22:0/1927695086 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 0x7f506419e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5063400640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064105180 0x7f50641a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 -- 172.21.15.22:0/1927695086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064105180 msgr2=0x7f50641a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064105180 0x7f50641a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 -- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 msgr2=0x7f506419dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f505a000640 1 -- 172.21.15.22:0/1927695086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504c009560 con 0x7f50641027d0 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.006+0000 7f5062a00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f505a000640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 0x7f506419e110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f504c005fd0 tx=0x7f504c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c00cd00 con 0x7f50641027d0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f504c00ce60 con 0x7f50641027d0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505c009840 con 0x7f50641027d0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c010430 con 0x7f50641027d0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.007+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50641a60a0 con 0x7f50641027d0 2024-03-24T06:37:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.008+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50640fd570 con 0x7f50641027d0 2024-03-24T06:37:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.010+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f504c0035f0 con 0x7f50641027d0 2024-03-24T06:37:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.012+0000 7f505be00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 0x7f50480a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.012+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f504c014070 con 0x7f50641027d0 2024-03-24T06:37:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.012+0000 7f5062a00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 0x7f50480a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.013+0000 7f5062a00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 0x7f50480a4600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5054011500 tx=0x7f5054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.016+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f504c096fc0 con 0x7f50641027d0 2024-03-24T06:37:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.261+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f506402d160 con 0x7f50641027d0 2024-03-24T06:37:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.348+0000 7f505be00640 1 -- 172.21.15.22:0/1927695086 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f504c09a680 con 0x7f50641027d0 2024-03-24T06:37:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.352+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 msgr2=0x7f50480a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.352+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 0x7f50480a4600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5054011500 tx=0x7f5054005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:28.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.352+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 msgr2=0x7f506419e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.352+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 0x7f506419e110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f504c005fd0 tx=0x7f504c00ee50 comp rx=0 tx=0).stop 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 shutdown_connections 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50480a2140 0x7f50480a4600 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5064105180 0x7f50641a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50641027d0 0x7f506419e110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 --2- 172.21.15.22:0/1927695086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50640ffe60 0x7f506419dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 >> 172.21.15.22:0/1927695086 conn(0x7f50640fba80 msgr2=0x7f50641007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 shutdown_connections 2024-03-24T06:37:28.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.353+0000 7f5063e00640 1 -- 172.21.15.22:0/1927695086 wait complete. 2024-03-24T06:37:28.374 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-24T06:37:28.374 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.592+0000 7fb4a0e00640 1 -- 172.21.15.22:0/4237770792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0ffe60 msgr2=0x7fb49c100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.592+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/4237770792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0ffe60 0x7fb49c100280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009900 tx=0x7fb48c02bc50 comp rx=0 tx=0).stop 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 -- 172.21.15.22:0/4237770792 shutdown_connections 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/4237770792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/4237770792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c1007c0 0x7fb49c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/4237770792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c0ffe60 0x7fb49c100280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 -- 172.21.15.22:0/4237770792 >> 172.21.15.22:0/4237770792 conn(0x7fb49c0fba80 msgr2=0x7fb49c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 -- 172.21.15.22:0/4237770792 shutdown_connections 2024-03-24T06:37:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.593+0000 7fb4a0e00640 1 -- 172.21.15.22:0/4237770792 wait complete. 2024-03-24T06:37:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 Processor -- start 2024-03-24T06:37:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 -- start start 2024-03-24T06:37:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c0ffe60 0x7fb49c19dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 0x7fb49c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c1a5760 con 0x7fb49c101160 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c1a58d0 con 0x7fb49c1007c0 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.594+0000 7fb4a0e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c1a5a90 con 0x7fb49c0ffe60 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39982/0 (socket says 172.21.15.22:39982) 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 -- 172.21.15.22:0/1311247666 learned_addr learned my addr 172.21.15.22:0/1311247666 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb492a00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 0x7fb49c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49b400640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c0ffe60 0x7fb49c19dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 -- 172.21.15.22:0/1311247666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c0ffe60 msgr2=0x7fb49c19dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c0ffe60 0x7fb49c19dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 -- 172.21.15.22:0/1311247666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 msgr2=0x7fb49c19e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 0x7fb49c19e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb49be00640 1 -- 172.21.15.22:0/1311247666 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb494009840 con 0x7fb49c101160 2024-03-24T06:37:28.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.595+0000 7fb492a00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 0x7fb49c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb49be00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb49400efd0 tx=0x7fb49400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494009b40 con 0x7fb49c101160 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48c009560 con 0x7fb49c101160 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb494003b70 con 0x7fb49c101160 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4940108f0 con 0x7fb49c101160 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.596+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb49c1a6070 con 0x7fb49c101160 2024-03-24T06:37:28.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.597+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb49c106920 con 0x7fb49c101160 2024-03-24T06:37:28.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.599+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb494003ce0 con 0x7fb49c101160 2024-03-24T06:37:28.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.601+0000 7fb498c00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 0x7fb4800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:28.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.601+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb494014070 con 0x7fb49c101160 2024-03-24T06:37:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.601+0000 7fb49b400640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 0x7fb4800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:28.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.602+0000 7fb49b400640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 0x7fb4800a4580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb48c000c00 tx=0x7fb48c02c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:28.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.605+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb494097230 con 0x7fb49c101160 2024-03-24T06:37:28.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:28 smithi022 ceph-mon[32003]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:28 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1487380284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:28 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1927695086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:28 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:28 smithi143 ceph-mon[36520]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:28 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1487380284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:28 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1927695086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:28 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:28 smithi155 ceph-mon[36262]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:28 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1487380284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:28 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1927695086' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:28 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T06:37:28.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:28.849+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fb49c02d130 con 0x7fb49c101160 2024-03-24T06:37:29.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.356+0000 7fb498c00640 1 -- 172.21.15.22:0/1311247666 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fb494099160 con 0x7fb49c101160 2024-03-24T06:37:29.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 msgr2=0x7fb4800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:29.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 0x7fb4800a4580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb48c000c00 tx=0x7fb48c02c800 comp rx=0 tx=0).stop 2024-03-24T06:37:29.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 msgr2=0x7fb49c1a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb49400efd0 tx=0x7fb49400c6a0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 shutdown_connections 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb4800a20c0 0x7fb4800a4580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb49c101160 0x7fb49c1a51f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb49c1007c0 0x7fb49c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 --2- 172.21.15.22:0/1311247666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb49c0ffe60 0x7fb49c19dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 >> 172.21.15.22:0/1311247666 conn(0x7fb49c0fba80 msgr2=0x7fb49c0fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.361+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 shutdown_connections 2024-03-24T06:37:29.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.362+0000 7fb4a0e00640 1 -- 172.21.15.22:0/1311247666 wait complete. 2024-03-24T06:37:29.388 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T06:37:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 -- 172.21.15.22:0/2033122625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 msgr2=0x7f24ac100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 --2- 172.21.15.22:0/2033122625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac100280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009900 tx=0x7f24a002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 -- 172.21.15.22:0/2033122625 shutdown_connections 2024-03-24T06:37:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 --2- 172.21.15.22:0/2033122625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac101160 0x7f24ac1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 --2- 172.21.15.22:0/2033122625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac1007c0 0x7f24ac100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 --2- 172.21.15.22:0/2033122625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac100280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 -- 172.21.15.22:0/2033122625 >> 172.21.15.22:0/2033122625 conn(0x7f24ac0fba80 msgr2=0x7f24ac0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 -- 172.21.15.22:0/2033122625 shutdown_connections 2024-03-24T06:37:29.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.615+0000 7f24b2200640 1 -- 172.21.15.22:0/2033122625 wait complete. 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.616+0000 7f24b2200640 1 Processor -- start 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.616+0000 7f24b2200640 1 -- start start 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 0x7f24ac199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1a1340 con 0x7f24ac0ffe60 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1a14b0 con 0x7f24ac101160 2024-03-24T06:37:29.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b2200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1a1670 con 0x7f24ac1007c0 2024-03-24T06:37:29.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b0e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:29.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:29.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24abe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 0x7f24ac199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:29.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59494/0 (socket says 172.21.15.22:59494) 2024-03-24T06:37:29.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.617+0000 7f24b1800640 1 -- 172.21.15.22:0/1222097510 learned_addr learned my addr 172.21.15.22:0/1222097510 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 -- 172.21.15.22:0/1222097510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 msgr2=0x7f24ac199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 0x7f24ac199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 -- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 msgr2=0x7f24ac199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 -- 172.21.15.22:0/1222097510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a0009560 con 0x7f24ac101160 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24abe00640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 0x7f24ac199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b0e00640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.618+0000 7f24b1800640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f24a400eaa0 tx=0x7f24a400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.619+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a400cdb0 con 0x7f24ac101160 2024-03-24T06:37:29.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.619+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ac1a1950 con 0x7f24ac101160 2024-03-24T06:37:29.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.619+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a4005020 con 0x7f24ac101160 2024-03-24T06:37:29.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.619+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a40116d0 con 0x7f24ac101160 2024-03-24T06:37:29.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.619+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ac1a1e60 con 0x7f24ac101160 2024-03-24T06:37:29.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.620+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24ac10acf0 con 0x7f24ac101160 2024-03-24T06:37:29.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.622+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f24a4005490 con 0x7f24ac101160 2024-03-24T06:37:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.623+0000 7f24a9600640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 0x7f24940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.623+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f24a40cdff0 con 0x7f24ac101160 2024-03-24T06:37:29.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.624+0000 7f24b0e00640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 0x7f24940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:29.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.625+0000 7f24b0e00640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 0x7f24940a4580 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f24a0000c00 tx=0x7f24a002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:29.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.626+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24a4096e10 con 0x7f24ac101160 2024-03-24T06:37:29.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:29 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T06:37:29.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:29 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:29.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:29 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T06:37:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:29 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T06:37:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:29 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:29 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T06:37:29.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:29 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T06:37:29.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:29 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:29.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:29 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T06:37:29.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:29.869+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f24ac10f470 con 0x7f24ac101160 2024-03-24T06:37:30.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.375+0000 7f24a9600640 1 -- 172.21.15.22:0/1222097510 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f24a401f020 con 0x7f24ac101160 2024-03-24T06:37:30.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 msgr2=0x7f24940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:30.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 0x7f24940a4580 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f24a0000c00 tx=0x7f24a002c7e0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 msgr2=0x7f24ac1a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:30.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f24a400eaa0 tx=0x7f24a400ef60 comp rx=0 tx=0).stop 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 shutdown_connections 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f24940a20c0 0x7f24940a4580 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ac101160 0x7f24ac1a0e00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ac1007c0 0x7f24ac199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 --2- 172.21.15.22:0/1222097510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ac0ffe60 0x7f24ac199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.378+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 >> 172.21.15.22:0/1222097510 conn(0x7f24ac0fba80 msgr2=0x7f24ac0fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.379+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 shutdown_connections 2024-03-24T06:37:30.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.379+0000 7f24b2200640 1 -- 172.21.15.22:0/1222097510 wait complete. 2024-03-24T06:37:30.404 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T06:37:30.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 -- 172.21.15.22:0/1383506681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c81073f0 msgr2=0x7f63c8109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 --2- 172.21.15.22:0/1383506681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c81073f0 0x7f63c8109840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009900 tx=0x7f63c002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 -- 172.21.15.22:0/1383506681 shutdown_connections 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 --2- 172.21.15.22:0/1383506681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c81073f0 0x7f63c8109840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 --2- 172.21.15.22:0/1383506681 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8104ac0 0x7f63c8106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 --2- 172.21.15.22:0/1383506681 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c8102190 0x7f63c8104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 -- 172.21.15.22:0/1383506681 >> 172.21.15.22:0/1383506681 conn(0x7f63c80fbba0 msgr2=0x7f63c80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 -- 172.21.15.22:0/1383506681 shutdown_connections 2024-03-24T06:37:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.607+0000 7f63cec00640 1 -- 172.21.15.22:0/1383506681 wait complete. 2024-03-24T06:37:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 Processor -- start 2024-03-24T06:37:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 -- start start 2024-03-24T06:37:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 0x7f63c819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f63c819f2d0 con 0x7f63c8102190 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f63c819f440 con 0x7f63c8104ac0 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.609+0000 7f63cec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f63c819f600 con 0x7f63c81073f0 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59512/0 (socket says 172.21.15.22:59512) 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7400640 1 -- 172.21.15.22:0/309940970 learned_addr learned my addr 172.21.15.22:0/309940970 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40014/0 (socket says 172.21.15.22:40014) 2024-03-24T06:37:30.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 0x7f63c819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:30.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 -- 172.21.15.22:0/309940970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 msgr2=0x7f63c8197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 -- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 msgr2=0x7f63c81977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63ccc00640 1 -- 172.21.15.22:0/309940970 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c0009560 con 0x7f63c81073f0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7400640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.610+0000 7f63c7e00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63ccc00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 0x7f63c819ed90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63c0036040 tx=0x7f63c0005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004490 con 0x7f63c81073f0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c819f880 con 0x7f63c81073f0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63c00052d0 con 0x7f63c81073f0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0003d90 con 0x7f63c81073f0 2024-03-24T06:37:30.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.611+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c819fde0 con 0x7f63c81073f0 2024-03-24T06:37:30.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.612+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c80fd690 con 0x7f63c81073f0 2024-03-24T06:37:30.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.614+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f63c00035f0 con 0x7f63c81073f0 2024-03-24T06:37:30.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.616+0000 7f63c4c00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 0x7f63ac0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:30.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.616+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f63c00f0770 con 0x7f63c81073f0 2024-03-24T06:37:30.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.616+0000 7f63c7e00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 0x7f63ac0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:30.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.617+0000 7f63c7e00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 0x7f63ac0a4600 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005fd0 tx=0x7f63b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:30.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.618+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f63c00b94a0 con 0x7f63c81073f0 2024-03-24T06:37:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T06:37:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T06:37:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T06:37:30.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-24T06:37:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-24T06:37:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1222097510' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:30 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1222097510' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:30 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T06:37:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T06:37:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1311247666' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T06:37:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T06:37:30.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-03-24T06:37:30.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-03-24T06:37:30.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-03-24T06:37:30.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1222097510' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:30 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T06:37:30.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:30.862+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f63c802d1d0 con 0x7f63c81073f0 2024-03-24T06:37:31.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.391+0000 7f63c4c00640 1 -- 172.21.15.22:0/309940970 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f63c0043020 con 0x7f63c81073f0 2024-03-24T06:37:31.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.394+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 msgr2=0x7f63ac0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.394+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 0x7f63ac0a4600 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005fd0 tx=0x7f63b8009500 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.394+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 msgr2=0x7f63c819ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.394+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 0x7f63c819ed90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63c0036040 tx=0x7f63c0005e20 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 shutdown_connections 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f63ac0a2140 0x7f63ac0a4600 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63c81073f0 0x7f63c819ed90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63c8104ac0 0x7f63c8197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 --2- 172.21.15.22:0/309940970 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f63c8102190 0x7f63c81977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 >> 172.21.15.22:0/309940970 conn(0x7f63c80fbba0 msgr2=0x7f63c8102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 shutdown_connections 2024-03-24T06:37:31.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.395+0000 7f63cec00640 1 -- 172.21.15.22:0/309940970 wait complete. 2024-03-24T06:37:31.421 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-24T06:37:31.421 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 -- 172.21.15.22:0/2271256548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1880ffe60 msgr2=0x7fb188100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 --2- 172.21.15.22:0/2271256548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1880ffe60 0x7fb188100280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009900 tx=0x7fb17c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 -- 172.21.15.22:0/2271256548 shutdown_connections 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 --2- 172.21.15.22:0/2271256548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb188101160 0x7fb1881015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 --2- 172.21.15.22:0/2271256548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 --2- 172.21.15.22:0/2271256548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1880ffe60 0x7fb188100280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 -- 172.21.15.22:0/2271256548 >> 172.21.15.22:0/2271256548 conn(0x7fb1880fba80 msgr2=0x7fb1880fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:31.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 -- 172.21.15.22:0/2271256548 shutdown_connections 2024-03-24T06:37:31.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.610+0000 7fb18f200640 1 -- 172.21.15.22:0/2271256548 wait complete. 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 Processor -- start 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 -- start start 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 0x7fb1881a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881a1360 con 0x7fb1880ffe60 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881a14d0 con 0x7fb1881007c0 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1881a1690 con 0x7fb188101160 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 0x7fb1881a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:31.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40028/0 (socket says 172.21.15.22:40028) 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.611+0000 7fb18de00640 1 -- 172.21.15.22:0/441280015 learned_addr learned my addr 172.21.15.22:0/441280015 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 -- 172.21.15.22:0/441280015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 msgr2=0x7fb1881a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 0x7fb1881a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 -- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 msgr2=0x7fb188199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 -- 172.21.15.22:0/441280015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c009560 con 0x7fb1881007c0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18de00640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18e800640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 0x7fb1881a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18d400640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188199db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb17800e9f0 tx=0x7fb17800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17800ccc0 con 0x7fb1881007c0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1881a1970 con 0x7fb1881007c0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb17800ce20 con 0x7fb1881007c0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.612+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178004a00 con 0x7fb1881007c0 2024-03-24T06:37:31.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.613+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1881a1e30 con 0x7fb1881007c0 2024-03-24T06:37:31.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.614+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18810ada0 con 0x7fb1881007c0 2024-03-24T06:37:31.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.616+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb1780050b0 con 0x7fb1881007c0 2024-03-24T06:37:31.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.617+0000 7fb182a00640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 0x7fb1700a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:31.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.618+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb178014070 con 0x7fb1881007c0 2024-03-24T06:37:31.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.618+0000 7fb18de00640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 0x7fb1700a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:31.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.618+0000 7fb18de00640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 0x7fb1700a4580 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb17c02cdd0 tx=0x7fb17c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:31.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.621+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb178096bf0 con 0x7fb1881007c0 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:active 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/309940970' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:active 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-24T06:37:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/309940970' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:31 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] up:active 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-24T06:37:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/309940970' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:31 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T06:37:31.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.847+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb18802d1a0 con 0x7fb1700a20c0 2024-03-24T06:37:31.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.889+0000 7fb182a00640 1 -- 172.21.15.22:0/441280015 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb18802d1a0 con 0x7fb1700a20c0 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 msgr2=0x7fb1700a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 0x7fb1700a4580 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb17c02cdd0 tx=0x7fb17c02c7e0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 msgr2=0x7fb188199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188199db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb17800e9f0 tx=0x7fb17800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 shutdown_connections 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb1700a20c0 0x7fb1700a4580 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb188101160 0x7fb1881a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1881007c0 0x7fb188199db0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 --2- 172.21.15.22:0/441280015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1880ffe60 0x7fb188199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:31.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 >> 172.21.15.22:0/441280015 conn(0x7fb1880fba80 msgr2=0x7fb1880fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:31.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 shutdown_connections 2024-03-24T06:37:31.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:31.892+0000 7fb18f200640 1 -- 172.21.15.22:0/441280015 wait complete. 2024-03-24T06:37:31.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 -- 172.21.15.22:0/1663543642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848075750 msgr2=0x7f4848073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 --2- 172.21.15.22:0/1663543642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848075750 0x7f4848073ba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4838009900 tx=0x7f483802bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 -- 172.21.15.22:0/1663543642 shutdown_connections 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 --2- 172.21.15.22:0/1663543642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 0x7f484810b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 --2- 172.21.15.22:0/1663543642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848074170 0x7f48480745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 --2- 172.21.15.22:0/1663543642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848075750 0x7f4848073ba0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 -- 172.21.15.22:0/1663543642 >> 172.21.15.22:0/1663543642 conn(0x7f48480fcac0 msgr2=0x7f48480feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:32.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 -- 172.21.15.22:0/1663543642 shutdown_connections 2024-03-24T06:37:32.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.120+0000 7f484ce00640 1 -- 172.21.15.22:0/1663543642 wait complete. 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.121+0000 7f484ce00640 1 Processor -- start 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.121+0000 7f484ce00640 1 -- start start 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 0x7f4848199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 0x7f48481a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f48481a1250 con 0x7f4848075750 2024-03-24T06:37:32.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f48481a13c0 con 0x7f4848074170 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f484ce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f48481a1580 con 0x7f48481093f0 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59566/0 (socket says 172.21.15.22:59566) 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 -- 172.21.15.22:0/188330806 learned_addr learned my addr 172.21.15.22:0/188330806 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847e00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 0x7f48481a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f483fe00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 0x7f4848199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 -- 172.21.15.22:0/188330806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 msgr2=0x7f48481a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 0x7f48481a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 -- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 msgr2=0x7f4848199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 0x7f4848199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f4847400640 1 -- 172.21.15.22:0/188330806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4838009560 con 0x7f4848074170 2024-03-24T06:37:32.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.122+0000 7f483fe00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 0x7f4848199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.123+0000 7f4847400640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4838006010 tx=0x7f48380040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.123+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48380044c0 con 0x7f4848074170 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.123+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48481a1800 con 0x7f4848074170 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.123+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48380052d0 con 0x7f4848074170 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.123+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838003e10 con 0x7f4848074170 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.124+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48481a1d10 con 0x7f4848074170 2024-03-24T06:37:32.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.125+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4848108120 con 0x7f4848074170 2024-03-24T06:37:32.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.127+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4838005440 con 0x7f4848074170 2024-03-24T06:37:32.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.128+0000 7f4844c00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 0x7f482c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.128+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f48380f02c0 con 0x7f4848074170 2024-03-24T06:37:32.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.128+0000 7f483fe00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 0x7f482c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.129+0000 7f483fe00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 0x7f482c0a4580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48300058c0 tx=0x7f4830005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:32.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.132+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f48380b9050 con 0x7f4848074170 2024-03-24T06:37:32.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:37:31.861+0000 7f01c3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:37:32.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:37:31.861+0000 7f01c3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:37:32.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:37:31.861+0000 7f01c3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:37:32.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:37:31.861+0000 7f01c3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:37:32.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:37:31 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:37:31.861+0000 7f01c3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:37:32.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.359+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f484802d1a0 con 0x7f482c0a20c0 2024-03-24T06:37:32.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.405+0000 7f4844c00640 1 -- 172.21.15.22:0/188330806 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f484802d1a0 con 0x7f482c0a20c0 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 msgr2=0x7f482c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 0x7f482c0a4580 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f48300058c0 tx=0x7f4830005850 comp rx=0 tx=0).stop 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 msgr2=0x7f4848199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4838006010 tx=0x7f48380040c0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 shutdown_connections 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f482c0a20c0 0x7f482c0a4580 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48481093f0 0x7f48481a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4848075750 0x7f4848199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 --2- 172.21.15.22:0/188330806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4848074170 0x7f4848199760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 >> 172.21.15.22:0/188330806 conn(0x7f48480fcac0 msgr2=0x7f48481052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 shutdown_connections 2024-03-24T06:37:32.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.407+0000 7f484ce00640 1 -- 172.21.15.22:0/188330806 wait complete. 2024-03-24T06:37:32.427 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T06:37:32.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 -- 172.21.15.22:0/3908400643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0102800 msgr2=0x7f0de0102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3908400643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0102800 0x7f0de0102c80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009900 tx=0x7f0dcc02bc50 comp rx=0 tx=0).stop 2024-03-24T06:37:32.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 -- 172.21.15.22:0/3908400643 shutdown_connections 2024-03-24T06:37:32.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3908400643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de01031c0 0x7f0de0109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3908400643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0102800 0x7f0de0102c80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3908400643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0101610 0x7f0de0101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 -- 172.21.15.22:0/3908400643 >> 172.21.15.22:0/3908400643 conn(0x7f0de00fcb80 msgr2=0x7f0de00fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 -- 172.21.15.22:0/3908400643 shutdown_connections 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.615+0000 7f0de5e00640 1 -- 172.21.15.22:0/3908400643 wait complete. 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 Processor -- start 2024-03-24T06:37:32.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 -- start start 2024-03-24T06:37:32.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 0x7f0de0197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de01031c0 0x7f0de019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0de019f1a0 con 0x7f0de0102800 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0de019f310 con 0x7f0de0101610 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de5e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0de019f4d0 con 0x7f0de01031c0 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59588/0 (socket says 172.21.15.22:59588) 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.617+0000 7f0de4a00640 1 -- 172.21.15.22:0/3751181572 learned_addr learned my addr 172.21.15.22:0/3751181572 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de5400640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de01031c0 0x7f0de019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0ddfe00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 0x7f0de0197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 -- 172.21.15.22:0/3751181572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de01031c0 msgr2=0x7f0de019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de01031c0 0x7f0de019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 -- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 msgr2=0x7f0de0197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 0x7f0de0197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 -- 172.21.15.22:0/3751181572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd4009490 con 0x7f0de0101610 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0ddfe00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 0x7f0de0197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:32.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de4a00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400e9c0 tx=0x7f0dd400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd400cde0 con 0x7f0de0101610 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc009560 con 0x7f0de0101610 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dd4010910 con 0x7f0de0101610 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.618+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4018960 con 0x7f0de0101610 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.619+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de019fb30 con 0x7f0de0101610 2024-03-24T06:37:32.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.619+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de01088d0 con 0x7f0de0101610 2024-03-24T06:37:32.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.622+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0dd4010a80 con 0x7f0de0101610 2024-03-24T06:37:32.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.623+0000 7f0ddd600640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 0x7f0dc80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:32.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.623+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0dd4014070 con 0x7f0de0101610 2024-03-24T06:37:32.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.623+0000 7f0ddfe00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 0x7f0dc80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:32.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.624+0000 7f0ddfe00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 0x7f0dc80a4580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0de0198bc0 tx=0x7f0dcc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:32.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.625+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0dd4097490 con 0x7f0de0101610 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:active 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-24T06:37:32.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-03-24T06:37:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-24T06:37:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:32 smithi022 ceph-mon[32003]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:active 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-24T06:37:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:32 smithi143 ceph-mon[36520]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] up:active 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-03-24T06:37:32.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-24T06:37:32.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:32 smithi155 ceph-mon[36262]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:32.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.846+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0de002d1d0 con 0x7f0dc80a20c0 2024-03-24T06:37:32.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.862+0000 7f0ddd600640 1 -- 172.21.15.22:0/3751181572 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0de002d1d0 con 0x7f0dc80a20c0 2024-03-24T06:37:32.862 INFO:teuthology.orchestra.run.smithi022.stdout:/volumes/qa/sv_0/81767d6f-d01b-45b2-a586-238d8b5c0ac6 2024-03-24T06:37:32.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.864+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 msgr2=0x7f0dc80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.864+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 0x7f0dc80a4580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0de0198bc0 tx=0x7f0dcc000bc0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.864+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 msgr2=0x7f0de01976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:32.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.864+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400e9c0 tx=0x7f0dd400ee80 comp rx=0 tx=0).stop 2024-03-24T06:37:32.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 shutdown_connections 2024-03-24T06:37:32.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0dc80a20c0 0x7f0dc80a4580 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de01031c0 0x7f0de019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0de0102800 0x7f0de0197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 --2- 172.21.15.22:0/3751181572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0de0101610 0x7f0de01976b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 >> 172.21.15.22:0/3751181572 conn(0x7f0de00fcb80 msgr2=0x7f0de0105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 shutdown_connections 2024-03-24T06:37:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:32.865+0000 7f0de5e00640 1 -- 172.21.15.22:0/3751181572 wait complete. 2024-03-24T06:37:32.883 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 -- 172.21.15.22:0/575742103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc100400 msgr2=0x7f11fc100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 --2- 172.21.15.22:0/575742103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc100400 0x7f11fc100820 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009900 tx=0x7f11f002bc40 comp rx=0 tx=0).stop 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 -- 172.21.15.22:0/575742103 shutdown_connections 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 --2- 172.21.15.22:0/575742103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc101fb0 0x7f11fc108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 --2- 172.21.15.22:0/575742103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc1015f0 0x7f11fc101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 --2- 172.21.15.22:0/575742103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc100400 0x7f11fc100820 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 -- 172.21.15.22:0/575742103 >> 172.21.15.22:0/575742103 conn(0x7f11fc0fb9b0 msgr2=0x7f11fc0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 -- 172.21.15.22:0/575742103 shutdown_connections 2024-03-24T06:37:33.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.096+0000 7f1202a00640 1 -- 172.21.15.22:0/575742103 wait complete. 2024-03-24T06:37:33.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 Processor -- start 2024-03-24T06:37:33.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 -- start start 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 0x7f11fc199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 0x7f11fc1a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a1290 con 0x7f11fc101fb0 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a1400 con 0x7f11fc100400 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.097+0000 7f1202a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a15c0 con 0x7f11fc1015f0 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1200c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 0x7f11fc199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1200c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 0x7f11fc199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:36230/0 (socket says 172.21.15.22:36230) 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1200c00640 1 -- 172.21.15.22:0/3613157964 learned_addr learned my addr 172.21.15.22:0/3613157964 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59606/0 (socket says 172.21.15.22:59606) 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1202000640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 0x7f11fc1a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 -- 172.21.15.22:0/3613157964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 msgr2=0x7f11fc199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 0x7f11fc199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 -- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 msgr2=0x7f11fc1a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 0x7f11fc1a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 -- 172.21.15.22:0/3613157964 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8009490 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1202000640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 0x7f11fc1a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f1201600640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f11f0036040 tx=0x7f11f0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.098+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f00043d0 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.099+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11f00052d0 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.099+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f0009560 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.099+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0003c60 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.099+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc1a1be0 con 0x7f11fc100400 2024-03-24T06:37:33.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.100+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc10be30 con 0x7f11fc100400 2024-03-24T06:37:33.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.102+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f11f0004df0 con 0x7f11fc100400 2024-03-24T06:37:33.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.103+0000 7f11fa000640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 0x7f11e40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.104+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11f00f0d70 con 0x7f11fc100400 2024-03-24T06:37:33.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.104+0000 7f1200c00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 0x7f11e40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.104+0000 7f1200c00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 0x7f11e40a4600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11fc19acb0 tx=0x7f11e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:33.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.109+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f11f00b9aa0 con 0x7f11fc100400 2024-03-24T06:37:33.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.332+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f11fc0fdb20 con 0x7f11e40a2140 2024-03-24T06:37:33.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.372+0000 7f11fa000640 1 -- 172.21.15.22:0/3613157964 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f11fc0fdb20 con 0x7f11e40a2140 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 msgr2=0x7f11e40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 0x7f11e40a4600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11fc19acb0 tx=0x7f11e8005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 msgr2=0x7f11fc1997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f11f0036040 tx=0x7f11f0005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 shutdown_connections 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f11e40a2140 0x7f11e40a4600 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc101fb0 0x7f11fc1a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc1015f0 0x7f11fc199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 --2- 172.21.15.22:0/3613157964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc100400 0x7f11fc1997a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 >> 172.21.15.22:0/3613157964 conn(0x7f11fc0fb9b0 msgr2=0x7f11fc0fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:33.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 shutdown_connections 2024-03-24T06:37:33.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.375+0000 7f1202a00640 1 -- 172.21.15.22:0/3613157964 wait complete. 2024-03-24T06:37:33.391 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 -- 172.21.15.22:0/1067809998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 msgr2=0x7ff82c103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 --2- 172.21.15.22:0/1067809998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c103d50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009900 tx=0x7ff81c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 -- 172.21.15.22:0/1067809998 shutdown_connections 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 --2- 172.21.15.22:0/1067809998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 0x7ff82c10bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 --2- 172.21.15.22:0/1067809998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 --2- 172.21.15.22:0/1067809998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c103d50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 -- 172.21.15.22:0/1067809998 >> 172.21.15.22:0/1067809998 conn(0x7ff82c0feec0 msgr2=0x7ff82c1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 -- 172.21.15.22:0/1067809998 shutdown_connections 2024-03-24T06:37:33.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.592+0000 7ff830a00640 1 -- 172.21.15.22:0/1067809998 wait complete. 2024-03-24T06:37:33.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.593+0000 7ff830a00640 1 Processor -- start 2024-03-24T06:37:33.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.593+0000 7ff830a00640 1 -- start start 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 0x7ff82c1a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1a14e0 con 0x7ff82c103930 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1a1650 con 0x7ff82c104b20 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff830a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c1a1810 con 0x7ff82c1054e0 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40104/0 (socket says 172.21.15.22:40104) 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 -- 172.21.15.22:0/2154770660 learned_addr learned my addr 172.21.15.22:0/2154770660 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829400640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff82a800640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 0x7ff82c1a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 -- 172.21.15.22:0/2154770660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 msgr2=0x7ff82c1a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 0x7ff82c1a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 -- 172.21.15.22:0/2154770660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 msgr2=0x7ff82c199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829e00640 1 -- 172.21.15.22:0/2154770660 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff818009490 con 0x7ff82c103930 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.594+0000 7ff829400640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff829e00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff81c000c00 tx=0x7ff81c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c0043d0 con 0x7ff82c103930 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81c0052d0 con 0x7ff82c103930 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c009560 con 0x7ff82c103930 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c003d30 con 0x7ff82c103930 2024-03-24T06:37:33.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.595+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c1a1e60 con 0x7ff82c103930 2024-03-24T06:37:33.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.597+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82c10ad20 con 0x7ff82c103930 2024-03-24T06:37:33.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.598+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff81c005440 con 0x7ff82c103930 2024-03-24T06:37:33.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.600+0000 7ff822a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 0x7ff8100a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:33.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.600+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff81c0f06f0 con 0x7ff82c103930 2024-03-24T06:37:33.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.600+0000 7ff829400640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 0x7ff8100a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:33.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.601+0000 7ff829400640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 0x7ff8100a4690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff82c19af00 tx=0x7ff818005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:33.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.604+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff81c0b94f0 con 0x7ff82c103930 2024-03-24T06:37:33.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:active 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-24T06:37:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:33 smithi022 ceph-mon[32003]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:active 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-24T06:37:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:33 smithi143 ceph-mon[36520]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] up:active 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-24T06:37:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:33 smithi155 ceph-mon[36262]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:33.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.831+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7ff82c02d1d0 con 0x7ff8100a21d0 2024-03-24T06:37:33.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.837+0000 7ff822a00640 1 -- 172.21.15.22:0/2154770660 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7ff82c02d1d0 con 0x7ff8100a21d0 2024-03-24T06:37:33.837 INFO:teuthology.orchestra.run.smithi022.stdout:/volumes/qa/sv_1/465aaf92-975d-4dec-a92a-7131150f7a52 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 msgr2=0x7ff8100a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 0x7ff8100a4690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff82c19af00 tx=0x7ff818005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 msgr2=0x7ff82c1999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff81c000c00 tx=0x7ff81c005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 shutdown_connections 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff8100a21d0 0x7ff8100a4690 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff82c1054e0 0x7ff82c1a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff82c104b20 0x7ff82c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 --2- 172.21.15.22:0/2154770660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff82c103930 0x7ff82c1999f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 >> 172.21.15.22:0/2154770660 conn(0x7ff82c0feec0 msgr2=0x7ff82c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 shutdown_connections 2024-03-24T06:37:33.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:33.840+0000 7ff830a00640 1 -- 172.21.15.22:0/2154770660 wait complete. 2024-03-24T06:37:33.859 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:37:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 -- 172.21.15.22:0/2257739724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648101750 msgr2=0x7f9648101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 --2- 172.21.15.22:0/2257739724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648101750 0x7f9648101b70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f963c009900 tx=0x7f963c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:37:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 -- 172.21.15.22:0/2257739724 shutdown_connections 2024-03-24T06:37:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 --2- 172.21.15.22:0/2257739724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 0x7f9648109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 --2- 172.21.15.22:0/2257739724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648102940 0x7f9648102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 --2- 172.21.15.22:0/2257739724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648101750 0x7f9648101b70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 -- 172.21.15.22:0/2257739724 >> 172.21.15.22:0/2257739724 conn(0x7f96480fccc0 msgr2=0x7f96480ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 -- 172.21.15.22:0/2257739724 shutdown_connections 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.070+0000 7f964fe00640 1 -- 172.21.15.22:0/2257739724 wait complete. 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.071+0000 7f964fe00640 1 Processor -- start 2024-03-24T06:37:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 -- start start 2024-03-24T06:37:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648101750 0x7f9648199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 0x7f96481a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f96481a1520 con 0x7f9648101750 2024-03-24T06:37:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f96481a1690 con 0x7f9648102940 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f96481a1850 con 0x7f9648103300 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648101750 0x7f9648199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59622/0 (socket says 172.21.15.22:59622) 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964ca00640 1 -- 172.21.15.22:0/588167542 learned_addr learned my addr 172.21.15.22:0/588167542 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.072+0000 7f964de00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 0x7f96481a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 -- 172.21.15.22:0/588167542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 msgr2=0x7f96481a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 0x7f96481a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 -- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648101750 msgr2=0x7f9648199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648101750 0x7f9648199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 -- 172.21.15.22:0/588167542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c009560 con 0x7f9648102940 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964ca00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f963400e9f0 tx=0x7f963400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634009700 con 0x7f9648102940 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f963400cd80 con 0x7f9648102940 2024-03-24T06:37:34.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96481a1b30 con 0x7f9648102940 2024-03-24T06:37:34.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.073+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634018920 con 0x7f9648102940 2024-03-24T06:37:34.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.074+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96481a2090 con 0x7f9648102940 2024-03-24T06:37:34.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.075+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f964810d170 con 0x7f9648102940 2024-03-24T06:37:34.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.078+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9634018a80 con 0x7f9648102940 2024-03-24T06:37:34.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.079+0000 7f9646000640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 0x7f96300a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.079+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9634014070 con 0x7f9648102940 2024-03-24T06:37:34.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.079+0000 7f964d400640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 0x7f96300a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.080+0000 7f964d400640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 0x7f96300a4690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f963c02c250 tx=0x7f963c02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:34.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.083+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9634053280 con 0x7f9648102940 2024-03-24T06:37:34.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.319+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96481a2520 con 0x7f9648102940 2024-03-24T06:37:34.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.320+0000 7f9646000640 1 -- 172.21.15.22:0/588167542 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7f9634097020 con 0x7f9648102940 2024-03-24T06:37:34.320 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:34.320 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:37:34.320 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 msgr2=0x7f96300a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 0x7f96300a4690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f963c02c250 tx=0x7f963c02c7f0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 msgr2=0x7f9648199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f963400e9f0 tx=0x7f963400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 shutdown_connections 2024-03-24T06:37:34.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96300a21d0 0x7f96300a4690 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9648103300 0x7f96481a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9648102940 0x7f9648199fc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 --2- 172.21.15.22:0/588167542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9648101750 0x7f9648199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.322+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 >> 172.21.15.22:0/588167542 conn(0x7f96480fccc0 msgr2=0x7f9648073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.323+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 shutdown_connections 2024-03-24T06:37:34.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.323+0000 7f964fe00640 1 -- 172.21.15.22:0/588167542 wait complete. 2024-03-24T06:37:34.342 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 -- 172.21.15.22:0/4079724524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e41050f0 msgr2=0x7fa7e4105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/4079724524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e41050f0 0x7fa7e4105570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009ca0 tx=0x7fa7dc02c100 comp rx=0 tx=0).stop 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 -- 172.21.15.22:0/4079724524 shutdown_connections 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/4079724524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e41050f0 0x7fa7e4105570 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/4079724524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 0x7fa7e4104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/4079724524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e4102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 -- 172.21.15.22:0/4079724524 >> 172.21.15.22:0/4079724524 conn(0x7fa7e40fb9f0 msgr2=0x7fa7e40fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:34.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 -- 172.21.15.22:0/4079724524 shutdown_connections 2024-03-24T06:37:34.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.548+0000 7fa7e9a00640 1 -- 172.21.15.22:0/4079724524 wait complete. 2024-03-24T06:37:34.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.549+0000 7fa7e9a00640 1 Processor -- start 2024-03-24T06:37:34.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 -- start start 2024-03-24T06:37:34.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 0x7fa7e419e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e41050f0 0x7fa7e41a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41a56c0 con 0x7fa7e4102740 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41a5830 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e41a59f0 con 0x7fa7e41050f0 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59628/0 (socket says 172.21.15.22:59628) 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e3e00640 1 -- 172.21.15.22:0/337547081 learned_addr learned my addr 172.21.15.22:0/337547081 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e9000640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e41050f0 0x7fa7e41a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.550+0000 7fa7e3400640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 0x7fa7e419e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 -- 172.21.15.22:0/337547081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e41050f0 msgr2=0x7fa7e41a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e41050f0 0x7fa7e41a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 -- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 msgr2=0x7fa7e419e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 0x7fa7e419e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 -- 172.21.15.22:0/337547081 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc009900 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e3e00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9f0 tx=0x7fa7d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:34.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4009700 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d400cd80 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41a5cd0 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4018920 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.551+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41a6190 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.553+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e40fd4e0 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.556+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa7d4018a80 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.557+0000 7fa7e0c00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 0x7fa7c80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.557+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa7d4014070 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.557+0000 7fa7e3400640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 0x7fa7c80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.558+0000 7fa7e3400640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 0x7fa7c80a4610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa7e419f0e0 tx=0x7fa7d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:34.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.561+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa7d4053280 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: pgmap v338: 129 pgs: 129 active+clean; 601 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:active 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:34 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/588167542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:34.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.761+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7e402d280 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.763+0000 7fa7e0c00640 1 -- 172.21.15.22:0/337547081 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7fa7d40970c0 con 0x7fa7e40ffdd0 2024-03-24T06:37:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:34.763 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 msgr2=0x7fa7c80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 0x7fa7c80a4610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa7e419f0e0 tx=0x7fa7d0005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 msgr2=0x7fa7e419dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400e9f0 tx=0x7fa7d400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 shutdown_connections 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa7c80a2150 0x7fa7c80a4610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e41050f0 0x7fa7e41a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7e4102740 0x7fa7e419e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 --2- 172.21.15.22:0/337547081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e40ffdd0 0x7fa7e419dbd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.765+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 >> 172.21.15.22:0/337547081 conn(0x7fa7e40fb9f0 msgr2=0x7fa7e4100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.766+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 shutdown_connections 2024-03-24T06:37:34.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.766+0000 7fa7e9a00640 1 -- 172.21.15.22:0/337547081 wait complete. 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: pgmap v338: 129 pgs: 129 active+clean; 601 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:active 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:34 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/588167542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: pgmap v338: 129 pgs: 129 active+clean; 601 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] up:active 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:37:34.780 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:37:34.781 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:34 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/588167542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:37:34.782 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T06:37:34.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.986+0000 7f82f6600640 1 -- 172.21.15.22:0/2561327032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 msgr2=0x7f82f0101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.986+0000 7f82f6600640 1 --2- 172.21.15.22:0/2561327032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0101b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009900 tx=0x7f82e002bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 -- 172.21.15.22:0/2561327032 shutdown_connections 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 --2- 172.21.15.22:0/2561327032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 0x7f82f0109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 --2- 172.21.15.22:0/2561327032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 0x7f82f0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 --2- 172.21.15.22:0/2561327032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0101b50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 -- 172.21.15.22:0/2561327032 >> 172.21.15.22:0/2561327032 conn(0x7f82f00fccc0 msgr2=0x7f82f00ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 -- 172.21.15.22:0/2561327032 shutdown_connections 2024-03-24T06:37:34.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.987+0000 7f82f6600640 1 -- 172.21.15.22:0/2561327032 wait complete. 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 Processor -- start 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 -- start start 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 0x7f82f0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 0x7f82f01a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01a14f0 con 0x7f82f0102920 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01a1660 con 0x7f82f01032e0 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82f6600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01a1820 con 0x7f82f0101730 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:34688/0 (socket says 172.21.15.22:34688) 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82ef400640 1 -- 172.21.15.22:0/1990992969 learned_addr learned my addr 172.21.15.22:0/1990992969 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82efe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 0x7f82f01a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.988+0000 7f82e7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 0x7f82f0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 -- 172.21.15.22:0/1990992969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 msgr2=0x7f82f01a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 0x7f82f01a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 -- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 msgr2=0x7f82f0199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 0x7f82f0199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 -- 172.21.15.22:0/1990992969 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e0009560 con 0x7f82f0101730 2024-03-24T06:37:34.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ef400640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82e0000c00 tx=0x7f82e0005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e00043d0 con 0x7f82f0101730 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e00052d0 con 0x7f82f0101730 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.989+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f01a1aa0 con 0x7f82f0101730 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.990+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0003d30 con 0x7f82f0101730 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.990+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f01a1fb0 con 0x7f82f0101730 2024-03-24T06:37:34.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.991+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f010d180 con 0x7f82f0101730 2024-03-24T06:37:34.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.992+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f82e0005440 con 0x7f82f0101730 2024-03-24T06:37:34.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.994+0000 7f82ecc00640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 0x7f82d40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:34.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.994+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f82e00f07c0 con 0x7f82f0101730 2024-03-24T06:37:34.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.994+0000 7f82e7e00640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 0x7f82d40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:34.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:34.995+0000 7f82e7e00640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 0x7f82d40a4610 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f82f019af10 tx=0x7f82d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:35.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.002+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f82e00b94a0 con 0x7f82f0101730 2024-03-24T06:37:35.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.276+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82f0075060 con 0x7f82d40a2150 2024-03-24T06:37:35.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.295+0000 7f82ecc00640 1 -- 172.21.15.22:0/1990992969 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340407 (secure 0 0 0) 0x7f82f0075060 con 0x7f82d40a2150 2024-03-24T06:37:35.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:35.303 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 msgr2=0x7f82d40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 0x7f82d40a4610 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f82f019af10 tx=0x7f82d8005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 msgr2=0x7f82f0199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82e0000c00 tx=0x7f82e0005e80 comp rx=0 tx=0).stop 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 shutdown_connections 2024-03-24T06:37:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f82d40a2150 0x7f82d40a4610 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f01032e0 0x7f82f01a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f82f0102920 0x7f82f0199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 --2- 172.21.15.22:0/1990992969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82f0101730 0x7f82f0199a00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 >> 172.21.15.22:0/1990992969 conn(0x7f82f00fccc0 msgr2=0x7f82f0073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 shutdown_connections 2024-03-24T06:37:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.302+0000 7f82f6600640 1 -- 172.21.15.22:0/1990992969 wait complete. 2024-03-24T06:37:35.321 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-24T06:37:35.104511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619174,"num_objects":81,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":146,"num_read_kb":126,"num_write":356,"num_write_kb":3018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":292,"ondisk_log_size":292,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383564,"kb_used_data":13612,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558260,"statfs":{"total":1151940427776,"available":1151547658240,"internally_reserved":0,"allocated":13938688,"data_stored":7580025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24166,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":126,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006170"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555946+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:37:02.555946+0000","last_peered":"2024-03-24T06:37:02.555946+0000","last_clean":"2024-03-24T06:37:02.555946+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:37:02.555946+0000","last_undegraded":"2024-03-24T06:37:02.555946+0000","last_fullsized":"2024-03-24T06:37:02.555946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814602+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:37:02.814602+0000","last_peered":"2024-03-24T06:37:02.814602+0000","last_clean":"2024-03-24T06:37:02.814602+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:37:02.814602+0000","last_undegraded":"2024-03-24T06:37:02.814602+0000","last_fullsized":"2024-03-24T06:37:02.814602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814499+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:37:02.814499+0000","last_peered":"2024-03-24T06:37:02.814499+0000","last_clean":"2024-03-24T06:37:02.814499+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:37:02.814499+0000","last_undegraded":"2024-03-24T06:37:02.814499+0000","last_fullsized":"2024-03-24T06:37:02.814499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555114+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:37:02.555114+0000","last_peered":"2024-03-24T06:37:02.555114+0000","last_clean":"2024-03-24T06:37:02.555114+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:37:02.555114+0000","last_undegraded":"2024-03-24T06:37:02.555114+0000","last_fullsized":"2024-03-24T06:37:02.555114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554643+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:37:02.554643+0000","last_peered":"2024-03-24T06:37:02.554643+0000","last_clean":"2024-03-24T06:37:02.554643+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:37:02.554643+0000","last_undegraded":"2024-03-24T06:37:02.554643+0000","last_fullsized":"2024-03-24T06:37:02.554643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.312905+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:37:02.312905+0000","last_peered":"2024-03-24T06:37:02.312905+0000","last_clean":"2024-03-24T06:37:02.312905+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:37:02.312905+0000","last_undegraded":"2024-03-24T06:37:02.312905+0000","last_fullsized":"2024-03-24T06:37:02.312905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225360+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:37:02.225360+0000","last_peered":"2024-03-24T06:37:02.225360+0000","last_clean":"2024-03-24T06:37:02.225360+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:37:02.225360+0000","last_undegraded":"2024-03-24T06:37:02.225360+0000","last_fullsized":"2024-03-24T06:37:02.225360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.319870+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:37:02.319870+0000","last_peered":"2024-03-24T06:37:02.319870+0000","last_clean":"2024-03-24T06:37:02.319870+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:37:02.319870+0000","last_undegraded":"2024-03-24T06:37:02.319870+0000","last_fullsized":"2024-03-24T06:37:02.319870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555424+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:37:02.555424+0000","last_peered":"2024-03-24T06:37:02.555424+0000","last_clean":"2024-03-24T06:37:02.555424+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:37:02.555424+0000","last_undegraded":"2024-03-24T06:37:02.555424+0000","last_fullsized":"2024-03-24T06:37:02.555424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.313331+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:37:02.313331+0000","last_peered":"2024-03-24T06:37:02.313331+0000","last_clean":"2024-03-24T06:37:02.313331+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:37:02.313331+0000","last_undegraded":"2024-03-24T06:37:02.313331+0000","last_fullsized":"2024-03-24T06:37:02.313331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225858+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:37:02.225858+0000","last_peered":"2024-03-24T06:37:02.225858+0000","last_clean":"2024-03-24T06:37:02.225858+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:37:02.225858+0000","last_undegraded":"2024-03-24T06:37:02.225858+0000","last_fullsized":"2024-03-24T06:37:02.225858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555016+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:37:02.555016+0000","last_peered":"2024-03-24T06:37:02.555016+0000","last_clean":"2024-03-24T06:37:02.555016+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:37:02.555016+0000","last_undegraded":"2024-03-24T06:37:02.555016+0000","last_fullsized":"2024-03-24T06:37:02.555016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556912+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:37:02.556912+0000","last_peered":"2024-03-24T06:37:02.556912+0000","last_clean":"2024-03-24T06:37:02.556912+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:37:02.556912+0000","last_undegraded":"2024-03-24T06:37:02.556912+0000","last_fullsized":"2024-03-24T06:37:02.556912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814261+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:37:02.814261+0000","last_peered":"2024-03-24T06:37:02.814261+0000","last_clean":"2024-03-24T06:37:02.814261+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:37:02.814261+0000","last_undegraded":"2024-03-24T06:37:02.814261+0000","last_fullsized":"2024-03-24T06:37:02.814261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814127+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:37:02.814127+0000","last_peered":"2024-03-24T06:37:02.814127+0000","last_clean":"2024-03-24T06:37:02.814127+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:37:02.814127+0000","last_undegraded":"2024-03-24T06:37:02.814127+0000","last_fullsized":"2024-03-24T06:37:02.814127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554191+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:37:02.554191+0000","last_peered":"2024-03-24T06:37:02.554191+0000","last_clean":"2024-03-24T06:37:02.554191+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:37:02.554191+0000","last_undegraded":"2024-03-24T06:37:02.554191+0000","last_fullsized":"2024-03-24T06:37:02.554191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555915+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:37:02.555915+0000","last_peered":"2024-03-24T06:37:02.555915+0000","last_clean":"2024-03-24T06:37:02.555915+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:37:02.555915+0000","last_undegraded":"2024-03-24T06:37:02.555915+0000","last_fullsized":"2024-03-24T06:37:02.555915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.231489+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:37:02.231489+0000","last_peered":"2024-03-24T06:37:02.231489+0000","last_clean":"2024-03-24T06:37:02.231489+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:37:02.231489+0000","last_undegraded":"2024-03-24T06:37:02.231489+0000","last_fullsized":"2024-03-24T06:37:02.231489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226603+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:37:02.226603+0000","last_peered":"2024-03-24T06:37:02.226603+0000","last_clean":"2024-03-24T06:37:02.226603+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:37:02.226603+0000","last_undegraded":"2024-03-24T06:37:02.226603+0000","last_fullsized":"2024-03-24T06:37:02.226603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226204+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:37:02.226204+0000","last_peered":"2024-03-24T06:37:02.226204+0000","last_clean":"2024-03-24T06:37:02.226204+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:37:02.226204+0000","last_undegraded":"2024-03-24T06:37:02.226204+0000","last_fullsized":"2024-03-24T06:37:02.226204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.400225+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:37:30.400225+0000","last_peered":"2024-03-24T06:37:30.400225+0000","last_clean":"2024-03-24T06:37:30.400225+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:37:30.400225+0000","last_undegraded":"2024-03-24T06:37:30.400225+0000","last_fullsized":"2024-03-24T06:37:30.400225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320610+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:37:02.320610+0000","last_peered":"2024-03-24T06:37:02.320610+0000","last_clean":"2024-03-24T06:37:02.320610+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:37:02.320610+0000","last_undegraded":"2024-03-24T06:37:02.320610+0000","last_fullsized":"2024-03-24T06:37:02.320610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554503+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:37:02.554503+0000","last_peered":"2024-03-24T06:37:02.554503+0000","last_clean":"2024-03-24T06:37:02.554503+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:37:02.554503+0000","last_undegraded":"2024-03-24T06:37:02.554503+0000","last_fullsized":"2024-03-24T06:37:02.554503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.321051+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:37:02.321051+0000","last_peered":"2024-03-24T06:37:02.321051+0000","last_clean":"2024-03-24T06:37:02.321051+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:37:02.321051+0000","last_undegraded":"2024-03-24T06:37:02.321051+0000","last_fullsized":"2024-03-24T06:37:02.321051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814865+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:37:02.814865+0000","last_peered":"2024-03-24T06:37:02.814865+0000","last_clean":"2024-03-24T06:37:02.814865+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:37:02.814865+0000","last_undegraded":"2024-03-24T06:37:02.814865+0000","last_fullsized":"2024-03-24T06:37:02.814865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554904+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:37:02.554904+0000","last_peered":"2024-03-24T06:37:02.554904+0000","last_clean":"2024-03-24T06:37:02.554904+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:37:02.554904+0000","last_undegraded":"2024-03-24T06:37:02.554904+0000","last_fullsized":"2024-03-24T06:37:02.554904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555340+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:37:02.555340+0000","last_peered":"2024-03-24T06:37:02.555340+0000","last_clean":"2024-03-24T06:37:02.555340+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:37:02.555340+0000","last_undegraded":"2024-03-24T06:37:02.555340+0000","last_fullsized":"2024-03-24T06:37:02.555340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554690+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:37:02.554690+0000","last_peered":"2024-03-24T06:37:02.554690+0000","last_clean":"2024-03-24T06:37:02.554690+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:37:02.554690+0000","last_undegraded":"2024-03-24T06:37:02.554690+0000","last_fullsized":"2024-03-24T06:37:02.554690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554776+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:37:02.554776+0000","last_peered":"2024-03-24T06:37:02.554776+0000","last_clean":"2024-03-24T06:37:02.554776+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:37:02.554776+0000","last_undegraded":"2024-03-24T06:37:02.554776+0000","last_fullsized":"2024-03-24T06:37:02.554776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555447+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:37:02.555447+0000","last_peered":"2024-03-24T06:37:02.555447+0000","last_clean":"2024-03-24T06:37:02.555447+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:37:02.555447+0000","last_undegraded":"2024-03-24T06:37:02.555447+0000","last_fullsized":"2024-03-24T06:37:02.555447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556207+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:37:02.556207+0000","last_peered":"2024-03-24T06:37:02.556207+0000","last_clean":"2024-03-24T06:37:02.556207+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:37:02.556207+0000","last_undegraded":"2024-03-24T06:37:02.556207+0000","last_fullsized":"2024-03-24T06:37:02.556207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226569+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:37:02.226569+0000","last_peered":"2024-03-24T06:37:02.226569+0000","last_clean":"2024-03-24T06:37:02.226569+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:37:02.226569+0000","last_undegraded":"2024-03-24T06:37:02.226569+0000","last_fullsized":"2024-03-24T06:37:02.226569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.381945+0000","last_change":"2024-03-24T06:36:55.728909+0000","last_active":"2024-03-24T06:37:29.381945+0000","last_peered":"2024-03-24T06:37:29.381945+0000","last_clean":"2024-03-24T06:37:29.381945+0000","last_became_active":"2024-03-24T06:36:55.727728+0000","last_became_peered":"2024-03-24T06:36:55.727728+0000","last_unstale":"2024-03-24T06:37:29.381945+0000","last_undegraded":"2024-03-24T06:37:29.381945+0000","last_fullsized":"2024-03-24T06:37:29.381945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:51.039212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554686+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:37:02.554686+0000","last_peered":"2024-03-24T06:37:02.554686+0000","last_clean":"2024-03-24T06:37:02.554686+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:37:02.554686+0000","last_undegraded":"2024-03-24T06:37:02.554686+0000","last_fullsized":"2024-03-24T06:37:02.554686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226196+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:37:02.226196+0000","last_peered":"2024-03-24T06:37:02.226196+0000","last_clean":"2024-03-24T06:37:02.226196+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:37:02.226196+0000","last_undegraded":"2024-03-24T06:37:02.226196+0000","last_fullsized":"2024-03-24T06:37:02.226196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.409514+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:37:31.409514+0000","last_peered":"2024-03-24T06:37:31.409514+0000","last_clean":"2024-03-24T06:37:31.409514+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:37:31.409514+0000","last_undegraded":"2024-03-24T06:37:31.409514+0000","last_fullsized":"2024-03-24T06:37:31.409514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555962+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:37:02.555962+0000","last_peered":"2024-03-24T06:37:02.555962+0000","last_clean":"2024-03-24T06:37:02.555962+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:37:02.555962+0000","last_undegraded":"2024-03-24T06:37:02.555962+0000","last_fullsized":"2024-03-24T06:37:02.555962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555846+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:37:02.555846+0000","last_peered":"2024-03-24T06:37:02.555846+0000","last_clean":"2024-03-24T06:37:02.555846+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:37:02.555846+0000","last_undegraded":"2024-03-24T06:37:02.555846+0000","last_fullsized":"2024-03-24T06:37:02.555846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.401660+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:37:30.401660+0000","last_peered":"2024-03-24T06:37:30.401660+0000","last_clean":"2024-03-24T06:37:30.401660+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:37:30.401660+0000","last_undegraded":"2024-03-24T06:37:30.401660+0000","last_fullsized":"2024-03-24T06:37:30.401660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320345+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:37:02.320345+0000","last_peered":"2024-03-24T06:37:02.320345+0000","last_clean":"2024-03-24T06:37:02.320345+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:37:02.320345+0000","last_undegraded":"2024-03-24T06:37:02.320345+0000","last_fullsized":"2024-03-24T06:37:02.320345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814393+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:37:02.814393+0000","last_peered":"2024-03-24T06:37:02.814393+0000","last_clean":"2024-03-24T06:37:02.814393+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:37:02.814393+0000","last_undegraded":"2024-03-24T06:37:02.814393+0000","last_fullsized":"2024-03-24T06:37:02.814393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.405471+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:37:31.405471+0000","last_peered":"2024-03-24T06:37:31.405471+0000","last_clean":"2024-03-24T06:37:31.405471+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:37:31.405471+0000","last_undegraded":"2024-03-24T06:37:31.405471+0000","last_fullsized":"2024-03-24T06:37:31.405471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.229222+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:37:02.229222+0000","last_peered":"2024-03-24T06:37:02.229222+0000","last_clean":"2024-03-24T06:37:02.229222+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:37:02.229222+0000","last_undegraded":"2024-03-24T06:37:02.229222+0000","last_fullsized":"2024-03-24T06:37:02.229222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320360+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:37:02.320360+0000","last_peered":"2024-03-24T06:37:02.320360+0000","last_clean":"2024-03-24T06:37:02.320360+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:37:02.320360+0000","last_undegraded":"2024-03-24T06:37:02.320360+0000","last_fullsized":"2024-03-24T06:37:02.320360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.387177+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:37:29.387177+0000","last_peered":"2024-03-24T06:37:29.387177+0000","last_clean":"2024-03-24T06:37:29.387177+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:37:29.387177+0000","last_undegraded":"2024-03-24T06:37:29.387177+0000","last_fullsized":"2024-03-24T06:37:29.387177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555349+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:37:02.555349+0000","last_peered":"2024-03-24T06:37:02.555349+0000","last_clean":"2024-03-24T06:37:02.555349+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:37:02.555349+0000","last_undegraded":"2024-03-24T06:37:02.555349+0000","last_fullsized":"2024-03-24T06:37:02.555349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.229106+0000","last_change":"2024-03-24T06:37:01.261900+0000","last_active":"2024-03-24T06:37:02.229106+0000","last_peered":"2024-03-24T06:37:02.229106+0000","last_clean":"2024-03-24T06:37:02.229106+0000","last_became_active":"2024-03-24T06:37:01.259109+0000","last_became_peered":"2024-03-24T06:37:01.259109+0000","last_unstale":"2024-03-24T06:37:02.229106+0000","last_undegraded":"2024-03-24T06:37:02.229106+0000","last_fullsized":"2024-03-24T06:37:02.229106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:37:37.030003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.413722+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:37:32.413722+0000","last_peered":"2024-03-24T06:37:32.413722+0000","last_clean":"2024-03-24T06:37:32.413722+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:37:32.413722+0000","last_undegraded":"2024-03-24T06:37:32.413722+0000","last_fullsized":"2024-03-24T06:37:32.413722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814780+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:37:02.814780+0000","last_peered":"2024-03-24T06:37:02.814780+0000","last_clean":"2024-03-24T06:37:02.814780+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:37:02.814780+0000","last_undegraded":"2024-03-24T06:37:02.814780+0000","last_fullsized":"2024-03-24T06:37:02.814780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555125+0000","last_change":"2024-03-24T06:37:01.253622+0000","last_active":"2024-03-24T06:37:02.555125+0000","last_peered":"2024-03-24T06:37:02.555125+0000","last_clean":"2024-03-24T06:37:02.555125+0000","last_became_active":"2024-03-24T06:37:01.253160+0000","last_became_peered":"2024-03-24T06:37:01.253160+0000","last_unstale":"2024-03-24T06:37:02.555125+0000","last_undegraded":"2024-03-24T06:37:02.555125+0000","last_fullsized":"2024-03-24T06:37:02.555125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:02.997406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.386240+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:37:29.386240+0000","last_peered":"2024-03-24T06:37:29.386240+0000","last_clean":"2024-03-24T06:37:29.386240+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:37:29.386240+0000","last_undegraded":"2024-03-24T06:37:29.386240+0000","last_fullsized":"2024-03-24T06:37:29.386240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554971+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:37:02.554971+0000","last_peered":"2024-03-24T06:37:02.554971+0000","last_clean":"2024-03-24T06:37:02.554971+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:37:02.554971+0000","last_undegraded":"2024-03-24T06:37:02.554971+0000","last_fullsized":"2024-03-24T06:37:02.554971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.230125+0000","last_change":"2024-03-24T06:37:01.248655+0000","last_active":"2024-03-24T06:37:02.230125+0000","last_peered":"2024-03-24T06:37:02.230125+0000","last_clean":"2024-03-24T06:37:02.230125+0000","last_became_active":"2024-03-24T06:37:01.245755+0000","last_became_peered":"2024-03-24T06:37:01.245755+0000","last_unstale":"2024-03-24T06:37:02.230125+0000","last_undegraded":"2024-03-24T06:37:02.230125+0000","last_fullsized":"2024-03-24T06:37:02.230125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:29.632351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'22","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.415247+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:37:32.415247+0000","last_peered":"2024-03-24T06:37:32.415247+0000","last_clean":"2024-03-24T06:37:32.415247+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:37:32.415247+0000","last_undegraded":"2024-03-24T06:37:32.415247+0000","last_fullsized":"2024-03-24T06:37:32.415247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555073+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:37:02.555073+0000","last_peered":"2024-03-24T06:37:02.555073+0000","last_clean":"2024-03-24T06:37:02.555073+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:37:02.555073+0000","last_undegraded":"2024-03-24T06:37:02.555073+0000","last_fullsized":"2024-03-24T06:37:02.555073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556714+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:37:02.556714+0000","last_peered":"2024-03-24T06:37:02.556714+0000","last_clean":"2024-03-24T06:37:02.556714+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:37:02.556714+0000","last_undegraded":"2024-03-24T06:37:02.556714+0000","last_fullsized":"2024-03-24T06:37:02.556714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.435468+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:37:33.435468+0000","last_peered":"2024-03-24T06:37:33.435468+0000","last_clean":"2024-03-24T06:37:33.435468+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:37:33.435468+0000","last_undegraded":"2024-03-24T06:37:33.435468+0000","last_fullsized":"2024-03-24T06:37:33.435468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557664+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:37:02.557664+0000","last_peered":"2024-03-24T06:37:02.557664+0000","last_clean":"2024-03-24T06:37:02.557664+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:37:02.557664+0000","last_undegraded":"2024-03-24T06:37:02.557664+0000","last_fullsized":"2024-03-24T06:37:02.557664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555285+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:37:02.555285+0000","last_peered":"2024-03-24T06:37:02.555285+0000","last_clean":"2024-03-24T06:37:02.555285+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:37:02.555285+0000","last_undegraded":"2024-03-24T06:37:02.555285+0000","last_fullsized":"2024-03-24T06:37:02.555285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.416184+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:37:31.416184+0000","last_peered":"2024-03-24T06:37:31.416184+0000","last_clean":"2024-03-24T06:37:31.416184+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:37:31.416184+0000","last_undegraded":"2024-03-24T06:37:31.416184+0000","last_fullsized":"2024-03-24T06:37:31.416184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225683+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:37:02.225683+0000","last_peered":"2024-03-24T06:37:02.225683+0000","last_clean":"2024-03-24T06:37:02.225683+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:37:02.225683+0000","last_undegraded":"2024-03-24T06:37:02.225683+0000","last_fullsized":"2024-03-24T06:37:02.225683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.231831+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:37:02.231831+0000","last_peered":"2024-03-24T06:37:02.231831+0000","last_clean":"2024-03-24T06:37:02.231831+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:37:02.231831+0000","last_undegraded":"2024-03-24T06:37:02.231831+0000","last_fullsized":"2024-03-24T06:37:02.231831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.404498+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:37:31.404498+0000","last_peered":"2024-03-24T06:37:31.404498+0000","last_clean":"2024-03-24T06:37:31.404498+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:37:31.404498+0000","last_undegraded":"2024-03-24T06:37:31.404498+0000","last_fullsized":"2024-03-24T06:37:31.404498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225434+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:37:02.225434+0000","last_peered":"2024-03-24T06:37:02.225434+0000","last_clean":"2024-03-24T06:37:02.225434+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:37:02.225434+0000","last_undegraded":"2024-03-24T06:37:02.225434+0000","last_fullsized":"2024-03-24T06:37:02.225434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557526+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:37:02.557526+0000","last_peered":"2024-03-24T06:37:02.557526+0000","last_clean":"2024-03-24T06:37:02.557526+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:37:02.557526+0000","last_undegraded":"2024-03-24T06:37:02.557526+0000","last_fullsized":"2024-03-24T06:37:02.557526+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.383520+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:37:29.383520+0000","last_peered":"2024-03-24T06:37:29.383520+0000","last_clean":"2024-03-24T06:37:29.383520+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:37:29.383520+0000","last_undegraded":"2024-03-24T06:37:29.383520+0000","last_fullsized":"2024-03-24T06:37:29.383520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555514+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:37:02.555514+0000","last_peered":"2024-03-24T06:37:02.555514+0000","last_clean":"2024-03-24T06:37:02.555514+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:37:02.555514+0000","last_undegraded":"2024-03-24T06:37:02.555514+0000","last_fullsized":"2024-03-24T06:37:02.555514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555452+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:37:02.555452+0000","last_peered":"2024-03-24T06:37:02.555452+0000","last_clean":"2024-03-24T06:37:02.555452+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:37:02.555452+0000","last_undegraded":"2024-03-24T06:37:02.555452+0000","last_fullsized":"2024-03-24T06:37:02.555452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555185+0000","last_change":"2024-03-24T06:36:55.722575+0000","last_active":"2024-03-24T06:37:02.555185+0000","last_peered":"2024-03-24T06:37:02.555185+0000","last_clean":"2024-03-24T06:37:02.555185+0000","last_became_active":"2024-03-24T06:36:55.722249+0000","last_became_peered":"2024-03-24T06:36:55.722249+0000","last_unstale":"2024-03-24T06:37:02.555185+0000","last_undegraded":"2024-03-24T06:37:02.555185+0000","last_fullsized":"2024-03-24T06:37:02.555185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.905215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555185+0000","last_change":"2024-03-24T06:36:56.750708+0000","last_active":"2024-03-24T06:37:02.555185+0000","last_peered":"2024-03-24T06:37:02.555185+0000","last_clean":"2024-03-24T06:37:02.555185+0000","last_became_active":"2024-03-24T06:36:56.750391+0000","last_became_peered":"2024-03-24T06:36:56.750391+0000","last_unstale":"2024-03-24T06:37:02.555185+0000","last_undegraded":"2024-03-24T06:37:02.555185+0000","last_fullsized":"2024-03-24T06:37:02.555185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:31.577213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554800+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:37:02.554800+0000","last_peered":"2024-03-24T06:37:02.554800+0000","last_clean":"2024-03-24T06:37:02.554800+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:37:02.554800+0000","last_undegraded":"2024-03-24T06:37:02.554800+0000","last_fullsized":"2024-03-24T06:37:02.554800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.230389+0000","last_change":"2024-03-24T06:36:55.738736+0000","last_active":"2024-03-24T06:37:02.230389+0000","last_peered":"2024-03-24T06:37:02.230389+0000","last_clean":"2024-03-24T06:37:02.230389+0000","last_became_active":"2024-03-24T06:36:55.738313+0000","last_became_peered":"2024-03-24T06:36:55.738313+0000","last_unstale":"2024-03-24T06:37:02.230389+0000","last_undegraded":"2024-03-24T06:37:02.230389+0000","last_fullsized":"2024-03-24T06:37:02.230389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:39.785983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.313098+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:37:02.313098+0000","last_peered":"2024-03-24T06:37:02.313098+0000","last_clean":"2024-03-24T06:37:02.313098+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:37:02.313098+0000","last_undegraded":"2024-03-24T06:37:02.313098+0000","last_fullsized":"2024-03-24T06:37:02.313098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557108+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:37:02.557108+0000","last_peered":"2024-03-24T06:37:02.557108+0000","last_clean":"2024-03-24T06:37:02.557108+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:37:02.557108+0000","last_undegraded":"2024-03-24T06:37:02.557108+0000","last_fullsized":"2024-03-24T06:37:02.557108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.438423+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:37:33.438423+0000","last_peered":"2024-03-24T06:37:33.438423+0000","last_clean":"2024-03-24T06:37:33.438423+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:37:33.438423+0000","last_undegraded":"2024-03-24T06:37:33.438423+0000","last_fullsized":"2024-03-24T06:37:33.438423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.312929+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:37:02.312929+0000","last_peered":"2024-03-24T06:37:02.312929+0000","last_clean":"2024-03-24T06:37:02.312929+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:37:02.312929+0000","last_undegraded":"2024-03-24T06:37:02.312929+0000","last_fullsized":"2024-03-24T06:37:02.312929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557450+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:37:02.557450+0000","last_peered":"2024-03-24T06:37:02.557450+0000","last_clean":"2024-03-24T06:37:02.557450+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:37:02.557450+0000","last_undegraded":"2024-03-24T06:37:02.557450+0000","last_fullsized":"2024-03-24T06:37:02.557450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556229+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:37:02.556229+0000","last_peered":"2024-03-24T06:37:02.556229+0000","last_clean":"2024-03-24T06:37:02.556229+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:37:02.556229+0000","last_undegraded":"2024-03-24T06:37:02.556229+0000","last_fullsized":"2024-03-24T06:37:02.556229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.435790+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:37:33.435790+0000","last_peered":"2024-03-24T06:37:33.435790+0000","last_clean":"2024-03-24T06:37:33.435790+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:37:33.435790+0000","last_undegraded":"2024-03-24T06:37:33.435790+0000","last_fullsized":"2024-03-24T06:37:33.435790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:03.963302+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:37:03.963302+0000","last_peered":"2024-03-24T06:37:03.963302+0000","last_clean":"2024-03-24T06:37:03.963302+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:37:03.963302+0000","last_undegraded":"2024-03-24T06:37:03.963302+0000","last_fullsized":"2024-03-24T06:37:03.963302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320240+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:37:02.320240+0000","last_peered":"2024-03-24T06:37:02.320240+0000","last_clean":"2024-03-24T06:37:02.320240+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:37:02.320240+0000","last_undegraded":"2024-03-24T06:37:02.320240+0000","last_fullsized":"2024-03-24T06:37:02.320240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.411713+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:37:31.411713+0000","last_peered":"2024-03-24T06:37:31.411713+0000","last_clean":"2024-03-24T06:37:31.411713+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:37:31.411713+0000","last_undegraded":"2024-03-24T06:37:31.411713+0000","last_fullsized":"2024-03-24T06:37:31.411713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225309+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:37:02.225309+0000","last_peered":"2024-03-24T06:37:02.225309+0000","last_clean":"2024-03-24T06:37:02.225309+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:37:02.225309+0000","last_undegraded":"2024-03-24T06:37:02.225309+0000","last_fullsized":"2024-03-24T06:37:02.225309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556434+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:37:02.556434+0000","last_peered":"2024-03-24T06:37:02.556434+0000","last_clean":"2024-03-24T06:37:02.556434+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:37:02.556434+0000","last_undegraded":"2024-03-24T06:37:02.556434+0000","last_fullsized":"2024-03-24T06:37:02.556434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556260+0000","last_change":"2024-03-24T06:36:56.764415+0000","last_active":"2024-03-24T06:37:02.556260+0000","last_peered":"2024-03-24T06:37:02.556260+0000","last_clean":"2024-03-24T06:37:02.556260+0000","last_became_active":"2024-03-24T06:36:56.762075+0000","last_became_peered":"2024-03-24T06:36:56.762075+0000","last_unstale":"2024-03-24T06:37:02.556260+0000","last_undegraded":"2024-03-24T06:37:02.556260+0000","last_fullsized":"2024-03-24T06:37:02.556260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:23.642005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.392497+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:37:30.392497+0000","last_peered":"2024-03-24T06:37:30.392497+0000","last_clean":"2024-03-24T06:37:30.392497+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:37:30.392497+0000","last_undegraded":"2024-03-24T06:37:30.392497+0000","last_fullsized":"2024-03-24T06:37:30.392497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555338+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:37:02.555338+0000","last_peered":"2024-03-24T06:37:02.555338+0000","last_clean":"2024-03-24T06:37:02.555338+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:37:02.555338+0000","last_undegraded":"2024-03-24T06:37:02.555338+0000","last_fullsized":"2024-03-24T06:37:02.555338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555242+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:37:02.555242+0000","last_peered":"2024-03-24T06:37:02.555242+0000","last_clean":"2024-03-24T06:37:02.555242+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:37:02.555242+0000","last_undegraded":"2024-03-24T06:37:02.555242+0000","last_fullsized":"2024-03-24T06:37:02.555242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557007+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:37:02.557007+0000","last_peered":"2024-03-24T06:37:02.557007+0000","last_clean":"2024-03-24T06:37:02.557007+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:37:02.557007+0000","last_undegraded":"2024-03-24T06:37:02.557007+0000","last_fullsized":"2024-03-24T06:37:02.557007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320516+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:37:02.320516+0000","last_peered":"2024-03-24T06:37:02.320516+0000","last_clean":"2024-03-24T06:37:02.320516+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:37:02.320516+0000","last_undegraded":"2024-03-24T06:37:02.320516+0000","last_fullsized":"2024-03-24T06:37:02.320516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.398616+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:37:30.398616+0000","last_peered":"2024-03-24T06:37:30.398616+0000","last_clean":"2024-03-24T06:37:30.398616+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:37:30.398616+0000","last_undegraded":"2024-03-24T06:37:30.398616+0000","last_fullsized":"2024-03-24T06:37:30.398616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320662+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:37:02.320662+0000","last_peered":"2024-03-24T06:37:02.320662+0000","last_clean":"2024-03-24T06:37:02.320662+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:37:02.320662+0000","last_undegraded":"2024-03-24T06:37:02.320662+0000","last_fullsized":"2024-03-24T06:37:02.320662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226196+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:37:02.226196+0000","last_peered":"2024-03-24T06:37:02.226196+0000","last_clean":"2024-03-24T06:37:02.226196+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:37:02.226196+0000","last_undegraded":"2024-03-24T06:37:02.226196+0000","last_fullsized":"2024-03-24T06:37:02.226196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.419322+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:37:32.419322+0000","last_peered":"2024-03-24T06:37:32.419322+0000","last_clean":"2024-03-24T06:37:32.419322+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:37:32.419322+0000","last_undegraded":"2024-03-24T06:37:32.419322+0000","last_fullsized":"2024-03-24T06:37:32.419322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814721+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:37:02.814721+0000","last_peered":"2024-03-24T06:37:02.814721+0000","last_clean":"2024-03-24T06:37:02.814721+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:37:02.814721+0000","last_undegraded":"2024-03-24T06:37:02.814721+0000","last_fullsized":"2024-03-24T06:37:02.814721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555236+0000","last_change":"2024-03-24T06:37:01.256625+0000","last_active":"2024-03-24T06:37:02.555236+0000","last_peered":"2024-03-24T06:37:02.555236+0000","last_clean":"2024-03-24T06:37:02.555236+0000","last_became_active":"2024-03-24T06:37:01.256152+0000","last_became_peered":"2024-03-24T06:37:01.256152+0000","last_unstale":"2024-03-24T06:37:02.555236+0000","last_undegraded":"2024-03-24T06:37:02.555236+0000","last_fullsized":"2024-03-24T06:37:02.555236+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:34.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.383711+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:37:29.383711+0000","last_peered":"2024-03-24T06:37:29.383711+0000","last_clean":"2024-03-24T06:37:29.383711+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:37:29.383711+0000","last_undegraded":"2024-03-24T06:37:29.383711+0000","last_fullsized":"2024-03-24T06:37:29.383711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554418+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:37:02.554418+0000","last_peered":"2024-03-24T06:37:02.554418+0000","last_clean":"2024-03-24T06:37:02.554418+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:37:02.554418+0000","last_undegraded":"2024-03-24T06:37:02.554418+0000","last_fullsized":"2024-03-24T06:37:02.554418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.232499+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:37:02.232499+0000","last_peered":"2024-03-24T06:37:02.232499+0000","last_clean":"2024-03-24T06:37:02.232499+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:37:02.232499+0000","last_undegraded":"2024-03-24T06:37:02.232499+0000","last_fullsized":"2024-03-24T06:37:02.232499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.387757+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:37:29.387757+0000","last_peered":"2024-03-24T06:37:29.387757+0000","last_clean":"2024-03-24T06:37:29.387757+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:37:29.387757+0000","last_undegraded":"2024-03-24T06:37:29.387757+0000","last_fullsized":"2024-03-24T06:37:29.387757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555473+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:37:02.555473+0000","last_peered":"2024-03-24T06:37:02.555473+0000","last_clean":"2024-03-24T06:37:02.555473+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:37:02.555473+0000","last_undegraded":"2024-03-24T06:37:02.555473+0000","last_fullsized":"2024-03-24T06:37:02.555473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554881+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:37:02.554881+0000","last_peered":"2024-03-24T06:37:02.554881+0000","last_clean":"2024-03-24T06:37:02.554881+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:37:02.554881+0000","last_undegraded":"2024-03-24T06:37:02.554881+0000","last_fullsized":"2024-03-24T06:37:02.554881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.412826+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:37:31.412826+0000","last_peered":"2024-03-24T06:37:31.412826+0000","last_clean":"2024-03-24T06:37:31.412826+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:37:31.412826+0000","last_undegraded":"2024-03-24T06:37:31.412826+0000","last_fullsized":"2024-03-24T06:37:31.412826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320962+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:37:02.320962+0000","last_peered":"2024-03-24T06:37:02.320962+0000","last_clean":"2024-03-24T06:37:02.320962+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:37:02.320962+0000","last_undegraded":"2024-03-24T06:37:02.320962+0000","last_fullsized":"2024-03-24T06:37:02.320962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554972+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:37:02.554972+0000","last_peered":"2024-03-24T06:37:02.554972+0000","last_clean":"2024-03-24T06:37:02.554972+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:37:02.554972+0000","last_undegraded":"2024-03-24T06:37:02.554972+0000","last_fullsized":"2024-03-24T06:37:02.554972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.381605+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:37:29.381605+0000","last_peered":"2024-03-24T06:37:29.381605+0000","last_clean":"2024-03-24T06:37:29.381605+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:37:29.381605+0000","last_undegraded":"2024-03-24T06:37:29.381605+0000","last_fullsized":"2024-03-24T06:37:29.381605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225603+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:37:02.225603+0000","last_peered":"2024-03-24T06:37:02.225603+0000","last_clean":"2024-03-24T06:37:02.225603+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:37:02.225603+0000","last_undegraded":"2024-03-24T06:37:02.225603+0000","last_fullsized":"2024-03-24T06:37:02.225603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554398+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:37:02.554398+0000","last_peered":"2024-03-24T06:37:02.554398+0000","last_clean":"2024-03-24T06:37:02.554398+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:37:02.554398+0000","last_undegraded":"2024-03-24T06:37:02.554398+0000","last_fullsized":"2024-03-24T06:37:02.554398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.440381+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:37:33.440381+0000","last_peered":"2024-03-24T06:37:33.440381+0000","last_clean":"2024-03-24T06:37:33.440381+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:37:33.440381+0000","last_undegraded":"2024-03-24T06:37:33.440381+0000","last_fullsized":"2024-03-24T06:37:33.440381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225921+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:37:02.225921+0000","last_peered":"2024-03-24T06:37:02.225921+0000","last_clean":"2024-03-24T06:37:02.225921+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:37:02.225921+0000","last_undegraded":"2024-03-24T06:37:02.225921+0000","last_fullsized":"2024-03-24T06:37:02.225921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814330+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:37:02.814330+0000","last_peered":"2024-03-24T06:37:02.814330+0000","last_clean":"2024-03-24T06:37:02.814330+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:37:02.814330+0000","last_undegraded":"2024-03-24T06:37:02.814330+0000","last_fullsized":"2024-03-24T06:37:02.814330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.412841+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:37:31.412841+0000","last_peered":"2024-03-24T06:37:31.412841+0000","last_clean":"2024-03-24T06:37:31.412841+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:37:31.412841+0000","last_undegraded":"2024-03-24T06:37:31.412841+0000","last_fullsized":"2024-03-24T06:37:31.412841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225478+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:37:02.225478+0000","last_peered":"2024-03-24T06:37:02.225478+0000","last_clean":"2024-03-24T06:37:02.225478+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:37:02.225478+0000","last_undegraded":"2024-03-24T06:37:02.225478+0000","last_fullsized":"2024-03-24T06:37:02.225478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556114+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:37:02.556114+0000","last_peered":"2024-03-24T06:37:02.556114+0000","last_clean":"2024-03-24T06:37:02.556114+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:37:02.556114+0000","last_undegraded":"2024-03-24T06:37:02.556114+0000","last_fullsized":"2024-03-24T06:37:02.556114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.420275+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:37:32.420275+0000","last_peered":"2024-03-24T06:37:32.420275+0000","last_clean":"2024-03-24T06:37:32.420275+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:37:32.420275+0000","last_undegraded":"2024-03-24T06:37:32.420275+0000","last_fullsized":"2024-03-24T06:37:32.420275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225904+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:37:02.225904+0000","last_peered":"2024-03-24T06:37:02.225904+0000","last_clean":"2024-03-24T06:37:02.225904+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:37:02.225904+0000","last_undegraded":"2024-03-24T06:37:02.225904+0000","last_fullsized":"2024-03-24T06:37:02.225904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555567+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:37:02.555567+0000","last_peered":"2024-03-24T06:37:02.555567+0000","last_clean":"2024-03-24T06:37:02.555567+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:37:02.555567+0000","last_undegraded":"2024-03-24T06:37:02.555567+0000","last_fullsized":"2024-03-24T06:37:02.555567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226394+0000","last_change":"2024-03-24T06:36:56.750417+0000","last_active":"2024-03-24T06:37:02.226394+0000","last_peered":"2024-03-24T06:37:02.226394+0000","last_clean":"2024-03-24T06:37:02.226394+0000","last_became_active":"2024-03-24T06:36:56.750024+0000","last_became_peered":"2024-03-24T06:36:56.750024+0000","last_unstale":"2024-03-24T06:37:02.226394+0000","last_undegraded":"2024-03-24T06:37:02.226394+0000","last_fullsized":"2024-03-24T06:37:02.226394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:16:14.426791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225663+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:37:02.225663+0000","last_peered":"2024-03-24T06:37:02.225663+0000","last_clean":"2024-03-24T06:37:02.225663+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:37:02.225663+0000","last_undegraded":"2024-03-24T06:37:02.225663+0000","last_fullsized":"2024-03-24T06:37:02.225663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814300+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:37:02.814300+0000","last_peered":"2024-03-24T06:37:02.814300+0000","last_clean":"2024-03-24T06:37:02.814300+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:37:02.814300+0000","last_undegraded":"2024-03-24T06:37:02.814300+0000","last_fullsized":"2024-03-24T06:37:02.814300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557145+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:37:02.557145+0000","last_peered":"2024-03-24T06:37:02.557145+0000","last_clean":"2024-03-24T06:37:02.557145+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:37:02.557145+0000","last_undegraded":"2024-03-24T06:37:02.557145+0000","last_fullsized":"2024-03-24T06:37:02.557145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555212+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:37:02.555212+0000","last_peered":"2024-03-24T06:37:02.555212+0000","last_clean":"2024-03-24T06:37:02.555212+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:37:02.555212+0000","last_undegraded":"2024-03-24T06:37:02.555212+0000","last_fullsized":"2024-03-24T06:37:02.555212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226265+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:37:02.226265+0000","last_peered":"2024-03-24T06:37:02.226265+0000","last_clean":"2024-03-24T06:37:02.226265+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:37:02.226265+0000","last_undegraded":"2024-03-24T06:37:02.226265+0000","last_fullsized":"2024-03-24T06:37:02.226265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320907+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:37:02.320907+0000","last_peered":"2024-03-24T06:37:02.320907+0000","last_clean":"2024-03-24T06:37:02.320907+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:37:02.320907+0000","last_undegraded":"2024-03-24T06:37:02.320907+0000","last_fullsized":"2024-03-24T06:37:02.320907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556224+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:37:02.556224+0000","last_peered":"2024-03-24T06:37:02.556224+0000","last_clean":"2024-03-24T06:37:02.556224+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:37:02.556224+0000","last_undegraded":"2024-03-24T06:37:02.556224+0000","last_fullsized":"2024-03-24T06:37:02.556224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320414+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:37:02.320414+0000","last_peered":"2024-03-24T06:37:02.320414+0000","last_clean":"2024-03-24T06:37:02.320414+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:37:02.320414+0000","last_undegraded":"2024-03-24T06:37:02.320414+0000","last_fullsized":"2024-03-24T06:37:02.320414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554524+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:37:02.554524+0000","last_peered":"2024-03-24T06:37:02.554524+0000","last_clean":"2024-03-24T06:37:02.554524+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:37:02.554524+0000","last_undegraded":"2024-03-24T06:37:02.554524+0000","last_fullsized":"2024-03-24T06:37:02.554524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556825+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:37:02.556825+0000","last_peered":"2024-03-24T06:37:02.556825+0000","last_clean":"2024-03-24T06:37:02.556825+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:37:02.556825+0000","last_undegraded":"2024-03-24T06:37:02.556825+0000","last_fullsized":"2024-03-24T06:37:02.556825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555349+0000","last_change":"2024-03-24T06:36:56.755968+0000","last_active":"2024-03-24T06:37:02.555349+0000","last_peered":"2024-03-24T06:37:02.555349+0000","last_clean":"2024-03-24T06:37:02.555349+0000","last_became_active":"2024-03-24T06:36:56.755651+0000","last_became_peered":"2024-03-24T06:36:56.755651+0000","last_unstale":"2024-03-24T06:37:02.555349+0000","last_undegraded":"2024-03-24T06:37:02.555349+0000","last_fullsized":"2024-03-24T06:37:02.555349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:00.246108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28689,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":174,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":82944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1079727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1074821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":478603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":519658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1074499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:37:35.326 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T06:37:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 -- 172.21.15.22:0/2987954727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 msgr2=0x7ff078100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2987954727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078100a70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff068009900 tx=0x7ff06802bc30 comp rx=0 tx=0).stop 2024-03-24T06:37:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 -- 172.21.15.22:0/2987954727 shutdown_connections 2024-03-24T06:37:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2987954727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff078108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2987954727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff078101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2987954727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078100a70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 -- 172.21.15.22:0/2987954727 >> 172.21.15.22:0/2987954727 conn(0x7ff0780fbbc0 msgr2=0x7ff0780fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 -- 172.21.15.22:0/2987954727 shutdown_connections 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.609+0000 7ff07ea00640 1 -- 172.21.15.22:0/2987954727 wait complete. 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 Processor -- start 2024-03-24T06:37:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 -- start start 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff07819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff0781a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781a15e0 con 0x7ff078102200 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781a1750 con 0x7ff078100650 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.611+0000 7ff07ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0781a1910 con 0x7ff078101840 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff07819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46768/0 (socket says 172.21.15.22:46768) 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 -- 172.21.15.22:0/2056604178 learned_addr learned my addr 172.21.15.22:0/2056604178 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff07ca00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff0781a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 -- 172.21.15.22:0/2056604178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 msgr2=0x7ff07819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff07819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 -- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 msgr2=0x7ff0781a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff0781a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 -- 172.21.15.22:0/2056604178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff068009560 con 0x7ff078100650 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077400640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff07819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff07ca00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff0781a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.612+0000 7ff077e00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff068000c00 tx=0x7ff0680040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.613+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0680044c0 con 0x7ff078100650 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.613+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0680052d0 con 0x7ff078100650 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.613+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068003e80 con 0x7ff078100650 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.613+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0781a1b90 con 0x7ff078100650 2024-03-24T06:37:35.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.613+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0781a2050 con 0x7ff078100650 2024-03-24T06:37:35.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.614+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07810c0b0 con 0x7ff078100650 2024-03-24T06:37:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.617+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff0680035f0 con 0x7ff078100650 2024-03-24T06:37:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.619+0000 7ff074c00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 0x7ff05c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.619+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff0680f08d0 con 0x7ff078100650 2024-03-24T06:37:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.619+0000 7ff077400640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 0x7ff05c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.620+0000 7ff077400640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 0x7ff05c0a4690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff07819b000 tx=0x7ff064005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.623+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff0680b96d0 con 0x7ff078100650 2024-03-24T06:37:35.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:35 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/337547081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:35.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:35 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/337547081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:35 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/337547081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:35.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.822+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff07806a210 con 0x7ff05c0a21d0 2024-03-24T06:37:35.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.854+0000 7ff074c00640 1 -- 172.21.15.22:0/2056604178 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340407 (secure 0 0 0) 0x7ff07806a210 con 0x7ff05c0a21d0 2024-03-24T06:37:35.859 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:35.861 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 msgr2=0x7ff05c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 0x7ff05c0a4690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff07819b000 tx=0x7ff064005f90 comp rx=0 tx=0).stop 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 msgr2=0x7ff078199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff068000c00 tx=0x7ff0680040c0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 shutdown_connections 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff05c0a21d0 0x7ff05c0a4690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff078102200 0x7ff0781a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff078101840 0x7ff07819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 --2- 172.21.15.22:0/2056604178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff078100650 0x7ff078199b40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:35.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 >> 172.21.15.22:0/2056604178 conn(0x7ff0780fbbc0 msgr2=0x7ff078069b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:35.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 shutdown_connections 2024-03-24T06:37:35.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:35.861+0000 7ff07ea00640 1 -- 172.21.15.22:0/2056604178 wait complete. 2024-03-24T06:37:35.880 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-24T06:37:35.104511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":619174,"num_objects":81,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":146,"num_read_kb":126,"num_write":356,"num_write_kb":3018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":292,"ondisk_log_size":292,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383564,"kb_used_data":13612,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558260,"statfs":{"total":1151940427776,"available":1151547658240,"internally_reserved":0,"allocated":13938688,"data_stored":7580025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24166,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":126,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006170"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555946+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:37:02.555946+0000","last_peered":"2024-03-24T06:37:02.555946+0000","last_clean":"2024-03-24T06:37:02.555946+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:37:02.555946+0000","last_undegraded":"2024-03-24T06:37:02.555946+0000","last_fullsized":"2024-03-24T06:37:02.555946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814602+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:37:02.814602+0000","last_peered":"2024-03-24T06:37:02.814602+0000","last_clean":"2024-03-24T06:37:02.814602+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:37:02.814602+0000","last_undegraded":"2024-03-24T06:37:02.814602+0000","last_fullsized":"2024-03-24T06:37:02.814602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814499+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:37:02.814499+0000","last_peered":"2024-03-24T06:37:02.814499+0000","last_clean":"2024-03-24T06:37:02.814499+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:37:02.814499+0000","last_undegraded":"2024-03-24T06:37:02.814499+0000","last_fullsized":"2024-03-24T06:37:02.814499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555114+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:37:02.555114+0000","last_peered":"2024-03-24T06:37:02.555114+0000","last_clean":"2024-03-24T06:37:02.555114+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:37:02.555114+0000","last_undegraded":"2024-03-24T06:37:02.555114+0000","last_fullsized":"2024-03-24T06:37:02.555114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554643+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:37:02.554643+0000","last_peered":"2024-03-24T06:37:02.554643+0000","last_clean":"2024-03-24T06:37:02.554643+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:37:02.554643+0000","last_undegraded":"2024-03-24T06:37:02.554643+0000","last_fullsized":"2024-03-24T06:37:02.554643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.312905+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:37:02.312905+0000","last_peered":"2024-03-24T06:37:02.312905+0000","last_clean":"2024-03-24T06:37:02.312905+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:37:02.312905+0000","last_undegraded":"2024-03-24T06:37:02.312905+0000","last_fullsized":"2024-03-24T06:37:02.312905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225360+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:37:02.225360+0000","last_peered":"2024-03-24T06:37:02.225360+0000","last_clean":"2024-03-24T06:37:02.225360+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:37:02.225360+0000","last_undegraded":"2024-03-24T06:37:02.225360+0000","last_fullsized":"2024-03-24T06:37:02.225360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.319870+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:37:02.319870+0000","last_peered":"2024-03-24T06:37:02.319870+0000","last_clean":"2024-03-24T06:37:02.319870+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:37:02.319870+0000","last_undegraded":"2024-03-24T06:37:02.319870+0000","last_fullsized":"2024-03-24T06:37:02.319870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555424+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:37:02.555424+0000","last_peered":"2024-03-24T06:37:02.555424+0000","last_clean":"2024-03-24T06:37:02.555424+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:37:02.555424+0000","last_undegraded":"2024-03-24T06:37:02.555424+0000","last_fullsized":"2024-03-24T06:37:02.555424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.313331+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:37:02.313331+0000","last_peered":"2024-03-24T06:37:02.313331+0000","last_clean":"2024-03-24T06:37:02.313331+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:37:02.313331+0000","last_undegraded":"2024-03-24T06:37:02.313331+0000","last_fullsized":"2024-03-24T06:37:02.313331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225858+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:37:02.225858+0000","last_peered":"2024-03-24T06:37:02.225858+0000","last_clean":"2024-03-24T06:37:02.225858+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:37:02.225858+0000","last_undegraded":"2024-03-24T06:37:02.225858+0000","last_fullsized":"2024-03-24T06:37:02.225858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555016+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:37:02.555016+0000","last_peered":"2024-03-24T06:37:02.555016+0000","last_clean":"2024-03-24T06:37:02.555016+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:37:02.555016+0000","last_undegraded":"2024-03-24T06:37:02.555016+0000","last_fullsized":"2024-03-24T06:37:02.555016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556912+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:37:02.556912+0000","last_peered":"2024-03-24T06:37:02.556912+0000","last_clean":"2024-03-24T06:37:02.556912+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:37:02.556912+0000","last_undegraded":"2024-03-24T06:37:02.556912+0000","last_fullsized":"2024-03-24T06:37:02.556912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814261+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:37:02.814261+0000","last_peered":"2024-03-24T06:37:02.814261+0000","last_clean":"2024-03-24T06:37:02.814261+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:37:02.814261+0000","last_undegraded":"2024-03-24T06:37:02.814261+0000","last_fullsized":"2024-03-24T06:37:02.814261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814127+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:37:02.814127+0000","last_peered":"2024-03-24T06:37:02.814127+0000","last_clean":"2024-03-24T06:37:02.814127+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:37:02.814127+0000","last_undegraded":"2024-03-24T06:37:02.814127+0000","last_fullsized":"2024-03-24T06:37:02.814127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554191+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:37:02.554191+0000","last_peered":"2024-03-24T06:37:02.554191+0000","last_clean":"2024-03-24T06:37:02.554191+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:37:02.554191+0000","last_undegraded":"2024-03-24T06:37:02.554191+0000","last_fullsized":"2024-03-24T06:37:02.554191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555915+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:37:02.555915+0000","last_peered":"2024-03-24T06:37:02.555915+0000","last_clean":"2024-03-24T06:37:02.555915+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:37:02.555915+0000","last_undegraded":"2024-03-24T06:37:02.555915+0000","last_fullsized":"2024-03-24T06:37:02.555915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.231489+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:37:02.231489+0000","last_peered":"2024-03-24T06:37:02.231489+0000","last_clean":"2024-03-24T06:37:02.231489+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:37:02.231489+0000","last_undegraded":"2024-03-24T06:37:02.231489+0000","last_fullsized":"2024-03-24T06:37:02.231489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226603+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:37:02.226603+0000","last_peered":"2024-03-24T06:37:02.226603+0000","last_clean":"2024-03-24T06:37:02.226603+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:37:02.226603+0000","last_undegraded":"2024-03-24T06:37:02.226603+0000","last_fullsized":"2024-03-24T06:37:02.226603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226204+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:37:02.226204+0000","last_peered":"2024-03-24T06:37:02.226204+0000","last_clean":"2024-03-24T06:37:02.226204+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:37:02.226204+0000","last_undegraded":"2024-03-24T06:37:02.226204+0000","last_fullsized":"2024-03-24T06:37:02.226204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.400225+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:37:30.400225+0000","last_peered":"2024-03-24T06:37:30.400225+0000","last_clean":"2024-03-24T06:37:30.400225+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:37:30.400225+0000","last_undegraded":"2024-03-24T06:37:30.400225+0000","last_fullsized":"2024-03-24T06:37:30.400225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320610+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:37:02.320610+0000","last_peered":"2024-03-24T06:37:02.320610+0000","last_clean":"2024-03-24T06:37:02.320610+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:37:02.320610+0000","last_undegraded":"2024-03-24T06:37:02.320610+0000","last_fullsized":"2024-03-24T06:37:02.320610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554503+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:37:02.554503+0000","last_peered":"2024-03-24T06:37:02.554503+0000","last_clean":"2024-03-24T06:37:02.554503+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:37:02.554503+0000","last_undegraded":"2024-03-24T06:37:02.554503+0000","last_fullsized":"2024-03-24T06:37:02.554503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.321051+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:37:02.321051+0000","last_peered":"2024-03-24T06:37:02.321051+0000","last_clean":"2024-03-24T06:37:02.321051+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:37:02.321051+0000","last_undegraded":"2024-03-24T06:37:02.321051+0000","last_fullsized":"2024-03-24T06:37:02.321051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814865+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:37:02.814865+0000","last_peered":"2024-03-24T06:37:02.814865+0000","last_clean":"2024-03-24T06:37:02.814865+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:37:02.814865+0000","last_undegraded":"2024-03-24T06:37:02.814865+0000","last_fullsized":"2024-03-24T06:37:02.814865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554904+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:37:02.554904+0000","last_peered":"2024-03-24T06:37:02.554904+0000","last_clean":"2024-03-24T06:37:02.554904+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:37:02.554904+0000","last_undegraded":"2024-03-24T06:37:02.554904+0000","last_fullsized":"2024-03-24T06:37:02.554904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555340+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:37:02.555340+0000","last_peered":"2024-03-24T06:37:02.555340+0000","last_clean":"2024-03-24T06:37:02.555340+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:37:02.555340+0000","last_undegraded":"2024-03-24T06:37:02.555340+0000","last_fullsized":"2024-03-24T06:37:02.555340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554690+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:37:02.554690+0000","last_peered":"2024-03-24T06:37:02.554690+0000","last_clean":"2024-03-24T06:37:02.554690+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:37:02.554690+0000","last_undegraded":"2024-03-24T06:37:02.554690+0000","last_fullsized":"2024-03-24T06:37:02.554690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554776+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:37:02.554776+0000","last_peered":"2024-03-24T06:37:02.554776+0000","last_clean":"2024-03-24T06:37:02.554776+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:37:02.554776+0000","last_undegraded":"2024-03-24T06:37:02.554776+0000","last_fullsized":"2024-03-24T06:37:02.554776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555447+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:37:02.555447+0000","last_peered":"2024-03-24T06:37:02.555447+0000","last_clean":"2024-03-24T06:37:02.555447+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:37:02.555447+0000","last_undegraded":"2024-03-24T06:37:02.555447+0000","last_fullsized":"2024-03-24T06:37:02.555447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556207+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:37:02.556207+0000","last_peered":"2024-03-24T06:37:02.556207+0000","last_clean":"2024-03-24T06:37:02.556207+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:37:02.556207+0000","last_undegraded":"2024-03-24T06:37:02.556207+0000","last_fullsized":"2024-03-24T06:37:02.556207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226569+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:37:02.226569+0000","last_peered":"2024-03-24T06:37:02.226569+0000","last_clean":"2024-03-24T06:37:02.226569+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:37:02.226569+0000","last_undegraded":"2024-03-24T06:37:02.226569+0000","last_fullsized":"2024-03-24T06:37:02.226569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.381945+0000","last_change":"2024-03-24T06:36:55.728909+0000","last_active":"2024-03-24T06:37:29.381945+0000","last_peered":"2024-03-24T06:37:29.381945+0000","last_clean":"2024-03-24T06:37:29.381945+0000","last_became_active":"2024-03-24T06:36:55.727728+0000","last_became_peered":"2024-03-24T06:36:55.727728+0000","last_unstale":"2024-03-24T06:37:29.381945+0000","last_undegraded":"2024-03-24T06:37:29.381945+0000","last_fullsized":"2024-03-24T06:37:29.381945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:51.039212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554686+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:37:02.554686+0000","last_peered":"2024-03-24T06:37:02.554686+0000","last_clean":"2024-03-24T06:37:02.554686+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:37:02.554686+0000","last_undegraded":"2024-03-24T06:37:02.554686+0000","last_fullsized":"2024-03-24T06:37:02.554686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226196+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:37:02.226196+0000","last_peered":"2024-03-24T06:37:02.226196+0000","last_clean":"2024-03-24T06:37:02.226196+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:37:02.226196+0000","last_undegraded":"2024-03-24T06:37:02.226196+0000","last_fullsized":"2024-03-24T06:37:02.226196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.409514+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:37:31.409514+0000","last_peered":"2024-03-24T06:37:31.409514+0000","last_clean":"2024-03-24T06:37:31.409514+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:37:31.409514+0000","last_undegraded":"2024-03-24T06:37:31.409514+0000","last_fullsized":"2024-03-24T06:37:31.409514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555962+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:37:02.555962+0000","last_peered":"2024-03-24T06:37:02.555962+0000","last_clean":"2024-03-24T06:37:02.555962+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:37:02.555962+0000","last_undegraded":"2024-03-24T06:37:02.555962+0000","last_fullsized":"2024-03-24T06:37:02.555962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555846+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:37:02.555846+0000","last_peered":"2024-03-24T06:37:02.555846+0000","last_clean":"2024-03-24T06:37:02.555846+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:37:02.555846+0000","last_undegraded":"2024-03-24T06:37:02.555846+0000","last_fullsized":"2024-03-24T06:37:02.555846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.401660+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:37:30.401660+0000","last_peered":"2024-03-24T06:37:30.401660+0000","last_clean":"2024-03-24T06:37:30.401660+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:37:30.401660+0000","last_undegraded":"2024-03-24T06:37:30.401660+0000","last_fullsized":"2024-03-24T06:37:30.401660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320345+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:37:02.320345+0000","last_peered":"2024-03-24T06:37:02.320345+0000","last_clean":"2024-03-24T06:37:02.320345+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:37:02.320345+0000","last_undegraded":"2024-03-24T06:37:02.320345+0000","last_fullsized":"2024-03-24T06:37:02.320345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814393+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:37:02.814393+0000","last_peered":"2024-03-24T06:37:02.814393+0000","last_clean":"2024-03-24T06:37:02.814393+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:37:02.814393+0000","last_undegraded":"2024-03-24T06:37:02.814393+0000","last_fullsized":"2024-03-24T06:37:02.814393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.405471+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:37:31.405471+0000","last_peered":"2024-03-24T06:37:31.405471+0000","last_clean":"2024-03-24T06:37:31.405471+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:37:31.405471+0000","last_undegraded":"2024-03-24T06:37:31.405471+0000","last_fullsized":"2024-03-24T06:37:31.405471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.229222+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:37:02.229222+0000","last_peered":"2024-03-24T06:37:02.229222+0000","last_clean":"2024-03-24T06:37:02.229222+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:37:02.229222+0000","last_undegraded":"2024-03-24T06:37:02.229222+0000","last_fullsized":"2024-03-24T06:37:02.229222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320360+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:37:02.320360+0000","last_peered":"2024-03-24T06:37:02.320360+0000","last_clean":"2024-03-24T06:37:02.320360+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:37:02.320360+0000","last_undegraded":"2024-03-24T06:37:02.320360+0000","last_fullsized":"2024-03-24T06:37:02.320360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.387177+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:37:29.387177+0000","last_peered":"2024-03-24T06:37:29.387177+0000","last_clean":"2024-03-24T06:37:29.387177+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:37:29.387177+0000","last_undegraded":"2024-03-24T06:37:29.387177+0000","last_fullsized":"2024-03-24T06:37:29.387177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555349+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:37:02.555349+0000","last_peered":"2024-03-24T06:37:02.555349+0000","last_clean":"2024-03-24T06:37:02.555349+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:37:02.555349+0000","last_undegraded":"2024-03-24T06:37:02.555349+0000","last_fullsized":"2024-03-24T06:37:02.555349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.229106+0000","last_change":"2024-03-24T06:37:01.261900+0000","last_active":"2024-03-24T06:37:02.229106+0000","last_peered":"2024-03-24T06:37:02.229106+0000","last_clean":"2024-03-24T06:37:02.229106+0000","last_became_active":"2024-03-24T06:37:01.259109+0000","last_became_peered":"2024-03-24T06:37:01.259109+0000","last_unstale":"2024-03-24T06:37:02.229106+0000","last_undegraded":"2024-03-24T06:37:02.229106+0000","last_fullsized":"2024-03-24T06:37:02.229106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:37:37.030003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.413722+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:37:32.413722+0000","last_peered":"2024-03-24T06:37:32.413722+0000","last_clean":"2024-03-24T06:37:32.413722+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:37:32.413722+0000","last_undegraded":"2024-03-24T06:37:32.413722+0000","last_fullsized":"2024-03-24T06:37:32.413722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814780+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:37:02.814780+0000","last_peered":"2024-03-24T06:37:02.814780+0000","last_clean":"2024-03-24T06:37:02.814780+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:37:02.814780+0000","last_undegraded":"2024-03-24T06:37:02.814780+0000","last_fullsized":"2024-03-24T06:37:02.814780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555125+0000","last_change":"2024-03-24T06:37:01.253622+0000","last_active":"2024-03-24T06:37:02.555125+0000","last_peered":"2024-03-24T06:37:02.555125+0000","last_clean":"2024-03-24T06:37:02.555125+0000","last_became_active":"2024-03-24T06:37:01.253160+0000","last_became_peered":"2024-03-24T06:37:01.253160+0000","last_unstale":"2024-03-24T06:37:02.555125+0000","last_undegraded":"2024-03-24T06:37:02.555125+0000","last_fullsized":"2024-03-24T06:37:02.555125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:02.997406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.386240+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:37:29.386240+0000","last_peered":"2024-03-24T06:37:29.386240+0000","last_clean":"2024-03-24T06:37:29.386240+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:37:29.386240+0000","last_undegraded":"2024-03-24T06:37:29.386240+0000","last_fullsized":"2024-03-24T06:37:29.386240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554971+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:37:02.554971+0000","last_peered":"2024-03-24T06:37:02.554971+0000","last_clean":"2024-03-24T06:37:02.554971+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:37:02.554971+0000","last_undegraded":"2024-03-24T06:37:02.554971+0000","last_fullsized":"2024-03-24T06:37:02.554971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.230125+0000","last_change":"2024-03-24T06:37:01.248655+0000","last_active":"2024-03-24T06:37:02.230125+0000","last_peered":"2024-03-24T06:37:02.230125+0000","last_clean":"2024-03-24T06:37:02.230125+0000","last_became_active":"2024-03-24T06:37:01.245755+0000","last_became_peered":"2024-03-24T06:37:01.245755+0000","last_unstale":"2024-03-24T06:37:02.230125+0000","last_undegraded":"2024-03-24T06:37:02.230125+0000","last_fullsized":"2024-03-24T06:37:02.230125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:29.632351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'22","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.415247+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:37:32.415247+0000","last_peered":"2024-03-24T06:37:32.415247+0000","last_clean":"2024-03-24T06:37:32.415247+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:37:32.415247+0000","last_undegraded":"2024-03-24T06:37:32.415247+0000","last_fullsized":"2024-03-24T06:37:32.415247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555073+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:37:02.555073+0000","last_peered":"2024-03-24T06:37:02.555073+0000","last_clean":"2024-03-24T06:37:02.555073+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:37:02.555073+0000","last_undegraded":"2024-03-24T06:37:02.555073+0000","last_fullsized":"2024-03-24T06:37:02.555073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556714+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:37:02.556714+0000","last_peered":"2024-03-24T06:37:02.556714+0000","last_clean":"2024-03-24T06:37:02.556714+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:37:02.556714+0000","last_undegraded":"2024-03-24T06:37:02.556714+0000","last_fullsized":"2024-03-24T06:37:02.556714+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.435468+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:37:33.435468+0000","last_peered":"2024-03-24T06:37:33.435468+0000","last_clean":"2024-03-24T06:37:33.435468+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:37:33.435468+0000","last_undegraded":"2024-03-24T06:37:33.435468+0000","last_fullsized":"2024-03-24T06:37:33.435468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557664+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:37:02.557664+0000","last_peered":"2024-03-24T06:37:02.557664+0000","last_clean":"2024-03-24T06:37:02.557664+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:37:02.557664+0000","last_undegraded":"2024-03-24T06:37:02.557664+0000","last_fullsized":"2024-03-24T06:37:02.557664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555285+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:37:02.555285+0000","last_peered":"2024-03-24T06:37:02.555285+0000","last_clean":"2024-03-24T06:37:02.555285+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:37:02.555285+0000","last_undegraded":"2024-03-24T06:37:02.555285+0000","last_fullsized":"2024-03-24T06:37:02.555285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.416184+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:37:31.416184+0000","last_peered":"2024-03-24T06:37:31.416184+0000","last_clean":"2024-03-24T06:37:31.416184+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:37:31.416184+0000","last_undegraded":"2024-03-24T06:37:31.416184+0000","last_fullsized":"2024-03-24T06:37:31.416184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225683+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:37:02.225683+0000","last_peered":"2024-03-24T06:37:02.225683+0000","last_clean":"2024-03-24T06:37:02.225683+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:37:02.225683+0000","last_undegraded":"2024-03-24T06:37:02.225683+0000","last_fullsized":"2024-03-24T06:37:02.225683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.231831+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:37:02.231831+0000","last_peered":"2024-03-24T06:37:02.231831+0000","last_clean":"2024-03-24T06:37:02.231831+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:37:02.231831+0000","last_undegraded":"2024-03-24T06:37:02.231831+0000","last_fullsized":"2024-03-24T06:37:02.231831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.404498+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:37:31.404498+0000","last_peered":"2024-03-24T06:37:31.404498+0000","last_clean":"2024-03-24T06:37:31.404498+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:37:31.404498+0000","last_undegraded":"2024-03-24T06:37:31.404498+0000","last_fullsized":"2024-03-24T06:37:31.404498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225434+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:37:02.225434+0000","last_peered":"2024-03-24T06:37:02.225434+0000","last_clean":"2024-03-24T06:37:02.225434+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:37:02.225434+0000","last_undegraded":"2024-03-24T06:37:02.225434+0000","last_fullsized":"2024-03-24T06:37:02.225434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557526+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:37:02.557526+0000","last_peered":"2024-03-24T06:37:02.557526+0000","last_clean":"2024-03-24T06:37:02.557526+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:37:02.557526+0000","last_undegraded":"2024-03-24T06:37:02.557526+0000","last_fullsized":"2024-03-24T06:37:02.557526+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.383520+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:37:29.383520+0000","last_peered":"2024-03-24T06:37:29.383520+0000","last_clean":"2024-03-24T06:37:29.383520+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:37:29.383520+0000","last_undegraded":"2024-03-24T06:37:29.383520+0000","last_fullsized":"2024-03-24T06:37:29.383520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555514+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:37:02.555514+0000","last_peered":"2024-03-24T06:37:02.555514+0000","last_clean":"2024-03-24T06:37:02.555514+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:37:02.555514+0000","last_undegraded":"2024-03-24T06:37:02.555514+0000","last_fullsized":"2024-03-24T06:37:02.555514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555452+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:37:02.555452+0000","last_peered":"2024-03-24T06:37:02.555452+0000","last_clean":"2024-03-24T06:37:02.555452+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:37:02.555452+0000","last_undegraded":"2024-03-24T06:37:02.555452+0000","last_fullsized":"2024-03-24T06:37:02.555452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555185+0000","last_change":"2024-03-24T06:36:55.722575+0000","last_active":"2024-03-24T06:37:02.555185+0000","last_peered":"2024-03-24T06:37:02.555185+0000","last_clean":"2024-03-24T06:37:02.555185+0000","last_became_active":"2024-03-24T06:36:55.722249+0000","last_became_peered":"2024-03-24T06:36:55.722249+0000","last_unstale":"2024-03-24T06:37:02.555185+0000","last_undegraded":"2024-03-24T06:37:02.555185+0000","last_fullsized":"2024-03-24T06:37:02.555185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.905215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555185+0000","last_change":"2024-03-24T06:36:56.750708+0000","last_active":"2024-03-24T06:37:02.555185+0000","last_peered":"2024-03-24T06:37:02.555185+0000","last_clean":"2024-03-24T06:37:02.555185+0000","last_became_active":"2024-03-24T06:36:56.750391+0000","last_became_peered":"2024-03-24T06:36:56.750391+0000","last_unstale":"2024-03-24T06:37:02.555185+0000","last_undegraded":"2024-03-24T06:37:02.555185+0000","last_fullsized":"2024-03-24T06:37:02.555185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:40:31.577213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554800+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:37:02.554800+0000","last_peered":"2024-03-24T06:37:02.554800+0000","last_clean":"2024-03-24T06:37:02.554800+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:37:02.554800+0000","last_undegraded":"2024-03-24T06:37:02.554800+0000","last_fullsized":"2024-03-24T06:37:02.554800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.230389+0000","last_change":"2024-03-24T06:36:55.738736+0000","last_active":"2024-03-24T06:37:02.230389+0000","last_peered":"2024-03-24T06:37:02.230389+0000","last_clean":"2024-03-24T06:37:02.230389+0000","last_became_active":"2024-03-24T06:36:55.738313+0000","last_became_peered":"2024-03-24T06:36:55.738313+0000","last_unstale":"2024-03-24T06:37:02.230389+0000","last_undegraded":"2024-03-24T06:37:02.230389+0000","last_fullsized":"2024-03-24T06:37:02.230389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:39.785983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.313098+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:37:02.313098+0000","last_peered":"2024-03-24T06:37:02.313098+0000","last_clean":"2024-03-24T06:37:02.313098+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:37:02.313098+0000","last_undegraded":"2024-03-24T06:37:02.313098+0000","last_fullsized":"2024-03-24T06:37:02.313098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557108+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:37:02.557108+0000","last_peered":"2024-03-24T06:37:02.557108+0000","last_clean":"2024-03-24T06:37:02.557108+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:37:02.557108+0000","last_undegraded":"2024-03-24T06:37:02.557108+0000","last_fullsized":"2024-03-24T06:37:02.557108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.438423+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:37:33.438423+0000","last_peered":"2024-03-24T06:37:33.438423+0000","last_clean":"2024-03-24T06:37:33.438423+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:37:33.438423+0000","last_undegraded":"2024-03-24T06:37:33.438423+0000","last_fullsized":"2024-03-24T06:37:33.438423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.312929+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:37:02.312929+0000","last_peered":"2024-03-24T06:37:02.312929+0000","last_clean":"2024-03-24T06:37:02.312929+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:37:02.312929+0000","last_undegraded":"2024-03-24T06:37:02.312929+0000","last_fullsized":"2024-03-24T06:37:02.312929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557450+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:37:02.557450+0000","last_peered":"2024-03-24T06:37:02.557450+0000","last_clean":"2024-03-24T06:37:02.557450+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:37:02.557450+0000","last_undegraded":"2024-03-24T06:37:02.557450+0000","last_fullsized":"2024-03-24T06:37:02.557450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556229+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:37:02.556229+0000","last_peered":"2024-03-24T06:37:02.556229+0000","last_clean":"2024-03-24T06:37:02.556229+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:37:02.556229+0000","last_undegraded":"2024-03-24T06:37:02.556229+0000","last_fullsized":"2024-03-24T06:37:02.556229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.435790+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:37:33.435790+0000","last_peered":"2024-03-24T06:37:33.435790+0000","last_clean":"2024-03-24T06:37:33.435790+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:37:33.435790+0000","last_undegraded":"2024-03-24T06:37:33.435790+0000","last_fullsized":"2024-03-24T06:37:33.435790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:03.963302+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:37:03.963302+0000","last_peered":"2024-03-24T06:37:03.963302+0000","last_clean":"2024-03-24T06:37:03.963302+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:37:03.963302+0000","last_undegraded":"2024-03-24T06:37:03.963302+0000","last_fullsized":"2024-03-24T06:37:03.963302+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320240+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:37:02.320240+0000","last_peered":"2024-03-24T06:37:02.320240+0000","last_clean":"2024-03-24T06:37:02.320240+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:37:02.320240+0000","last_undegraded":"2024-03-24T06:37:02.320240+0000","last_fullsized":"2024-03-24T06:37:02.320240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.411713+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:37:31.411713+0000","last_peered":"2024-03-24T06:37:31.411713+0000","last_clean":"2024-03-24T06:37:31.411713+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:37:31.411713+0000","last_undegraded":"2024-03-24T06:37:31.411713+0000","last_fullsized":"2024-03-24T06:37:31.411713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225309+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:37:02.225309+0000","last_peered":"2024-03-24T06:37:02.225309+0000","last_clean":"2024-03-24T06:37:02.225309+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:37:02.225309+0000","last_undegraded":"2024-03-24T06:37:02.225309+0000","last_fullsized":"2024-03-24T06:37:02.225309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556434+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:37:02.556434+0000","last_peered":"2024-03-24T06:37:02.556434+0000","last_clean":"2024-03-24T06:37:02.556434+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:37:02.556434+0000","last_undegraded":"2024-03-24T06:37:02.556434+0000","last_fullsized":"2024-03-24T06:37:02.556434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556260+0000","last_change":"2024-03-24T06:36:56.764415+0000","last_active":"2024-03-24T06:37:02.556260+0000","last_peered":"2024-03-24T06:37:02.556260+0000","last_clean":"2024-03-24T06:37:02.556260+0000","last_became_active":"2024-03-24T06:36:56.762075+0000","last_became_peered":"2024-03-24T06:36:56.762075+0000","last_unstale":"2024-03-24T06:37:02.556260+0000","last_undegraded":"2024-03-24T06:37:02.556260+0000","last_fullsized":"2024-03-24T06:37:02.556260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:31:23.642005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.392497+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:37:30.392497+0000","last_peered":"2024-03-24T06:37:30.392497+0000","last_clean":"2024-03-24T06:37:30.392497+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:37:30.392497+0000","last_undegraded":"2024-03-24T06:37:30.392497+0000","last_fullsized":"2024-03-24T06:37:30.392497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555338+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:37:02.555338+0000","last_peered":"2024-03-24T06:37:02.555338+0000","last_clean":"2024-03-24T06:37:02.555338+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:37:02.555338+0000","last_undegraded":"2024-03-24T06:37:02.555338+0000","last_fullsized":"2024-03-24T06:37:02.555338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555242+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:37:02.555242+0000","last_peered":"2024-03-24T06:37:02.555242+0000","last_clean":"2024-03-24T06:37:02.555242+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:37:02.555242+0000","last_undegraded":"2024-03-24T06:37:02.555242+0000","last_fullsized":"2024-03-24T06:37:02.555242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557007+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:37:02.557007+0000","last_peered":"2024-03-24T06:37:02.557007+0000","last_clean":"2024-03-24T06:37:02.557007+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:37:02.557007+0000","last_undegraded":"2024-03-24T06:37:02.557007+0000","last_fullsized":"2024-03-24T06:37:02.557007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320516+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:37:02.320516+0000","last_peered":"2024-03-24T06:37:02.320516+0000","last_clean":"2024-03-24T06:37:02.320516+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:37:02.320516+0000","last_undegraded":"2024-03-24T06:37:02.320516+0000","last_fullsized":"2024-03-24T06:37:02.320516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:30.398616+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:37:30.398616+0000","last_peered":"2024-03-24T06:37:30.398616+0000","last_clean":"2024-03-24T06:37:30.398616+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:37:30.398616+0000","last_undegraded":"2024-03-24T06:37:30.398616+0000","last_fullsized":"2024-03-24T06:37:30.398616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320662+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:37:02.320662+0000","last_peered":"2024-03-24T06:37:02.320662+0000","last_clean":"2024-03-24T06:37:02.320662+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:37:02.320662+0000","last_undegraded":"2024-03-24T06:37:02.320662+0000","last_fullsized":"2024-03-24T06:37:02.320662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226196+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:37:02.226196+0000","last_peered":"2024-03-24T06:37:02.226196+0000","last_clean":"2024-03-24T06:37:02.226196+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:37:02.226196+0000","last_undegraded":"2024-03-24T06:37:02.226196+0000","last_fullsized":"2024-03-24T06:37:02.226196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.419322+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:37:32.419322+0000","last_peered":"2024-03-24T06:37:32.419322+0000","last_clean":"2024-03-24T06:37:32.419322+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:37:32.419322+0000","last_undegraded":"2024-03-24T06:37:32.419322+0000","last_fullsized":"2024-03-24T06:37:32.419322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814721+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:37:02.814721+0000","last_peered":"2024-03-24T06:37:02.814721+0000","last_clean":"2024-03-24T06:37:02.814721+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:37:02.814721+0000","last_undegraded":"2024-03-24T06:37:02.814721+0000","last_fullsized":"2024-03-24T06:37:02.814721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555236+0000","last_change":"2024-03-24T06:37:01.256625+0000","last_active":"2024-03-24T06:37:02.555236+0000","last_peered":"2024-03-24T06:37:02.555236+0000","last_clean":"2024-03-24T06:37:02.555236+0000","last_became_active":"2024-03-24T06:37:01.256152+0000","last_became_peered":"2024-03-24T06:37:01.256152+0000","last_unstale":"2024-03-24T06:37:02.555236+0000","last_undegraded":"2024-03-24T06:37:02.555236+0000","last_fullsized":"2024-03-24T06:37:02.555236+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:58:34.253874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.383711+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:37:29.383711+0000","last_peered":"2024-03-24T06:37:29.383711+0000","last_clean":"2024-03-24T06:37:29.383711+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:37:29.383711+0000","last_undegraded":"2024-03-24T06:37:29.383711+0000","last_fullsized":"2024-03-24T06:37:29.383711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554418+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:37:02.554418+0000","last_peered":"2024-03-24T06:37:02.554418+0000","last_clean":"2024-03-24T06:37:02.554418+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:37:02.554418+0000","last_undegraded":"2024-03-24T06:37:02.554418+0000","last_fullsized":"2024-03-24T06:37:02.554418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.232499+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:37:02.232499+0000","last_peered":"2024-03-24T06:37:02.232499+0000","last_clean":"2024-03-24T06:37:02.232499+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:37:02.232499+0000","last_undegraded":"2024-03-24T06:37:02.232499+0000","last_fullsized":"2024-03-24T06:37:02.232499+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.387757+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:37:29.387757+0000","last_peered":"2024-03-24T06:37:29.387757+0000","last_clean":"2024-03-24T06:37:29.387757+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:37:29.387757+0000","last_undegraded":"2024-03-24T06:37:29.387757+0000","last_fullsized":"2024-03-24T06:37:29.387757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555473+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:37:02.555473+0000","last_peered":"2024-03-24T06:37:02.555473+0000","last_clean":"2024-03-24T06:37:02.555473+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:37:02.555473+0000","last_undegraded":"2024-03-24T06:37:02.555473+0000","last_fullsized":"2024-03-24T06:37:02.555473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554881+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:37:02.554881+0000","last_peered":"2024-03-24T06:37:02.554881+0000","last_clean":"2024-03-24T06:37:02.554881+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:37:02.554881+0000","last_undegraded":"2024-03-24T06:37:02.554881+0000","last_fullsized":"2024-03-24T06:37:02.554881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.412826+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:37:31.412826+0000","last_peered":"2024-03-24T06:37:31.412826+0000","last_clean":"2024-03-24T06:37:31.412826+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:37:31.412826+0000","last_undegraded":"2024-03-24T06:37:31.412826+0000","last_fullsized":"2024-03-24T06:37:31.412826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320962+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:37:02.320962+0000","last_peered":"2024-03-24T06:37:02.320962+0000","last_clean":"2024-03-24T06:37:02.320962+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:37:02.320962+0000","last_undegraded":"2024-03-24T06:37:02.320962+0000","last_fullsized":"2024-03-24T06:37:02.320962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554972+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:37:02.554972+0000","last_peered":"2024-03-24T06:37:02.554972+0000","last_clean":"2024-03-24T06:37:02.554972+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:37:02.554972+0000","last_undegraded":"2024-03-24T06:37:02.554972+0000","last_fullsized":"2024-03-24T06:37:02.554972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:29.381605+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:37:29.381605+0000","last_peered":"2024-03-24T06:37:29.381605+0000","last_clean":"2024-03-24T06:37:29.381605+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:37:29.381605+0000","last_undegraded":"2024-03-24T06:37:29.381605+0000","last_fullsized":"2024-03-24T06:37:29.381605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225603+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:37:02.225603+0000","last_peered":"2024-03-24T06:37:02.225603+0000","last_clean":"2024-03-24T06:37:02.225603+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:37:02.225603+0000","last_undegraded":"2024-03-24T06:37:02.225603+0000","last_fullsized":"2024-03-24T06:37:02.225603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554398+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:37:02.554398+0000","last_peered":"2024-03-24T06:37:02.554398+0000","last_clean":"2024-03-24T06:37:02.554398+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:37:02.554398+0000","last_undegraded":"2024-03-24T06:37:02.554398+0000","last_fullsized":"2024-03-24T06:37:02.554398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:33.440381+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:37:33.440381+0000","last_peered":"2024-03-24T06:37:33.440381+0000","last_clean":"2024-03-24T06:37:33.440381+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:37:33.440381+0000","last_undegraded":"2024-03-24T06:37:33.440381+0000","last_fullsized":"2024-03-24T06:37:33.440381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225921+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:37:02.225921+0000","last_peered":"2024-03-24T06:37:02.225921+0000","last_clean":"2024-03-24T06:37:02.225921+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:37:02.225921+0000","last_undegraded":"2024-03-24T06:37:02.225921+0000","last_fullsized":"2024-03-24T06:37:02.225921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814330+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:37:02.814330+0000","last_peered":"2024-03-24T06:37:02.814330+0000","last_clean":"2024-03-24T06:37:02.814330+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:37:02.814330+0000","last_undegraded":"2024-03-24T06:37:02.814330+0000","last_fullsized":"2024-03-24T06:37:02.814330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:31.412841+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:37:31.412841+0000","last_peered":"2024-03-24T06:37:31.412841+0000","last_clean":"2024-03-24T06:37:31.412841+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:37:31.412841+0000","last_undegraded":"2024-03-24T06:37:31.412841+0000","last_fullsized":"2024-03-24T06:37:31.412841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225478+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:37:02.225478+0000","last_peered":"2024-03-24T06:37:02.225478+0000","last_clean":"2024-03-24T06:37:02.225478+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:37:02.225478+0000","last_undegraded":"2024-03-24T06:37:02.225478+0000","last_fullsized":"2024-03-24T06:37:02.225478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556114+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:37:02.556114+0000","last_peered":"2024-03-24T06:37:02.556114+0000","last_clean":"2024-03-24T06:37:02.556114+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:37:02.556114+0000","last_undegraded":"2024-03-24T06:37:02.556114+0000","last_fullsized":"2024-03-24T06:37:02.556114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:32.420275+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:37:32.420275+0000","last_peered":"2024-03-24T06:37:32.420275+0000","last_clean":"2024-03-24T06:37:32.420275+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:37:32.420275+0000","last_undegraded":"2024-03-24T06:37:32.420275+0000","last_fullsized":"2024-03-24T06:37:32.420275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225904+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:37:02.225904+0000","last_peered":"2024-03-24T06:37:02.225904+0000","last_clean":"2024-03-24T06:37:02.225904+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:37:02.225904+0000","last_undegraded":"2024-03-24T06:37:02.225904+0000","last_fullsized":"2024-03-24T06:37:02.225904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555567+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:37:02.555567+0000","last_peered":"2024-03-24T06:37:02.555567+0000","last_clean":"2024-03-24T06:37:02.555567+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:37:02.555567+0000","last_undegraded":"2024-03-24T06:37:02.555567+0000","last_fullsized":"2024-03-24T06:37:02.555567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226394+0000","last_change":"2024-03-24T06:36:56.750417+0000","last_active":"2024-03-24T06:37:02.226394+0000","last_peered":"2024-03-24T06:37:02.226394+0000","last_clean":"2024-03-24T06:37:02.226394+0000","last_became_active":"2024-03-24T06:36:56.750024+0000","last_became_peered":"2024-03-24T06:36:56.750024+0000","last_unstale":"2024-03-24T06:37:02.226394+0000","last_undegraded":"2024-03-24T06:37:02.226394+0000","last_fullsized":"2024-03-24T06:37:02.226394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:16:14.426791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.225663+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:37:02.225663+0000","last_peered":"2024-03-24T06:37:02.225663+0000","last_clean":"2024-03-24T06:37:02.225663+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:37:02.225663+0000","last_undegraded":"2024-03-24T06:37:02.225663+0000","last_fullsized":"2024-03-24T06:37:02.225663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.814300+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:37:02.814300+0000","last_peered":"2024-03-24T06:37:02.814300+0000","last_clean":"2024-03-24T06:37:02.814300+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:37:02.814300+0000","last_undegraded":"2024-03-24T06:37:02.814300+0000","last_fullsized":"2024-03-24T06:37:02.814300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.557145+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:37:02.557145+0000","last_peered":"2024-03-24T06:37:02.557145+0000","last_clean":"2024-03-24T06:37:02.557145+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:37:02.557145+0000","last_undegraded":"2024-03-24T06:37:02.557145+0000","last_fullsized":"2024-03-24T06:37:02.557145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555212+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:37:02.555212+0000","last_peered":"2024-03-24T06:37:02.555212+0000","last_clean":"2024-03-24T06:37:02.555212+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:37:02.555212+0000","last_undegraded":"2024-03-24T06:37:02.555212+0000","last_fullsized":"2024-03-24T06:37:02.555212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.226265+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:37:02.226265+0000","last_peered":"2024-03-24T06:37:02.226265+0000","last_clean":"2024-03-24T06:37:02.226265+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:37:02.226265+0000","last_undegraded":"2024-03-24T06:37:02.226265+0000","last_fullsized":"2024-03-24T06:37:02.226265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320907+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:37:02.320907+0000","last_peered":"2024-03-24T06:37:02.320907+0000","last_clean":"2024-03-24T06:37:02.320907+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:37:02.320907+0000","last_undegraded":"2024-03-24T06:37:02.320907+0000","last_fullsized":"2024-03-24T06:37:02.320907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556224+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:37:02.556224+0000","last_peered":"2024-03-24T06:37:02.556224+0000","last_clean":"2024-03-24T06:37:02.556224+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:37:02.556224+0000","last_undegraded":"2024-03-24T06:37:02.556224+0000","last_fullsized":"2024-03-24T06:37:02.556224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.320414+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:37:02.320414+0000","last_peered":"2024-03-24T06:37:02.320414+0000","last_clean":"2024-03-24T06:37:02.320414+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:37:02.320414+0000","last_undegraded":"2024-03-24T06:37:02.320414+0000","last_fullsized":"2024-03-24T06:37:02.320414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.554524+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:37:02.554524+0000","last_peered":"2024-03-24T06:37:02.554524+0000","last_clean":"2024-03-24T06:37:02.554524+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:37:02.554524+0000","last_undegraded":"2024-03-24T06:37:02.554524+0000","last_fullsized":"2024-03-24T06:37:02.554524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.556825+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:37:02.556825+0000","last_peered":"2024-03-24T06:37:02.556825+0000","last_clean":"2024-03-24T06:37:02.556825+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:37:02.556825+0000","last_undegraded":"2024-03-24T06:37:02.556825+0000","last_fullsized":"2024-03-24T06:37:02.556825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T06:37:02.555349+0000","last_change":"2024-03-24T06:36:56.755968+0000","last_active":"2024-03-24T06:37:02.555349+0000","last_peered":"2024-03-24T06:37:02.555349+0000","last_clean":"2024-03-24T06:37:02.555349+0000","last_became_active":"2024-03-24T06:36:56.755651+0000","last_became_peered":"2024-03-24T06:36:56.755651+0000","last_unstale":"2024-03-24T06:37:02.555349+0000","last_undegraded":"2024-03-24T06:37:02.555349+0000","last_fullsized":"2024-03-24T06:37:02.555349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:00.246108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":28689,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":174,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":82944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1079727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1074821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":478603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":519658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1074499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:37:35.885 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T06:37:35.885 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T06:37:35.885 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T06:37:36.397 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:36 smithi143 ceph-mon[36520]: pgmap v339: 129 pgs: 129 active+clean; 605 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-24T06:37:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:36 smithi143 ceph-mon[36520]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:36 smithi143 ceph-mon[36520]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:36.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:36 smithi022 ceph-mon[32003]: pgmap v339: 129 pgs: 129 active+clean; 605 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-24T06:37:36.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:36 smithi022 ceph-mon[32003]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:36.685 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:36 smithi022 ceph-mon[32003]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:36 smithi155 ceph-mon[36262]: pgmap v339: 129 pgs: 129 active+clean; 605 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-24T06:37:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:36 smithi155 ceph-mon[36262]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:36 smithi155 ceph-mon[36262]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:37:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:38 smithi022 ceph-mon[32003]: pgmap v340: 129 pgs: 129 active+clean; 606 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-24T06:37:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:38 smithi155 ceph-mon[36262]: pgmap v340: 129 pgs: 129 active+clean; 606 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-24T06:37:38.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 -- 172.21.15.22:0/4203568575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103c30 msgr2=0x7efcec10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:38.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 --2- 172.21.15.22:0/4203568575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103c30 0x7efcec10a4e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efce400b0d0 tx=0x7efce400b3e0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 -- 172.21.15.22:0/4203568575 shutdown_connections 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 --2- 172.21.15.22:0/4203568575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec102090 0x7efcec1024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 --2- 172.21.15.22:0/4203568575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103c30 0x7efcec10a4e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 --2- 172.21.15.22:0/4203568575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec103290 0x7efcec1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 -- 172.21.15.22:0/4203568575 >> 172.21.15.22:0/4203568575 conn(0x7efcec0fd670 msgr2=0x7efcec0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 -- 172.21.15.22:0/4203568575 shutdown_connections 2024-03-24T06:37:38.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.837+0000 7efcf1600700 1 -- 172.21.15.22:0/4203568575 wait complete. 2024-03-24T06:37:38.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.839+0000 7efcf1600700 1 Processor -- start 2024-03-24T06:37:38.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 -- start start 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec102090 0x7efcec075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec103c30 0x7efcec074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efcec0752b0 con 0x7efcec103c30 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efcec075420 con 0x7efcec103290 2024-03-24T06:37:38.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.840+0000 7efcf1600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efcec1a20d0 con 0x7efcec102090 2024-03-24T06:37:38.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:38.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46786/0 (socket says 172.21.15.22:46786) 2024-03-24T06:37:38.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 -- 172.21.15.22:0/2560991163 learned_addr learned my addr 172.21.15.22:0/2560991163 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:38.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efcf0c00700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec103c30 0x7efcec074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:38.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efcebe00700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec102090 0x7efcec075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 -- 172.21.15.22:0/2560991163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec102090 msgr2=0x7efcec075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec102090 0x7efcec075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 -- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec103c30 msgr2=0x7efcec074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec103c30 0x7efcec074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.841+0000 7efceb400700 1 -- 172.21.15.22:0/2560991163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce4009d00 con 0x7efcec103290 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efceb400700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efcd800e9a0 tx=0x7efcd800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8009790 con 0x7efcec103290 2024-03-24T06:37:38.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcd8010430 con 0x7efcec103290 2024-03-24T06:37:38.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8018680 con 0x7efcec103290 2024-03-24T06:37:38.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcec1a23b0 con 0x7efcec103290 2024-03-24T06:37:38.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.842+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcec1a2900 con 0x7efcec103290 2024-03-24T06:37:38.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.843+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcec109380 con 0x7efcec103290 2024-03-24T06:37:38.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.845+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7efcd8010910 con 0x7efcec103290 2024-03-24T06:37:38.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.846+0000 7efce8c00700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 0x7efcd00a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:38.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.847+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efcd8014070 con 0x7efcec103290 2024-03-24T06:37:38.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.847+0000 7efcebe00700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 0x7efcd00a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:38.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.847+0000 7efcebe00700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 0x7efcd00a47d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efcec1030f0 tx=0x7efcdc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:38.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:38.850+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efcd8053280 con 0x7efcec103290 2024-03-24T06:37:38.927 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:38 smithi143 ceph-mon[36520]: pgmap v340: 129 pgs: 129 active+clean; 606 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-03-24T06:37:39.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.223+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7efcec10dc50 con 0x7efcec103290 2024-03-24T06:37:39.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.229+0000 7efce8c00700 1 -- 172.21.15.22:0/2560991163 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7efcd8096e30 con 0x7efcec103290 2024-03-24T06:37:39.229 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-24T06:37:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAzyv9lYL94DRAA/II4WAvNQ8gZyphc9NndyQ== 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 msgr2=0x7efcd00a47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 0x7efcd00a47d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efcec1030f0 tx=0x7efcdc009260 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 msgr2=0x7efcec0745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efcd800e9a0 tx=0x7efcd800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 shutdown_connections 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efcec103c30 0x7efcec074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcd00a2310 0x7efcd00a47d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efcec102090 0x7efcec075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 --2- 172.21.15.22:0/2560991163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcec103290 0x7efcec0745f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:39.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 >> 172.21.15.22:0/2560991163 conn(0x7efcec0fd670 msgr2=0x7efcec1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:39.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 shutdown_connections 2024-03-24T06:37:39.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:39.232+0000 7efcf1600700 1 -- 172.21.15.22:0/2560991163 wait complete. 2024-03-24T06:37:39.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:39 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2560991163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:39 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:39 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T06:37:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:39 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2560991163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:39 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:39 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T06:37:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:39 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2560991163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:39 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T06:37:39.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:39 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T06:37:40.148 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:37:40.148 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T06:37:40.148 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T06:37:40.306 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T06:37:40.306 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T06:37:40.307 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T06:37:40.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:40 smithi155 ceph-mon[36262]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-24T06:37:40.815 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:40.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:40 smithi022 ceph-mon[32003]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-24T06:37:40.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:40 smithi143 ceph-mon[36520]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-24T06:37:42.754 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:42 smithi022 ceph-mon[32003]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-24T06:37:42.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:42 smithi155 ceph-mon[36262]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-24T06:37:42.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:42 smithi143 ceph-mon[36520]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-24T06:37:43.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 -- 172.21.15.22:0/809667438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 msgr2=0x7f491410f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 --2- 172.21.15.22:0/809667438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491410f150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f490c00b0a0 tx=0x7f490c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 -- 172.21.15.22:0/809667438 shutdown_connections 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 --2- 172.21.15.22:0/809667438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491410f150 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 --2- 172.21.15.22:0/809667438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f4914102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 --2- 172.21.15.22:0/809667438 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 0x7f4914106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 -- 172.21.15.22:0/809667438 >> 172.21.15.22:0/809667438 conn(0x7f4914079360 msgr2=0x7f4914079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 -- 172.21.15.22:0/809667438 shutdown_connections 2024-03-24T06:37:43.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.085+0000 7f491bc00700 1 -- 172.21.15.22:0/809667438 wait complete. 2024-03-24T06:37:43.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.087+0000 7f491bc00700 1 Processor -- start 2024-03-24T06:37:43.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.087+0000 7f491bc00700 1 -- start start 2024-03-24T06:37:43.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.087+0000 7f491bc00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:43.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f491bc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 0x7f49141a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:43.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f491bc00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:43.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f491bc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f491419af50 con 0x7f4914102e60 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f491bc00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f491419b090 con 0x7f49141024b0 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46816/0 (socket says 172.21.15.22:46816) 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 -- 172.21.15.22:0/3435562699 learned_addr learned my addr 172.21.15.22:0/3435562699 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f491419b250 con 0x7f4914107580 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4913e00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 0x7f49141a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919c00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 -- 172.21.15.22:0/3435562699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 msgr2=0x7f491419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 -- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 msgr2=0x7f49141a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 0x7f49141a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919200700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4908009520 con 0x7f49141024b0 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.088+0000 7f4919c00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:43.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f4919200700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f490800eb30 tx=0x7f490800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:43.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908009880 con 0x7f49141024b0 2024-03-24T06:37:43.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4908004500 con 0x7f49141024b0 2024-03-24T06:37:43.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c009d00 con 0x7f49141024b0 2024-03-24T06:37:43.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908003e70 con 0x7f49141024b0 2024-03-24T06:37:43.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.089+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49141a7600 con 0x7f49141024b0 2024-03-24T06:37:43.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.090+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4914068ec0 con 0x7f49141024b0 2024-03-24T06:37:43.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.092+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f490800cbd0 con 0x7f49141024b0 2024-03-24T06:37:43.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.094+0000 7f4911600700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 0x7f48fc0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:43.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.094+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4908014070 con 0x7f49141024b0 2024-03-24T06:37:43.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.094+0000 7f4913e00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 0x7f48fc0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:43.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.095+0000 7f4913e00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 0x7f48fc0a4720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4914076490 tx=0x7f4900009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:43.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.098+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4908097010 con 0x7f49141024b0 2024-03-24T06:37:43.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.413+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f49141122f0 con 0x7f49141024b0 2024-03-24T06:37:43.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.418+0000 7f4911600700 1 -- 172.21.15.22:0/3435562699 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+261465 (secure 0 0 0) 0x7f490809a6d0 con 0x7f49141024b0 2024-03-24T06:37:43.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:43.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.423+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 msgr2=0x7f48fc0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:43.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.423+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 0x7f48fc0a4720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4914076490 tx=0x7f4900009260 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.423+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 msgr2=0x7f49141a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.423+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f490800eb30 tx=0x7f490800ee40 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 shutdown_connections 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f48fc0a2260 0x7f48fc0a4720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4914107580 0x7f491419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49141024b0 0x7f49141a0730 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 --2- 172.21.15.22:0/3435562699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4914102e60 0x7f49141a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 >> 172.21.15.22:0/3435562699 conn(0x7f4914079360 msgr2=0x7f491410d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 shutdown_connections 2024-03-24T06:37:43.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:43.424+0000 7f491bc00700 1 -- 172.21.15.22:0/3435562699 wait complete. 2024-03-24T06:37:43.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:43 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3435562699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T06:37:43.792 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3435562699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T06:37:43.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:43 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3435562699' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T06:37:44.207 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":503368228},{"type":"v1","addr":"172.21.15.22:6801","nonce":503368228}]},"active_addr":"172.21.15.22:6801/503368228","active_change":"2024-03-24T06:28:54.941758+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.22:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":4236060832}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3288864197}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":440133497}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":673082346}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":1447507319}]}]} 2024-03-24T06:37:44.212 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T06:37:44.212 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T06:37:44.212 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T06:37:44.673 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:44 smithi155 ceph-mon[36262]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-03-24T06:37:44.828 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:44 smithi022 ceph-mon[32003]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-03-24T06:37:44.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:44 smithi143 ceph-mon[36520]: pgmap v343: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-03-24T06:37:46.607 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:46 smithi022 ceph-mon[32003]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 20 op/s 2024-03-24T06:37:46.729 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:46 smithi143 ceph-mon[36520]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 20 op/s 2024-03-24T06:37:46.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:46 smithi155 ceph-mon[36262]: pgmap v344: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 20 op/s 2024-03-24T06:37:46.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 -- 172.21.15.22:0/592448251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 msgr2=0x7f6bf8074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 --2- 172.21.15.22:0/592448251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf8074810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b0a0 tx=0x7f6be800b3b0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 -- 172.21.15.22:0/592448251 shutdown_connections 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 --2- 172.21.15.22:0/592448251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 0x7f6bf8110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 --2- 172.21.15.22:0/592448251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf8074810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 --2- 172.21.15.22:0/592448251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 -- 172.21.15.22:0/592448251 >> 172.21.15.22:0/592448251 conn(0x7f6bf8079360 msgr2=0x7f6bf8079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 -- 172.21.15.22:0/592448251 shutdown_connections 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.930+0000 7f6bfe000700 1 -- 172.21.15.22:0/592448251 wait complete. 2024-03-24T06:37:46.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.931+0000 7f6bfe000700 1 Processor -- start 2024-03-24T06:37:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 -- start start 2024-03-24T06:37:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf81a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 0x7f6bf819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf819aff0 con 0x7f6bf8074d50 2024-03-24T06:37:46.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf819b130 con 0x7f6bf810ec00 2024-03-24T06:37:46.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.932+0000 7f6bfe000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf819b2f0 con 0x7f6bf8074430 2024-03-24T06:37:46.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:46.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42252/0 (socket says 172.21.15.22:42252) 2024-03-24T06:37:46.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 -- 172.21.15.22:0/711043081 learned_addr learned my addr 172.21.15.22:0/711043081 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7e00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 0x7f6bf819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6befe00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf81a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 -- 172.21.15.22:0/711043081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 msgr2=0x7f6bf819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 0x7f6bf819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 -- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 msgr2=0x7f6bf81a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf81a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 -- 172.21.15.22:0/711043081 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be0009520 con 0x7f6bf8074430 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6befe00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf81a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.933+0000 7f6bf7400700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b070 tx=0x7f6be800be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:46.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.934+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800e040 con 0x7f6bf8074430 2024-03-24T06:37:46.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.934+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6be801d470 con 0x7f6bf8074430 2024-03-24T06:37:46.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.934+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be8009d00 con 0x7f6bf8074430 2024-03-24T06:37:46.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.934+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be801b6a0 con 0x7f6bf8074430 2024-03-24T06:37:46.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.934+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf81a7650 con 0x7f6bf8074430 2024-03-24T06:37:46.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.935+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf8068ec0 con 0x7f6bf8074430 2024-03-24T06:37:46.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.937+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6be801d5e0 con 0x7f6bf8074430 2024-03-24T06:37:46.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.938+0000 7f6bf4c00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 0x7f6bdc0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:46.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.939+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6be80ceec0 con 0x7f6bf8074430 2024-03-24T06:37:46.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.939+0000 7f6befe00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 0x7f6bdc0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:46.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.940+0000 7f6befe00700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 0x7f6bdc0a4720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6bf81108a0 tx=0x7f6be0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:46.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:46.942+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6be8097b20 con 0x7f6bf8074430 2024-03-24T06:37:47.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.168+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bf8114570 con 0x7f6bf8074430 2024-03-24T06:37:47.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.169+0000 7f6bf4c00700 1 -- 172.21.15.22:0/711043081 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7f6be809b1e0 con 0x7f6bf8074430 2024-03-24T06:37:47.170 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:47.170 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:47.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 msgr2=0x7f6bdc0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:47.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 0x7f6bdc0a4720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6bf81108a0 tx=0x7f6be0009260 comp rx=0 tx=0).stop 2024-03-24T06:37:47.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 msgr2=0x7f6bf81a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:47.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b070 tx=0x7f6be800be80 comp rx=0 tx=0).stop 2024-03-24T06:37:47.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 shutdown_connections 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6bdc0a2260 0x7f6bdc0a4720 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf810ec00 0x7f6bf819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6bf8074430 0x7f6bf81a0780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 --2- 172.21.15.22:0/711043081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6bf8074d50 0x7f6bf81a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 >> 172.21.15.22:0/711043081 conn(0x7f6bf8079360 msgr2=0x7f6bf80ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 shutdown_connections 2024-03-24T06:37:47.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:47.173+0000 7f6bfe000700 1 -- 172.21.15.22:0/711043081 wait complete. 2024-03-24T06:37:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/711043081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:47.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/711043081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:47.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/711043081' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:48.044 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T06:37:48.044 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T06:37:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:48 smithi155 ceph-mon[36262]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 KiB/s wr, 16 op/s 2024-03-24T06:37:48.869 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:48 smithi022 ceph-mon[32003]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 KiB/s wr, 16 op/s 2024-03-24T06:37:48.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:48 smithi143 ceph-mon[36520]: pgmap v345: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 KiB/s wr, 16 op/s 2024-03-24T06:37:50.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 -- 172.21.15.22:0/1623432651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102cf0 msgr2=0x7fa700106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 --2- 172.21.15.22:0/1623432651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102cf0 0x7fa700106d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009990 tx=0x7fa6ec009ca0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 -- 172.21.15.22:0/1623432651 shutdown_connections 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 --2- 172.21.15.22:0/1623432651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 0x7fa70010eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 --2- 172.21.15.22:0/1623432651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102340 0x7fa700102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 --2- 172.21.15.22:0/1623432651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102cf0 0x7fa700106d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 -- 172.21.15.22:0/1623432651 >> 172.21.15.22:0/1623432651 conn(0x7fa7000fdbd0 msgr2=0x7fa7000ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 -- 172.21.15.22:0/1623432651 shutdown_connections 2024-03-24T06:37:50.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.823+0000 7fa706200700 1 -- 172.21.15.22:0/1623432651 wait complete. 2024-03-24T06:37:50.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.824+0000 7fa706200700 1 Processor -- start 2024-03-24T06:37:50.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.825+0000 7fa706200700 1 -- start start 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 0x7fa700074f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001a6aa0 con 0x7fa700107420 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001a6c10 con 0x7fa700102340 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa706200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001a6dd0 con 0x7fa700102cf0 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa704e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa6ffe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:50.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa704e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44540/0 (socket says 172.21.15.22:44540) 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa6ffe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42282/0 (socket says 172.21.15.22:42282) 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa704e00700 1 -- 172.21.15.22:0/3837896881 learned_addr learned my addr 172.21.15.22:0/3837896881 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.826+0000 7fa705800700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 0x7fa700074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 -- 172.21.15.22:0/3837896881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 msgr2=0x7fa700074500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 -- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 msgr2=0x7fa700074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 0x7fa700074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 -- 172.21.15.22:0/3837896881 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec0095f0 con 0x7fa700102cf0 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa704e00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:37:50.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6ffe00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa70006ab50 tx=0x7fa6ec005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec01d070 con 0x7fa700102cf0 2024-03-24T06:37:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6ec00b610 con 0x7fa700102cf0 2024-03-24T06:37:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec022e90 con 0x7fa700102cf0 2024-03-24T06:37:50.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.827+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7001a7050 con 0x7fa700102cf0 2024-03-24T06:37:50.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.828+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7001a7460 con 0x7fa700102cf0 2024-03-24T06:37:50.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.829+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa70010a740 con 0x7fa700102cf0 2024-03-24T06:37:50.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.830+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa6ec022590 con 0x7fa700102cf0 2024-03-24T06:37:50.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.832+0000 7fa6fd600700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 0x7fa6e80a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:37:50.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.832+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa6ec0cf170 con 0x7fa700102cf0 2024-03-24T06:37:50.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.832+0000 7fa704e00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 0x7fa6e80a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:37:50.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.833+0000 7fa704e00700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 0x7fa6e80a46a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4011550 tx=0x7fa6f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:37:50.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:50.840+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa6ec05e280 con 0x7fa700102cf0 2024-03-24T06:37:50.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:50 smithi022 ceph-mon[32003]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 14 op/s 2024-03-24T06:37:51.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.071+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7000759e0 con 0x7fa700102cf0 2024-03-24T06:37:51.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.073+0000 7fa6fd600700 1 -- 172.21.15.22:0/3837896881 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24137 (secure 0 0 0) 0x7fa6ec097dd0 con 0x7fa700102cf0 2024-03-24T06:37:51.072 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:37:51.072 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":89,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:37:01.829415+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.075+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 msgr2=0x7fa6e80a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.075+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 0x7fa6e80a46a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4011550 tx=0x7fa6f4009260 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.075+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 msgr2=0x7fa700074a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.075+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa70006ab50 tx=0x7fa6ec005740 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 shutdown_connections 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa700107420 0x7fa700074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa6e80a21e0 0x7fa6e80a46a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa700102340 0x7fa700074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 --2- 172.21.15.22:0/3837896881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa700102cf0 0x7fa700074a40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 >> 172.21.15.22:0/3837896881 conn(0x7fa7000fdbd0 msgr2=0x7fa7000fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 shutdown_connections 2024-03-24T06:37:51.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:37:51.076+0000 7fa706200700 1 -- 172.21.15.22:0/3837896881 wait complete. 2024-03-24T06:37:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:50 smithi143 ceph-mon[36520]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 14 op/s 2024-03-24T06:37:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:50 smithi155 ceph-mon[36262]: pgmap v346: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 14 op/s 2024-03-24T06:37:51.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T06:37:51.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T06:37:51.987 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T06:37:51.987 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T06:37:51.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T06:37:51.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T06:37:51.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T06:37:51.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T06:37:51.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T06:37:51.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T06:37:51.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T06:37:51.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T06:37:52.130 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:51 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3837896881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:52.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:51 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3837896881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:52.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:51 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3837896881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:37:53.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:52 smithi022 ceph-mon[32003]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T06:37:53.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:52 smithi143 ceph-mon[36520]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T06:37:53.230 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.261 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.274 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:52 smithi155 ceph-mon[36262]: pgmap v347: 129 pgs: 129 active+clean; 640 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-24T06:37:53.284 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.291 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.295 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.303 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.314 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:53.603 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:55.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:54 smithi022 ceph-mon[32003]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-24T06:37:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:54 smithi143 ceph-mon[36520]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-24T06:37:55.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:54 smithi155 ceph-mon[36262]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-24T06:37:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-24T06:37:56.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T06:37:56.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-24T06:37:56.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:55 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-24T06:37:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T06:37:56.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-24T06:37:56.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:55 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-24T06:37:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-24T06:37:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-24T06:37:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-24T06:37:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-24T06:37:56.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:55 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-24T06:37:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T06:37:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-24T06:37:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T06:37:57.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-24T06:37:57.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:56 smithi022 ceph-mon[32003]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-24T06:37:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T06:37:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-24T06:37:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-24T06:37:57.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:56 smithi143 ceph-mon[36520]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-03-24T06:37:57.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-24T06:37:57.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:56 smithi155 ceph-mon[36262]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T06:37:57.832 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:37:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:57 smithi022 ceph-mon[32003]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T06:37:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:37:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:57 smithi143 ceph-mon[36520]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T06:37:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:37:58.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:57 smithi155 ceph-mon[36262]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T06:37:58.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:37:59.028 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:37:58 smithi022 ceph-mon[32003]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-24T06:37:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:37:58 smithi143 ceph-mon[36520]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-24T06:37:59.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:37:58 smithi155 ceph-mon[36262]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-24T06:38:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:00 smithi143 ceph-mon[36520]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 9 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:00 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:00 smithi155 ceph-mon[36262]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 9 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:00 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.562 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:00 smithi022 ceph-mon[32003]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 9 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:00.562 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.562 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.562 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.562 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:00 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:00.987 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:02 smithi143 ceph-mon[36520]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:02 smithi155 ceph-mon[36262]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:02.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:02 smithi022 ceph-mon[32003]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:04.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:04 smithi022 ceph-mon[32003]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:04.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 -- 172.21.15.22:0/4224628783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c108250 msgr2=0x7f607c108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 --2- 172.21.15.22:0/4224628783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c108250 0x7f607c108630 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f60700098f0 tx=0x7f6070009c00 comp rx=0 tx=0).stop 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 -- 172.21.15.22:0/4224628783 shutdown_connections 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 --2- 172.21.15.22:0/4224628783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c06a850 0x7f607c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 --2- 172.21.15.22:0/4224628783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c108250 0x7f607c108630 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 --2- 172.21.15.22:0/4224628783 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c069eb0 0x7f607c06a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 -- 172.21.15.22:0/4224628783 >> 172.21.15.22:0/4224628783 conn(0x7f607c078030 msgr2=0x7f607c078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 -- 172.21.15.22:0/4224628783 shutdown_connections 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.839+0000 7f6083c00700 1 -- 172.21.15.22:0/4224628783 wait complete. 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.840+0000 7f6083c00700 1 Processor -- start 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.840+0000 7f6083c00700 1 -- start start 2024-03-24T06:38:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c069eb0 0x7f607c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c06a850 0x7f607c1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c108250 0x7f607c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f607c19aec0 con 0x7f607c108250 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f607c19b030 con 0x7f607c06a850 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6083c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f607c19b1f0 con 0x7f607c069eb0 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6081200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c069eb0 0x7f607c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6081200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c069eb0 0x7f607c1a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:36784/0 (socket says 172.21.15.22:36784) 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.841+0000 7f6081200700 1 -- 172.21.15.22:0/2130717668 learned_addr learned my addr 172.21.15.22:0/2130717668 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.842+0000 7f607be00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c06a850 0x7f607c1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.842+0000 7f607be00700 1 -- 172.21.15.22:0/2130717668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c069eb0 msgr2=0x7f607c1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.842+0000 7f607be00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f607c069eb0 0x7f607c1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.842+0000 7f607be00700 1 -- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c108250 msgr2=0x7f607c19a810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.842+0000 7f6081c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c108250 0x7f607c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f607be00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c108250 0x7f607c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f607be00700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6070009520 con 0x7f607c06a850 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f6081c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f607c108250 0x7f607c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f607be00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c06a850 0x7f607c1a0bd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f606800e9f0 tx=0x7f606800edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606800cbd0 con 0x7f607c06a850 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f606800cd30 con 0x7f607c06a850 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068018710 con 0x7f607c06a850 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.843+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c19b4d0 con 0x7f607c06a850 2024-03-24T06:38:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.844+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c1a76c0 con 0x7f607c06a850 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.845+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f607c112530 con 0x7f607c06a850 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.847+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6068018870 con 0x7f607c06a850 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.849+0000 7f6079600700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f60640a21e0 0x7f60640a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.849+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6068014070 con 0x7f607c06a850 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.849+0000 7f6081200700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f60640a21e0 0x7f60640a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.849+0000 7f6079600700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] conn(0x7f60640ad340 0x7f60640af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.850+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60640afeb0 con 0x7f60640ad340 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.850+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f60680ceec0 con 0x7f607c06a850 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.850+0000 7f6081c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] conn(0x7f60640ad340 0x7f60640af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.850+0000 7f6081200700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f60640a21e0 0x7f60640a46a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f607000a010 tx=0x7f6070019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.850+0000 7f6081c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] conn(0x7f60640ad340 0x7f60640af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.851+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== osd.4 v2:172.21.15.143:6808/4206631935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f60640afeb0 con 0x7f60640ad340 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.870+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 --> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f607c10ddc0 con 0x7f60640ad340 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.870+0000 7f6079600700 1 -- 172.21.15.22:0/2130717668 <== osd.4 v2:172.21.15.143:6808/4206631935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f607c10ddc0 con 0x7f60640ad340 2024-03-24T06:38:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] conn(0x7f60640ad340 msgr2=0x7f60640af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] conn(0x7f60640ad340 0x7f60640af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f60640a21e0 msgr2=0x7f60640a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f60640a21e0 0x7f60640a46a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f607000a010 tx=0x7f6070019040 comp rx=0 tx=0).stop 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c06a850 msgr2=0x7f607c1a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 --2- 172.21.15.22:0/2130717668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f607c06a850 0x7f607c1a0bd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f606800e9f0 tx=0x7f606800edb0 comp rx=0 tx=0).stop 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6081c00700 1 -- 172.21.15.22:0/2130717668 reap_dead start 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 shutdown_connections 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 >> 172.21.15.22:0/2130717668 conn(0x7f607c078030 msgr2=0x7f607c10d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 shutdown_connections 2024-03-24T06:38:04.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:03.871+0000 7f6083c00700 1 -- 172.21.15.22:0/2130717668 wait complete. 2024-03-24T06:38:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:04 smithi155 ceph-mon[36262]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:04 smithi143 ceph-mon[36520]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 8 op/s; 80 B/s, 0 objects/s recovering 2024-03-24T06:38:05.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 -- 172.21.15.22:0/3274988803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec108250 msgr2=0x7fabec108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3274988803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec108250 0x7fabec108630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fabe00099c0 tx=0x7fabe0009cd0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 -- 172.21.15.22:0/3274988803 shutdown_connections 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3274988803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec06a850 0x7fabec10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3274988803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec108250 0x7fabec108630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3274988803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec06a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.390+0000 7fabf3c00700 1 -- 172.21.15.22:0/3274988803 >> 172.21.15.22:0/3274988803 conn(0x7fabec078030 msgr2=0x7fabec078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.391+0000 7fabf3c00700 1 -- 172.21.15.22:0/3274988803 shutdown_connections 2024-03-24T06:38:05.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.391+0000 7fabf3c00700 1 -- 172.21.15.22:0/3274988803 wait complete. 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.392+0000 7fabf3c00700 1 Processor -- start 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.392+0000 7fabf3c00700 1 -- start start 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.392+0000 7fabf3c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec1011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.392+0000 7fabf3c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec06a850 0x7fabec101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.392+0000 7fabf3c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec108250 0x7fabec101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:05.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf3c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1a6b10 con 0x7fabec108250 2024-03-24T06:38:05.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf3c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1a6c80 con 0x7fabec069eb0 2024-03-24T06:38:05.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf3c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabec1a6e40 con 0x7fabec06a850 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf1200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec1011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec06a850 0x7fabec101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf1c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec108250 0x7fabec101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf1200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec1011d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58916/0 (socket says 172.21.15.22:58916) 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf1200700 1 -- 172.21.15.22:0/3026073408 learned_addr learned my addr 172.21.15.22:0/3026073408 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 -- 172.21.15.22:0/3026073408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 msgr2=0x7fabec1011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec1011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 -- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec108250 msgr2=0x7fabec101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec108250 0x7fabec101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabebe00700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe00095f0 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.393+0000 7fabf1200700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabec069eb0 0x7fabec1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabebe00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec06a850 0x7fabec101710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fabd800e9f0 tx=0x7fabd800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd800c9c0 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd8010430 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8018680 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec1a7120 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.394+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec1a75d0 con 0x7fabec06a850 2024-03-24T06:38:05.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.395+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fabec112530 con 0x7fabec06a850 2024-03-24T06:38:05.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.397+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fabd8010910 con 0x7fabec06a850 2024-03-24T06:38:05.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabe9600700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fabd40a2260 0x7fabd40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fabd8014070 con 0x7fabec06a850 2024-03-24T06:38:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabf1200700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fabd40a2260 0x7fabd40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabe9600700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] conn(0x7fabd40ad3c0 0x7fabd40af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:05.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabd40aff30 con 0x7fabd40ad3c0 2024-03-24T06:38:05.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.400+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fabd80d06f0 con 0x7fabec06a850 2024-03-24T06:38:05.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.401+0000 7fabf1c00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] conn(0x7fabd40ad3c0 0x7fabd40af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:05.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.401+0000 7fabf1200700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fabd40a2260 0x7fabd40a4720 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fabe0000c00 tx=0x7fabe0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:05.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.401+0000 7fabf1c00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] conn(0x7fabd40ad3c0 0x7fabd40af860 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:05.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.402+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== osd.9 v2:172.21.15.22:6826/2177893113 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fabd40aff30 con 0x7fabd40ad3c0 2024-03-24T06:38:05.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.426+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 --> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fabec062910 con 0x7fabd40ad3c0 2024-03-24T06:38:05.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.427+0000 7fabe9600700 1 -- 172.21.15.22:0/3026073408 <== osd.9 v2:172.21.15.22:6826/2177893113 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fabec062910 con 0x7fabd40ad3c0 2024-03-24T06:38:05.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.427+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] conn(0x7fabd40ad3c0 msgr2=0x7fabd40af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.427+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] conn(0x7fabd40ad3c0 0x7fabd40af860 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fabd40a2260 msgr2=0x7fabd40a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fabd40a2260 0x7fabd40a4720 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fabe0000c00 tx=0x7fabe0005fb0 comp rx=0 tx=0).stop 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec06a850 msgr2=0x7fabec101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 --2- 172.21.15.22:0/3026073408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabec06a850 0x7fabec101710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fabd800e9f0 tx=0x7fabd800ed00 comp rx=0 tx=0).stop 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf1c00700 1 -- 172.21.15.22:0/3026073408 reap_dead start 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 shutdown_connections 2024-03-24T06:38:05.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 >> 172.21.15.22:0/3026073408 conn(0x7fabec078030 msgr2=0x7fabec0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:05.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 shutdown_connections 2024-03-24T06:38:05.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:05.428+0000 7fabf3c00700 1 -- 172.21.15.22:0/3026073408 wait complete. 2024-03-24T06:38:06.351 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout:141733920837 2024-03-24T06:38:06.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T06:38:06.440 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:06 smithi022 ceph-mon[32003]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 69 B/s, 0 objects/s recovering 2024-03-24T06:38:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:06 smithi155 ceph-mon[36262]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 69 B/s, 0 objects/s recovering 2024-03-24T06:38:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:06 smithi143 ceph-mon[36520]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s; 69 B/s, 0 objects/s recovering 2024-03-24T06:38:08.477 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:08 smithi022 ceph-mon[32003]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 62 B/s, 0 objects/s recovering 2024-03-24T06:38:08.526 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:08 smithi155 ceph-mon[36262]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 62 B/s, 0 objects/s recovering 2024-03-24T06:38:08.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:08 smithi143 ceph-mon[36520]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 62 B/s, 0 objects/s recovering 2024-03-24T06:38:08.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 -- 172.21.15.22:0/1854520297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 msgr2=0x7f58a410f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 --2- 172.21.15.22:0/1854520297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 0x7f58a410f090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f58a000b0a0 tx=0x7f58a000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 -- 172.21.15.22:0/1854520297 shutdown_connections 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 --2- 172.21.15.22:0/1854520297 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58a4108250 0x7f58a4108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 --2- 172.21.15.22:0/1854520297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 0x7f58a410f090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 --2- 172.21.15.22:0/1854520297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a406a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 -- 172.21.15.22:0/1854520297 >> 172.21.15.22:0/1854520297 conn(0x7f58a4078030 msgr2=0x7f58a4078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 -- 172.21.15.22:0/1854520297 shutdown_connections 2024-03-24T06:38:08.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.817+0000 7f58ac800700 1 -- 172.21.15.22:0/1854520297 wait complete. 2024-03-24T06:38:08.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.819+0000 7f58ac800700 1 Processor -- start 2024-03-24T06:38:08.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.819+0000 7f58ac800700 1 -- start start 2024-03-24T06:38:08.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a41011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 0x7f58a4101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58a4108250 0x7f58a4101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41a6cf0 con 0x7f58a4108250 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41a6e60 con 0x7f58a406a850 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58ac800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41a7020 con 0x7f58a4069eb0 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58a9e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a41011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.820+0000 7f58a9e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a41011d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37112/0 (socket says 172.21.15.22:37112) 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 -- 172.21.15.22:0/44459502 learned_addr learned my addr 172.21.15.22:0/44459502 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:08.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9400700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 0x7f58a4101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58aa800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58a4108250 0x7f58a4101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 -- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 msgr2=0x7f58a4101710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58a406a850 0x7f58a4101710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 -- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58a4108250 msgr2=0x7f58a4101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58a4108250 0x7f58a4101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a0009d00 con 0x7f58a4069eb0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.821+0000 7f58a9e00700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a41011d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f589800e9a0 tx=0x7f589800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.822+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898009790 con 0x7f58a4069eb0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.822+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f589800cbd0 con 0x7f58a4069eb0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.822+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a41a7300 con 0x7f58a4069eb0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.822+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898018680 con 0x7f58a4069eb0 2024-03-24T06:38:08.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.822+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41a7850 con 0x7f58a4069eb0 2024-03-24T06:38:08.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.823+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58a4112500 con 0x7f58a4069eb0 2024-03-24T06:38:08.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.825+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f589800cd40 con 0x7f58a4069eb0 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.826+0000 7f589ea00700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f588c0a2260 0x7f588c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.826+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5898014070 con 0x7f58a4069eb0 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.827+0000 7f58a9400700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f588c0a2260 0x7f588c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.827+0000 7f589ea00700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] conn(0x7f588c0ad3c0 0x7f588c0af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.827+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f588c0aff30 con 0x7f588c0ad3c0 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.827+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f58980ced80 con 0x7f58a4069eb0 2024-03-24T06:38:08.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.827+0000 7f58aa800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] conn(0x7f588c0ad3c0 0x7f588c0af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:08.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.828+0000 7f58aa800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] conn(0x7f588c0ad3c0 0x7f588c0af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:08.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.828+0000 7f58a9400700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f588c0a2260 0x7f588c0a4720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f58a4078610 tx=0x7f5894009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:08.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.829+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== osd.1 v2:172.21.15.143:6800/2665804124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f588c0aff30 con 0x7f588c0ad3c0 2024-03-24T06:38:08.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.847+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 --> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f58a40628c0 con 0x7f588c0ad3c0 2024-03-24T06:38:08.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f589ea00700 1 -- 172.21.15.22:0/44459502 <== osd.1 v2:172.21.15.143:6800/2665804124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f58a40628c0 con 0x7f588c0ad3c0 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] conn(0x7f588c0ad3c0 msgr2=0x7f588c0af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] conn(0x7f588c0ad3c0 0x7f588c0af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f588c0a2260 msgr2=0x7f588c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f588c0a2260 0x7f588c0a4720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f58a4078610 tx=0x7f5894009260 comp rx=0 tx=0).stop 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 msgr2=0x7f58a41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58ac800700 1 --2- 172.21.15.22:0/44459502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58a4069eb0 0x7f58a41011d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f589800e9a0 tx=0x7f589800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.848+0000 7f58aa800700 1 -- 172.21.15.22:0/44459502 reap_dead start 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.849+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 shutdown_connections 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.849+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 >> 172.21.15.22:0/44459502 conn(0x7f58a4078030 msgr2=0x7f58a40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.849+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 shutdown_connections 2024-03-24T06:38:08.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:08.849+0000 7f58ac800700 1 -- 172.21.15.22:0/44459502 wait complete. 2024-03-24T06:38:10.490 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:10 smithi022 ceph-mon[32003]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 53 B/s, 0 objects/s recovering 2024-03-24T06:38:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:10 smithi155 ceph-mon[36262]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 53 B/s, 0 objects/s recovering 2024-03-24T06:38:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:10 smithi143 ceph-mon[36520]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 53 B/s, 0 objects/s recovering 2024-03-24T06:38:11.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 -- 172.21.15.22:0/1982197696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67406a9a0 msgr2=0x7fe67406ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 --2- 172.21.15.22:0/1982197696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67406a9a0 0x7fe67406ad80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6640098f0 tx=0x7fe664009c00 comp rx=0 tx=0).stop 2024-03-24T06:38:11.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 -- 172.21.15.22:0/1982197696 shutdown_connections 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 --2- 172.21.15.22:0/1982197696 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe674107580 0x7fe67410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 --2- 172.21.15.22:0/1982197696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe67406a9a0 0x7fe67406ad80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 --2- 172.21.15.22:0/1982197696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406b350 0x7fe674106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 -- 172.21.15.22:0/1982197696 >> 172.21.15.22:0/1982197696 conn(0x7fe674077150 msgr2=0x7fe674077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 -- 172.21.15.22:0/1982197696 shutdown_connections 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.770+0000 7fe67a000700 1 -- 172.21.15.22:0/1982197696 wait complete. 2024-03-24T06:38:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.772+0000 7fe67a000700 1 Processor -- start 2024-03-24T06:38:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.772+0000 7fe67a000700 1 -- start start 2024-03-24T06:38:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.772+0000 7fe67a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 0x7fe6741a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe67a000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe67406b350 0x7fe6741a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe67a000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe674107580 0x7fe67419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe67a000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe67419aeb0 con 0x7fe67406b350 2024-03-24T06:38:11.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe67a000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe67419b020 con 0x7fe674107580 2024-03-24T06:38:11.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe67a000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe67419b1e0 con 0x7fe67406a9a0 2024-03-24T06:38:11.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 0x7fe6741a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 0x7fe6741a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37154/0 (socket says 172.21.15.22:37154) 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 -- 172.21.15.22:0/395736805 learned_addr learned my addr 172.21.15.22:0/395736805 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673e00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe674107580 0x7fe67419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe672a00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe67406b350 0x7fe6741a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 -- 172.21.15.22:0/395736805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe674107580 msgr2=0x7fe67419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe674107580 0x7fe67419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 -- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe67406b350 msgr2=0x7fe6741a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.773+0000 7fe673400700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe67406b350 0x7fe6741a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.774+0000 7fe673400700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6600095f0 con 0x7fe67406a9a0 2024-03-24T06:38:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.774+0000 7fe672a00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe67406b350 0x7fe6741a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.774+0000 7fe673400700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 0x7fe6741a0680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe664005e90 tx=0x7fe66400fa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.775+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66401c070 con 0x7fe67406a9a0 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.775+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe66400ba40 con 0x7fe67406a9a0 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.775+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe664003cd0 con 0x7fe67406a9a0 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.775+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe664009520 con 0x7fe67406a9a0 2024-03-24T06:38:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.775+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67419b690 con 0x7fe67406a9a0 2024-03-24T06:38:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.776+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe674103930 con 0x7fe67406a9a0 2024-03-24T06:38:11.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.780+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe66400b560 con 0x7fe67406a9a0 2024-03-24T06:38:11.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.781+0000 7fe66be00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6580a2260 0x7fe6580a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.781+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe6640cf490 con 0x7fe67406a9a0 2024-03-24T06:38:11.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.781+0000 7fe672a00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6580a2260 0x7fe6580a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.782+0000 7fe66be00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] conn(0x7fe6580ad3c0 0x7fe6580af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.782+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6580aff30 con 0x7fe6580ad3c0 2024-03-24T06:38:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.782+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe6640cf800 con 0x7fe67406a9a0 2024-03-24T06:38:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.782+0000 7fe673e00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] conn(0x7fe6580ad3c0 0x7fe6580af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.782+0000 7fe672a00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6580a2260 0x7fe6580a4720 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe67410efe0 tx=0x7fe660009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.783+0000 7fe673e00700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] conn(0x7fe6580ad3c0 0x7fe6580af860 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.784+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== osd.11 v2:172.21.15.155:6824/2681113949 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe6580aff30 con 0x7fe6580ad3c0 2024-03-24T06:38:11.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.802+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 --> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe6741125e0 con 0x7fe6580ad3c0 2024-03-24T06:38:11.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.802+0000 7fe66be00700 1 -- 172.21.15.22:0/395736805 <== osd.11 v2:172.21.15.155:6824/2681113949 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe6741125e0 con 0x7fe6580ad3c0 2024-03-24T06:38:11.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] conn(0x7fe6580ad3c0 msgr2=0x7fe6580af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] conn(0x7fe6580ad3c0 0x7fe6580af860 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6580a2260 msgr2=0x7fe6580a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe6580a2260 0x7fe6580a4720 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe67410efe0 tx=0x7fe660009310 comp rx=0 tx=0).stop 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 msgr2=0x7fe6741a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 --2- 172.21.15.22:0/395736805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe67406a9a0 0x7fe6741a0680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe664005e90 tx=0x7fe66400fa90 comp rx=0 tx=0).stop 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe673e00700 1 -- 172.21.15.22:0/395736805 reap_dead start 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 shutdown_connections 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 >> 172.21.15.22:0/395736805 conn(0x7fe674077150 msgr2=0x7fe67410d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 shutdown_connections 2024-03-24T06:38:11.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.803+0000 7fe67a000700 1 -- 172.21.15.22:0/395736805 wait complete. 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.679+0000 7ff01de00700 1 -- 172.21.15.22:0/1858956290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 msgr2=0x7ff018108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.679+0000 7ff01de00700 1 --2- 172.21.15.22:0/1858956290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 0x7ff018108630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0080099c0 tx=0x7ff008009cd0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.679+0000 7ff01de00700 1 -- 172.21.15.22:0/1858956290 shutdown_connections 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 --2- 172.21.15.22:0/1858956290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff01806a850 0x7ff01810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 --2- 172.21.15.22:0/1858956290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 0x7ff018108630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 --2- 172.21.15.22:0/1858956290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff01806a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 -- 172.21.15.22:0/1858956290 >> 172.21.15.22:0/1858956290 conn(0x7ff018078030 msgr2=0x7ff018078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:11.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 -- 172.21.15.22:0/1858956290 shutdown_connections 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.680+0000 7ff01de00700 1 -- 172.21.15.22:0/1858956290 wait complete. 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.681+0000 7ff01de00700 1 Processor -- start 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.681+0000 7ff01de00700 1 -- start start 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff0181a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff01806a850 0x7ff0181a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 0x7ff01819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff01819af10 con 0x7ff01806a850 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff01819b080 con 0x7ff018069eb0 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff01de00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff01819b240 con 0x7ff018108250 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff016a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff0181a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff016a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff0181a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58938/0 (socket says 172.21.15.22:58938) 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.682+0000 7ff016a00700 1 -- 172.21.15.22:0/301238443 learned_addr learned my addr 172.21.15.22:0/301238443 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016000700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff01806a850 0x7ff0181a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff017400700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 0x7ff01819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 -- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 msgr2=0x7ff01819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff018108250 0x7ff01819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 -- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff01806a850 msgr2=0x7ff0181a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff01806a850 0x7ff0181a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff004009520 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff016a00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff0181a0690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c010 tx=0x7ff00800f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00801d070 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff008003680 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.683+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008022ea0 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.684+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0080095f0 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.684+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0181a7440 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.685+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff018112530 con 0x7ff018069eb0 2024-03-24T06:38:11.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.687+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff008022470 con 0x7ff018069eb0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff01ca00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feffc0a21e0 0x7feffc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff0080cfa80 con 0x7ff018069eb0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff016000700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feffc0a21e0 0x7feffc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff01ca00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] conn(0x7feffc0ad340 0x7feffc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feffc0afeb0 con 0x7feffc0ad340 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.689+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ff0080cfdf0 con 0x7ff018069eb0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.690+0000 7ff017400700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] conn(0x7feffc0ad340 0x7feffc0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.690+0000 7ff016000700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feffc0a21e0 0x7feffc0a46a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff01819bf20 tx=0x7ff004009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.690+0000 7ff017400700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] conn(0x7feffc0ad340 0x7feffc0af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.691+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== osd.8 v2:172.21.15.155:6816/3178166768 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feffc0afeb0 con 0x7feffc0ad340 2024-03-24T06:38:11.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.710+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 --> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff01810ddc0 con 0x7feffc0ad340 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.710+0000 7ff01ca00700 1 -- 172.21.15.22:0/301238443 <== osd.8 v2:172.21.15.155:6816/3178166768 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff01810ddc0 con 0x7feffc0ad340 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] conn(0x7feffc0ad340 msgr2=0x7feffc0af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] conn(0x7feffc0ad340 0x7feffc0af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feffc0a21e0 msgr2=0x7feffc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7feffc0a21e0 0x7feffc0a46a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff01819bf20 tx=0x7ff004009260 comp rx=0 tx=0).stop 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 msgr2=0x7ff0181a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 --2- 172.21.15.22:0/301238443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff018069eb0 0x7ff0181a0690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c010 tx=0x7ff00800f740 comp rx=0 tx=0).stop 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff017400700 1 -- 172.21.15.22:0/301238443 reap_dead start 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 shutdown_connections 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 >> 172.21.15.22:0/301238443 conn(0x7ff018078030 msgr2=0x7ff01810d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:11.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 shutdown_connections 2024-03-24T06:38:11.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:11.711+0000 7ff01de00700 1 -- 172.21.15.22:0/301238443 wait complete. 2024-03-24T06:38:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:12 smithi155 ceph-mon[36262]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:12.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:12 smithi022 ceph-mon[32003]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:12.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:12 smithi143 ceph-mon[36520]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 -- 172.21.15.22:0/4081778839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c077050 msgr2=0x7fcb9c112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 --2- 172.21.15.22:0/4081778839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c077050 0x7fcb9c112b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009990 tx=0x7fcb88009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 -- 172.21.15.22:0/4081778839 shutdown_connections 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 --2- 172.21.15.22:0/4081778839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 0x7fcb9c076b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 --2- 172.21.15.22:0/4081778839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c1130b0 0x7fcb9c1154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 --2- 172.21.15.22:0/4081778839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c077050 0x7fcb9c112b70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 -- 172.21.15.22:0/4081778839 >> 172.21.15.22:0/4081778839 conn(0x7fcb9c0ffee0 msgr2=0x7fcb9c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:12.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.957+0000 7fcba1e00700 1 -- 172.21.15.22:0/4081778839 shutdown_connections 2024-03-24T06:38:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.958+0000 7fcba1e00700 1 -- 172.21.15.22:0/4081778839 wait complete. 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.959+0000 7fcba1e00700 1 Processor -- start 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.959+0000 7fcba1e00700 1 -- start start 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.959+0000 7fcba1e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 0x7fcb9c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.959+0000 7fcba1e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 0x7fcb9c19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.959+0000 7fcba1e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c1130b0 0x7fcb9c1a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcba1e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1a3dd0 con 0x7fcb9c076730 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcba1e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1a3f40 con 0x7fcb9c1130b0 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcba1e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1a4100 con 0x7fcb9c077050 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9aa00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 0x7fcb9c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 0x7fcb9c19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9aa00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 0x7fcb9c19eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33926/0 (socket says 172.21.15.22:33926) 2024-03-24T06:38:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 0x7fcb9c19f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37192/0 (socket says 172.21.15.22:37192) 2024-03-24T06:38:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9aa00700 1 -- 172.21.15.22:0/3086758273 learned_addr learned my addr 172.21.15.22:0/3086758273 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9b400700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c1130b0 0x7fcb9c1a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 -- 172.21.15.22:0/3086758273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c1130b0 msgr2=0x7fcb9c1a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c1130b0 0x7fcb9c1a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 -- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 msgr2=0x7fcb9c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb9c076730 0x7fcb9c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.960+0000 7fcb9a000700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb880095f0 con 0x7fcb9c077050 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcb9a000700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 0x7fcb9c19f430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb88005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8801d070 con 0x7fcb9c077050 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9c1a4380 con 0x7fcb9c077050 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8800b610 con 0x7fcb9c077050 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88022e00 con 0x7fcb9c077050 2024-03-24T06:38:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.961+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9c1abc40 con 0x7fcb9c077050 2024-03-24T06:38:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.962+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb9c118c10 con 0x7fcb9c077050 2024-03-24T06:38:12.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.964+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fcb88022500 con 0x7fcb9c077050 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcba0a00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcb800a21e0 0x7fcb800a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcb880cfe60 con 0x7fcb9c077050 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcb9aa00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcb800a21e0 0x7fcb800a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcba0a00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] conn(0x7fcb800ad340 0x7fcb800af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb800afeb0 con 0x7fcb800ad340 2024-03-24T06:38:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.966+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fcb880d01d0 con 0x7fcb9c077050 2024-03-24T06:38:12.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.967+0000 7fcb9b400700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] conn(0x7fcb800ad340 0x7fcb800af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:12.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.967+0000 7fcb9aa00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcb800a21e0 0x7fcb800a46a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009d20 tx=0x7fcb8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:12.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.967+0000 7fcb9b400700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] conn(0x7fcb800ad340 0x7fcb800af7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.968+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== osd.3 v2:172.21.15.22:6810/3163117945 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb800afeb0 con 0x7fcb800ad340 2024-03-24T06:38:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.003+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 --> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb9c062890 con 0x7fcb800ad340 2024-03-24T06:38:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.003+0000 7fcba0a00700 1 -- 172.21.15.22:0/3086758273 <== osd.3 v2:172.21.15.22:6810/3163117945 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcb9c062890 con 0x7fcb800ad340 2024-03-24T06:38:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] conn(0x7fcb800ad340 msgr2=0x7fcb800af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] conn(0x7fcb800ad340 0x7fcb800af7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcb800a21e0 msgr2=0x7fcb800a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcb800a21e0 0x7fcb800a46a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009d20 tx=0x7fcb8c009260 comp rx=0 tx=0).stop 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 msgr2=0x7fcb9c19f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 --2- 172.21.15.22:0/3086758273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb9c077050 0x7fcb9c19f430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098e0 tx=0x7fcb88005720 comp rx=0 tx=0).stop 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcb9b400700 1 -- 172.21.15.22:0/3086758273 reap_dead start 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 shutdown_connections 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.004+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 >> 172.21.15.22:0/3086758273 conn(0x7fcb9c0ffee0 msgr2=0x7fcb9c1022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.005+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 shutdown_connections 2024-03-24T06:38:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.005+0000 7fcba1e00700 1 -- 172.21.15.22:0/3086758273 wait complete. 2024-03-24T06:38:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 -- 172.21.15.22:0/2950171892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9594107580 msgr2=0x7f959410f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 --2- 172.21.15.22:0/2950171892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9594107580 0x7f959410f150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f958c00b0a0 tx=0x7f958c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 -- 172.21.15.22:0/2950171892 shutdown_connections 2024-03-24T06:38:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 --2- 172.21.15.22:0/2950171892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9594107580 0x7f959410f150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 --2- 172.21.15.22:0/2950171892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95941024b0 0x7f9594102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 --2- 172.21.15.22:0/2950171892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9594102e60 0x7f9594106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 -- 172.21.15.22:0/2950171892 >> 172.21.15.22:0/2950171892 conn(0x7f9594079360 msgr2=0x7f9594079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 -- 172.21.15.22:0/2950171892 shutdown_connections 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.229+0000 7f959a800700 1 -- 172.21.15.22:0/2950171892 wait complete. 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.230+0000 7f959a800700 1 Processor -- start 2024-03-24T06:38:13.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.230+0000 7f959a800700 1 -- start start 2024-03-24T06:38:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 0x7f95941a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9594102e60 0x7f95941a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9594107580 0x7f959419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f959419af50 con 0x7f9594107580 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f959419b090 con 0x7f95941024b0 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f959a800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f959419b250 con 0x7f9594102e60 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f9593400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 0x7f95941a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f9593400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 0x7f95941a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59030/0 (socket says 172.21.15.22:59030) 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f9593400700 1 -- 172.21.15.22:0/856127879 learned_addr learned my addr 172.21.15.22:0/856127879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.231+0000 7f9593e00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9594107580 0x7f959419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9592a00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9594102e60 0x7f95941a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 -- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9594102e60 msgr2=0x7f95941a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9594102e60 0x7f95941a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 -- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9594107580 msgr2=0x7f959419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9594107580 0x7f959419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c009d00 con 0x7f95941024b0 2024-03-24T06:38:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9592a00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9594102e60 0x7f95941a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f9593400700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 0x7f95941a0730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f958400e9a0 tx=0x7f958400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.232+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584009790 con 0x7f95941024b0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.233+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958400cbd0 con 0x7f95941024b0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.233+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959419b530 con 0x7f95941024b0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.233+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584018680 con 0x7f95941024b0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.233+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941a77d0 con 0x7f95941024b0 2024-03-24T06:38:13.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.234+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9574001030 con 0x7f95941024b0 2024-03-24T06:38:13.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.235+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f958400cd40 con 0x7f95941024b0 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f958be00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f95780a2260 0x7f95780a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9584014070 con 0x7f95941024b0 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f9592a00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f95780a2260 0x7f95780a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f958be00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] conn(0x7f95780ad3c0 0x7f95780af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95780aff30 con 0x7f95780ad3c0 2024-03-24T06:38:13.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f95840d0780 con 0x7f95941024b0 2024-03-24T06:38:13.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.237+0000 7f9593e00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] conn(0x7f95780ad3c0 0x7f95780af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.238+0000 7f9592a00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f95780a2260 0x7f95780a4720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9594076490 tx=0x7f9580009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.238+0000 7f9593e00700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] conn(0x7f95780ad3c0 0x7f95780af860 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.239+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== osd.5 v2:172.21.15.155:6808/455132245 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f95780aff30 con 0x7f95780ad3c0 2024-03-24T06:38:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.259+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 --> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9574002dc0 con 0x7f95780ad3c0 2024-03-24T06:38:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f958be00700 1 -- 172.21.15.22:0/856127879 <== osd.5 v2:172.21.15.155:6808/455132245 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9574002dc0 con 0x7f95780ad3c0 2024-03-24T06:38:13.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] conn(0x7f95780ad3c0 msgr2=0x7f95780af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] conn(0x7f95780ad3c0 0x7f95780af860 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f95780a2260 msgr2=0x7f95780a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f95780a2260 0x7f95780a4720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9594076490 tx=0x7f9580009260 comp rx=0 tx=0).stop 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 msgr2=0x7f95941a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.260+0000 7f959a800700 1 --2- 172.21.15.22:0/856127879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95941024b0 0x7f95941a0730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f958400e9a0 tx=0x7f958400ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.261+0000 7f9593e00700 1 -- 172.21.15.22:0/856127879 reap_dead start 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.261+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 shutdown_connections 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.261+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 >> 172.21.15.22:0/856127879 conn(0x7f9594079360 msgr2=0x7f959410d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.261+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 shutdown_connections 2024-03-24T06:38:13.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.261+0000 7f959a800700 1 -- 172.21.15.22:0/856127879 wait complete. 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.561+0000 7f058b400700 1 -- 172.21.15.22:0/1028054301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 msgr2=0x7f058410f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.561+0000 7f058b400700 1 --2- 172.21.15.22:0/1028054301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 0x7f058410f150 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f057c009f90 tx=0x7f057c00b330 comp rx=0 tx=0).stop 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 -- 172.21.15.22:0/1028054301 shutdown_connections 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 --2- 172.21.15.22:0/1028054301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05841024b0 0x7f0584102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 --2- 172.21.15.22:0/1028054301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 0x7f058410f150 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 --2- 172.21.15.22:0/1028054301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0584102e60 0x7f0584106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 -- 172.21.15.22:0/1028054301 >> 172.21.15.22:0/1028054301 conn(0x7f0584079360 msgr2=0x7f0584079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 -- 172.21.15.22:0/1028054301 shutdown_connections 2024-03-24T06:38:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.562+0000 7f058b400700 1 -- 172.21.15.22:0/1028054301 wait complete. 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 Processor -- start 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 -- start start 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 0x7f0584074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0584102e60 0x7f0584074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 0x7f0584074f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f05841a6cd0 con 0x7f0584102e60 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f05841a6e40 con 0x7f0584107580 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.563+0000 7f058b400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05841a7000 con 0x7f05841024b0 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 0x7f0584074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 0x7f0584074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37224/0 (socket says 172.21.15.22:37224) 2024-03-24T06:38:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 -- 172.21.15.22:0/3881496673 learned_addr learned my addr 172.21.15.22:0/3881496673 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0583e00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0584102e60 0x7f0584074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0589400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 0x7f0584074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 -- 172.21.15.22:0/3881496673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 msgr2=0x7f0584074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0584107580 0x7f0584074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 -- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0584102e60 msgr2=0x7f0584074a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0584102e60 0x7f0584074a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.564+0000 7f0588a00700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05700095f0 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f0588a00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 0x7f0584074500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f057800e9a0 tx=0x7f057800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578009790 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0578010430 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057c009c40 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578018680 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.565+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05841a7650 con 0x7f05841024b0 2024-03-24T06:38:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.566+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0564001030 con 0x7f05841024b0 2024-03-24T06:38:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.569+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f057800ca10 con 0x7f05841024b0 2024-03-24T06:38:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.570+0000 7f0581600700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f056c0a21e0 0x7f056c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.570+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0578014070 con 0x7f05841024b0 2024-03-24T06:38:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.570+0000 7f0583e00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f056c0a21e0 0x7f056c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.571+0000 7f0581600700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] conn(0x7f056c0ad340 0x7f056c0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.571+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f056c0afeb0 con 0x7f056c0ad340 2024-03-24T06:38:13.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.571+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f05780cecc0 con 0x7f05841024b0 2024-03-24T06:38:13.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.571+0000 7f0589400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] conn(0x7f056c0ad340 0x7f056c0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.571+0000 7f0583e00700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f056c0a21e0 0x7f056c0a46a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f05841089c0 tx=0x7f0570009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.572+0000 7f0589400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] conn(0x7f056c0ad340 0x7f056c0af7e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.573+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== osd.0 v2:172.21.15.22:6802/671685493 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f056c0afeb0 con 0x7f056c0ad340 2024-03-24T06:38:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.591+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 --> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0564002dc0 con 0x7f056c0ad340 2024-03-24T06:38:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f0581600700 1 -- 172.21.15.22:0/3881496673 <== osd.0 v2:172.21.15.22:6802/671685493 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0564002dc0 con 0x7f056c0ad340 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] conn(0x7f056c0ad340 msgr2=0x7f056c0af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] conn(0x7f056c0ad340 0x7f056c0af7e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f056c0a21e0 msgr2=0x7f056c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f056c0a21e0 0x7f056c0a46a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f05841089c0 tx=0x7f0570009310 comp rx=0 tx=0).stop 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 msgr2=0x7f0584074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f058b400700 1 --2- 172.21.15.22:0/3881496673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05841024b0 0x7f0584074500 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f057800e9a0 tx=0x7f057800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.592+0000 7f0589400700 1 -- 172.21.15.22:0/3881496673 reap_dead start 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.593+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 shutdown_connections 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.593+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 >> 172.21.15.22:0/3881496673 conn(0x7f0584079360 msgr2=0x7f0584100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.593+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 shutdown_connections 2024-03-24T06:38:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:13.593+0000 7f058b400700 1 -- 172.21.15.22:0/3881496673 wait complete. 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 -- 172.21.15.22:0/3992791352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c074d50 msgr2=0x7fbf2c10e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 --2- 172.21.15.22:0/3992791352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c074d50 0x7fbf2c10e6c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbf180098f0 tx=0x7fbf18009c00 comp rx=0 tx=0).stop 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 -- 172.21.15.22:0/3992791352 shutdown_connections 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 --2- 172.21.15.22:0/3992791352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c10ec00 0x7fbf2c110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 --2- 172.21.15.22:0/3992791352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074430 0x7fbf2c074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 --2- 172.21.15.22:0/3992791352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c074d50 0x7fbf2c10e6c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 -- 172.21.15.22:0/3992791352 >> 172.21.15.22:0/3992791352 conn(0x7fbf2c079330 msgr2=0x7fbf2c079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 -- 172.21.15.22:0/3992791352 shutdown_connections 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.583+0000 7fbf33a00700 1 -- 172.21.15.22:0/3992791352 wait complete. 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.585+0000 7fbf33a00700 1 Processor -- start 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.585+0000 7fbf33a00700 1 -- start start 2024-03-24T06:38:13.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.585+0000 7fbf33a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c074430 0x7fbf2c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf33a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074d50 0x7fbf2c1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf33a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c10ec00 0x7fbf2c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf33a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c19ae70 con 0x7fbf2c074430 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf33a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c19afe0 con 0x7fbf2c10ec00 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf33a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbf2c19b1a0 con 0x7fbf2c074d50 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf31a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c10ec00 0x7fbf2c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf31a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c10ec00 0x7fbf2c19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58980/0 (socket says 172.21.15.22:58980) 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf31a00700 1 -- 172.21.15.22:0/1223933432 learned_addr learned my addr 172.21.15.22:0/1223933432 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf31000700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c074430 0x7fbf2c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.586+0000 7fbf2be00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074d50 0x7fbf2c1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 -- 172.21.15.22:0/1223933432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074d50 msgr2=0x7fbf2c1a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074d50 0x7fbf2c1a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 -- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c10ec00 msgr2=0x7fbf2c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf2c10ec00 0x7fbf2c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf200095f0 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf2be00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf2c074d50 0x7fbf2c1a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf31000700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c074430 0x7fbf2c1a0640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000eb90 tx=0x7fbf2000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20009970 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf20004500 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20010430 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.587+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf18009520 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.588+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2c1a74a0 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.589+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf2c114790 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.591+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fbf20003680 con 0x7fbf2c074430 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.593+0000 7fbf29600700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbf140a2260 0x7fbf140a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf2be00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbf140a2260 0x7fbf140a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbf20014070 con 0x7fbf2c074430 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf29600700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] conn(0x7fbf140ad3c0 0x7fbf140af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf140aff30 con 0x7fbf140ad3c0 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbf200ce9d0 con 0x7fbf2c074430 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.594+0000 7fbf2be00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbf140a2260 0x7fbf140a4720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009810 tx=0x7fbf18019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.595+0000 7fbf31a00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] conn(0x7fbf140ad3c0 0x7fbf140af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.595+0000 7fbf31a00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] conn(0x7fbf140ad3c0 0x7fbf140af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:13.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.596+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== osd.7 v2:172.21.15.143:6816/1253244511 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf140aff30 con 0x7fbf140ad3c0 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.614+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 --> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf2c062910 con 0x7fbf140ad3c0 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf29600700 1 -- 172.21.15.22:0/1223933432 <== osd.7 v2:172.21.15.143:6816/1253244511 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbf2c062910 con 0x7fbf140ad3c0 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] conn(0x7fbf140ad3c0 msgr2=0x7fbf140af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] conn(0x7fbf140ad3c0 0x7fbf140af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbf140a2260 msgr2=0x7fbf140a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbf140a2260 0x7fbf140a4720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009810 tx=0x7fbf18019040 comp rx=0 tx=0).stop 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c074430 msgr2=0x7fbf2c1a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 --2- 172.21.15.22:0/1223933432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbf2c074430 0x7fbf2c1a0640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000eb90 tx=0x7fbf2000eea0 comp rx=0 tx=0).stop 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf31a00700 1 -- 172.21.15.22:0/1223933432 reap_dead start 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 shutdown_connections 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.615+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 >> 172.21.15.22:0/1223933432 conn(0x7fbf2c079330 msgr2=0x7fbf2c0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.616+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 shutdown_connections 2024-03-24T06:38:13.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:12.616+0000 7fbf33a00700 1 -- 172.21.15.22:0/1223933432 wait complete. 2024-03-24T06:38:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:14 smithi155 ceph-mon[36262]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:14.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:14 smithi022 ceph-mon[32003]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:14 smithi143 ceph-mon[36520]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:14.895 INFO:teuthology.orchestra.run.smithi022.stdout:253403070512 2024-03-24T06:38:14.896 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T06:38:15.785 INFO:teuthology.orchestra.run.smithi022.stdout:219043332147 2024-03-24T06:38:15.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T06:38:16.480 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:16 smithi143 ceph-mon[36520]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:16.526 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:16 smithi155 ceph-mon[36262]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:16.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:16 smithi022 ceph-mon[32003]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:16.723 INFO:teuthology.orchestra.run.smithi022.stdout:343597383709 2024-03-24T06:38:16.724 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T06:38:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:18 smithi155 ceph-mon[36262]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:18.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:18 smithi022 ceph-mon[32003]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:18.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:18 smithi143 ceph-mon[36520]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 -- 172.21.15.22:0/1340638204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 msgr2=0x7f9860074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 --2- 172.21.15.22:0/1340638204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f9860074810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9850009990 tx=0x7f9850009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 -- 172.21.15.22:0/1340638204 shutdown_connections 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 --2- 172.21.15.22:0/1340638204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f986010ec00 0x7f9860110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 --2- 172.21.15.22:0/1340638204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f9860074810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 --2- 172.21.15.22:0/1340638204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9860074d50 0x7f986010e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.439+0000 7f9865600700 1 -- 172.21.15.22:0/1340638204 >> 172.21.15.22:0/1340638204 conn(0x7f9860079330 msgr2=0x7f9860079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.440+0000 7f9865600700 1 -- 172.21.15.22:0/1340638204 shutdown_connections 2024-03-24T06:38:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.440+0000 7f9865600700 1 -- 172.21.15.22:0/1340638204 wait complete. 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.441+0000 7f9865600700 1 Processor -- start 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.441+0000 7f9865600700 1 -- start start 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f98601a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9860074d50 0x7f98601a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f986010ec00 0x7f986019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f986019ae90 con 0x7f9860074d50 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f986019b000 con 0x7f986010ec00 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f9865600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f986019b1c0 con 0x7f9860074430 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985ea00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f98601a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985ea00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f98601a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50392/0 (socket says 172.21.15.22:50392) 2024-03-24T06:38:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985ea00700 1 -- 172.21.15.22:0/3846472704 learned_addr learned my addr 172.21.15.22:0/3846472704 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985e000700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9860074d50 0x7f98601a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985f400700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f986010ec00 0x7f986019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985ea00700 1 -- 172.21.15.22:0/3846472704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f986010ec00 msgr2=0x7f986019a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.442+0000 7f985ea00700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f986010ec00 0x7f986019a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f985ea00700 1 -- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9860074d50 msgr2=0x7f98601a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f985ea00700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9860074d50 0x7f98601a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f985ea00700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98500095f0 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f985ea00700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f98601a0660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9850005770 tx=0x7f98500057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985001d070 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985000b610 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986019b440 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850021e10 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.443+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98601a7690 con 0x7f9860074430 2024-03-24T06:38:18.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.444+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9860114790 con 0x7f9860074430 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.446+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9850021510 con 0x7f9860074430 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f9857400700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f98440a2260 0x7f98440a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f98500cfc60 con 0x7f9860074430 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f985e000700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f98440a2260 0x7f98440a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f9857400700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] conn(0x7f98440ad3c0 0x7f98440af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98440aff30 con 0x7f98440ad3c0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f98500cffd0 con 0x7f9860074430 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.448+0000 7f985f400700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] conn(0x7f98440ad3c0 0x7f98440af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.449+0000 7f985e000700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f98440a2260 0x7f98440a4720 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9860110910 tx=0x7f984c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.449+0000 7f985f400700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] conn(0x7f98440ad3c0 0x7f98440af860 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.450+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== osd.2 v2:172.21.15.155:6800/2561722475 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98440aff30 con 0x7f98440ad3c0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.469+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 --> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98600ffd90 con 0x7f98440ad3c0 2024-03-24T06:38:18.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.469+0000 7f9857400700 1 -- 172.21.15.22:0/3846472704 <== osd.2 v2:172.21.15.155:6800/2561722475 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f98600ffd90 con 0x7f98440ad3c0 2024-03-24T06:38:18.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] conn(0x7f98440ad3c0 msgr2=0x7f98440af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] conn(0x7f98440ad3c0 0x7f98440af860 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f98440a2260 msgr2=0x7f98440a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f98440a2260 0x7f98440a4720 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9860110910 tx=0x7f984c009260 comp rx=0 tx=0).stop 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 msgr2=0x7f98601a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 --2- 172.21.15.22:0/3846472704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9860074430 0x7f98601a0660 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9850005770 tx=0x7f98500057a0 comp rx=0 tx=0).stop 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f985f400700 1 -- 172.21.15.22:0/3846472704 reap_dead start 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 shutdown_connections 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.470+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 >> 172.21.15.22:0/3846472704 conn(0x7f9860079330 msgr2=0x7f98600ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.471+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 shutdown_connections 2024-03-24T06:38:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:17.471+0000 7f9865600700 1 -- 172.21.15.22:0/3846472704 wait complete. 2024-03-24T06:38:19.749 INFO:teuthology.orchestra.run.smithi022.stdout:55834574933 2024-03-24T06:38:19.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T06:38:19.843 INFO:teuthology.orchestra.run.smithi022.stdout:107374182474 2024-03-24T06:38:19.844 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T06:38:19.845 INFO:teuthology.orchestra.run.smithi022.stdout:287762808870 2024-03-24T06:38:19.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T06:38:20.101 INFO:teuthology.orchestra.run.smithi022.stdout:163208757313 2024-03-24T06:38:20.101 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T06:38:20.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:20 smithi022 ceph-mon[32003]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:20 smithi155 ceph-mon[36262]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:20.651 INFO:teuthology.orchestra.run.smithi022.stdout:77309411411 2024-03-24T06:38:20.652 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T06:38:20.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:20 smithi143 ceph-mon[36520]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.089+0000 7ff1b3e00700 1 -- 172.21.15.22:0/2608657313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 msgr2=0x7ff1ac112e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.089+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/2608657313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac112e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00098f0 tx=0x7ff1a0009c00 comp rx=0 tx=0).stop 2024-03-24T06:38:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.091+0000 7ff1b3e00700 1 -- 172.21.15.22:0/2608657313 shutdown_connections 2024-03-24T06:38:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.091+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/2608657313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac074430 0x7ff1ac074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.091+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/2608657313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.091+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/2608657313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac112e90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.091+0000 7ff1b3e00700 1 -- 172.21.15.22:0/2608657313 >> 172.21.15.22:0/2608657313 conn(0x7ff1ac079360 msgr2=0x7ff1ac079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.092+0000 7ff1b3e00700 1 -- 172.21.15.22:0/2608657313 shutdown_connections 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.092+0000 7ff1b3e00700 1 -- 172.21.15.22:0/2608657313 wait complete. 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.094+0000 7ff1b3e00700 1 Processor -- start 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.094+0000 7ff1b3e00700 1 -- start start 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.094+0000 7ff1b3e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac074430 0x7ff1ac10e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.094+0000 7ff1b3e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac10e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b3e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b3e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac107ba0 con 0x7ff1ac074430 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b3e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac107d10 con 0x7ff1ac1133d0 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b3e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac107ed0 con 0x7ff1ac074d50 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b0a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac10e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:20.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b1e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b0a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac10e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50398/0 (socket says 172.21.15.22:50398) 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b1e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51792/0 (socket says 172.21.15.22:51792) 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.095+0000 7ff1b0a00700 1 -- 172.21.15.22:0/1996604507 learned_addr learned my addr 172.21.15.22:0/1996604507 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.096+0000 7ff1b1400700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac074430 0x7ff1ac10e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 -- 172.21.15.22:0/1996604507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 msgr2=0x7ff1ac10e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac074d50 0x7ff1ac10e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 -- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac074430 msgr2=0x7ff1ac10e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac074430 0x7ff1ac10e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a0009520 con 0x7ff1ac1133d0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.097+0000 7ff1b1e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1073d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400efc0 tx=0x7ff1a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.098+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4018070 con 0x7ff1ac1133d0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.098+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a400f040 con 0x7ff1ac1133d0 2024-03-24T06:38:20.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.098+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400cd70 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.099+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac108150 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.099+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac10c060 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.100+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1ac118c40 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.104+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff1a4004020 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.107+0000 7ff1aa000700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a23b0 0x7ff1940a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.108+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff1a40cdc80 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.108+0000 7ff1aa000700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] conn(0x7ff1940ad510 0x7ff1940af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.108+0000 7ff1b1400700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a23b0 0x7ff1940a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.108+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1940b0080 con 0x7ff1940ad510 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.108+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ff1a40d02a0 con 0x7ff1ac1133d0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.109+0000 7ff1b0a00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] conn(0x7ff1940ad510 0x7ff1940af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.109+0000 7ff1b0a00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] conn(0x7ff1940ad510 0x7ff1940af9b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.110+0000 7ff1b1400700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a23b0 0x7ff1940a4870 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff198005950 tx=0x7ff1980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.110+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== osd.10 v2:172.21.15.143:6824/2026852023 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1940b0080 con 0x7ff1940ad510 2024-03-24T06:38:20.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.142+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 --> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1ac112b60 con 0x7ff1940ad510 2024-03-24T06:38:20.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.144+0000 7ff1aa000700 1 -- 172.21.15.22:0/1996604507 <== osd.10 v2:172.21.15.143:6824/2026852023 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff1ac112b60 con 0x7ff1940ad510 2024-03-24T06:38:20.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] conn(0x7ff1940ad510 msgr2=0x7ff1940af9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] conn(0x7ff1940ad510 0x7ff1940af9b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a23b0 msgr2=0x7ff1940a4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a23b0 0x7ff1940a4870 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff198005950 tx=0x7ff1980058e0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 msgr2=0x7ff1ac1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 --2- 172.21.15.22:0/1996604507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac1133d0 0x7ff1ac1073d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400efc0 tx=0x7ff1a400c3a0 comp rx=0 tx=0).stop 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b1e00700 1 -- 172.21.15.22:0/1996604507 reap_dead start 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 shutdown_connections 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.145+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 >> 172.21.15.22:0/1996604507 conn(0x7ff1ac079360 msgr2=0x7ff1ac112200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.146+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 shutdown_connections 2024-03-24T06:38:20.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:20.146+0000 7ff1b3e00700 1 -- 172.21.15.22:0/1996604507 wait complete. 2024-03-24T06:38:21.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 -- 172.21.15.22:0/3141308907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0069eb0 msgr2=0x7fc9d006a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 --2- 172.21.15.22:0/3141308907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0069eb0 0x7fc9d006a310 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0098f0 tx=0x7fc9bc009c00 comp rx=0 tx=0).stop 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 -- 172.21.15.22:0/3141308907 shutdown_connections 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 --2- 172.21.15.22:0/3141308907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9d0108250 0x7fc9d0108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 --2- 172.21.15.22:0/3141308907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9d006a850 0x7fc9d010f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 --2- 172.21.15.22:0/3141308907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0069eb0 0x7fc9d006a310 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.602+0000 7fc9d7600700 1 -- 172.21.15.22:0/3141308907 >> 172.21.15.22:0/3141308907 conn(0x7fc9d0078030 msgr2=0x7fc9d0078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.603+0000 7fc9d7600700 1 -- 172.21.15.22:0/3141308907 shutdown_connections 2024-03-24T06:38:21.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.603+0000 7fc9d7600700 1 -- 172.21.15.22:0/3141308907 wait complete. 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.605+0000 7fc9d7600700 1 Processor -- start 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.605+0000 7fc9d7600700 1 -- start start 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.605+0000 7fc9d7600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9d0069eb0 0x7fc9d01a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.605+0000 7fc9d7600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9d006a850 0x7fc9d01a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.605+0000 7fc9d7600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 0x7fc9d010cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d7600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d010d630 con 0x7fc9d0069eb0 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d7600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d010d7a0 con 0x7fc9d0108250 2024-03-24T06:38:21.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d7600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d010d960 con 0x7fc9d006a850 2024-03-24T06:38:21.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 0x7fc9d010cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:21.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d5600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 0x7fc9d010cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51810/0 (socket says 172.21.15.22:51810) 2024-03-24T06:38:21.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d5600700 1 -- 172.21.15.22:0/974491220 learned_addr learned my addr 172.21.15.22:0/974491220 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9cfe00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9d006a850 0x7fc9d01a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.606+0000 7fc9d4c00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9d0069eb0 0x7fc9d01a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 -- 172.21.15.22:0/974491220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9d006a850 msgr2=0x7fc9d01a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9d006a850 0x7fc9d01a4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 -- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9d0069eb0 msgr2=0x7fc9d01a4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9d0069eb0 0x7fc9d01a4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc009520 con 0x7fc9d0108250 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9d5600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 0x7fc9d010cf90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c110 tx=0x7fc9c800c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.607+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8017070 con 0x7fc9d0108250 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.608+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c800e050 con 0x7fc9d0108250 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.608+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8007a80 con 0x7fc9d0108250 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.608+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d010dbe0 con 0x7fc9d0108250 2024-03-24T06:38:21.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.608+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d01aba70 con 0x7fc9d0108250 2024-03-24T06:38:21.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.610+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9d0112500 con 0x7fc9d0108250 2024-03-24T06:38:21.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.612+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc9c8007be0 con 0x7fc9d0108250 2024-03-24T06:38:21.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.614+0000 7fc9cd600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc9b80a2260 0x7fc9b80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:21.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.615+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc9c80cdc80 con 0x7fc9d0108250 2024-03-24T06:38:21.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.615+0000 7fc9d4c00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc9b80a2260 0x7fc9b80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:21.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.615+0000 7fc9cd600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] conn(0x7fc9b80ad3c0 0x7fc9b80af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:21.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.615+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9b80aff30 con 0x7fc9b80ad3c0 2024-03-24T06:38:21.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.615+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc9c80d22a0 con 0x7fc9d0108250 2024-03-24T06:38:21.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.616+0000 7fc9d4c00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc9b80a2260 0x7fc9b80a4720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009be0 tx=0x7fc9c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:21.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.616+0000 7fc9cfe00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] conn(0x7fc9b80ad3c0 0x7fc9b80af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:21.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.617+0000 7fc9cfe00700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] conn(0x7fc9b80ad3c0 0x7fc9b80af860 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:21.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.618+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== osd.6 v2:172.21.15.22:6818/3272271064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9b80aff30 con 0x7fc9b80ad3c0 2024-03-24T06:38:21.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.642+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 --> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc9d0102520 con 0x7fc9b80ad3c0 2024-03-24T06:38:21.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.643+0000 7fc9cd600700 1 -- 172.21.15.22:0/974491220 <== osd.6 v2:172.21.15.22:6818/3272271064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc9d0102520 con 0x7fc9b80ad3c0 2024-03-24T06:38:21.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.643+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] conn(0x7fc9b80ad3c0 msgr2=0x7fc9b80af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.643+0000 7fc9d7600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] conn(0x7fc9b80ad3c0 0x7fc9b80af860 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:21.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc9b80a2260 msgr2=0x7fc9b80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d7600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc9b80a2260 0x7fc9b80a4720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4009be0 tx=0x7fc9c4009260 comp rx=0 tx=0).stop 2024-03-24T06:38:21.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 msgr2=0x7fc9d010cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:21.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d7600700 1 --2- 172.21.15.22:0/974491220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9d0108250 0x7fc9d010cf90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c110 tx=0x7fc9c800c420 comp rx=0 tx=0).stop 2024-03-24T06:38:21.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d5600700 1 -- 172.21.15.22:0/974491220 reap_dead start 2024-03-24T06:38:21.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.644+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 shutdown_connections 2024-03-24T06:38:21.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.645+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 >> 172.21.15.22:0/974491220 conn(0x7fc9d0078030 msgr2=0x7fc9d0101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:21.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.645+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 shutdown_connections 2024-03-24T06:38:21.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:21.645+0000 7fc9d7600700 1 -- 172.21.15.22:0/974491220 wait complete. 2024-03-24T06:38:21.995 INFO:teuthology.orchestra.run.smithi022.stdout:34359738459 2024-03-24T06:38:21.996 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T06:38:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:22 smithi155 ceph-mon[36262]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:38:22.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:22 smithi022 ceph-mon[32003]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:38:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:22 smithi143 ceph-mon[36520]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:38:23.450 INFO:teuthology.orchestra.run.smithi022.stdout:309237645348 2024-03-24T06:38:23.451 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T06:38:23.485 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.491 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.492 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.628 INFO:teuthology.orchestra.run.smithi022.stdout:197568495676 2024-03-24T06:38:23.628 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T06:38:23.651 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.718 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.735 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.737 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:23.963 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:24.524 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:24 smithi022 ceph-mon[32003]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-24T06:38:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:24 smithi155 ceph-mon[36262]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-24T06:38:24.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:24 smithi143 ceph-mon[36520]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-24T06:38:24.913 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:24.914 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:26.288 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:26 smithi022 ceph-mon[32003]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T06:38:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:26 smithi155 ceph-mon[36262]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T06:38:26.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:26 smithi143 ceph-mon[36520]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T06:38:28.395 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:28 smithi022 ceph-mon[32003]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:28 smithi155 ceph-mon[36262]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:28 smithi143 ceph-mon[36520]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:30.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:30 smithi022 ceph-mon[32003]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-24T06:38:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:30 smithi155 ceph-mon[36262]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-24T06:38:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:30 smithi143 ceph-mon[36520]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-24T06:38:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:32 smithi155 ceph-mon[36262]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:32.536 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:32 smithi022 ceph-mon[32003]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:32 smithi143 ceph-mon[36520]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.985+0000 7f6ccc000700 1 -- 172.21.15.22:0/694844712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 msgr2=0x7f6cc410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.985+0000 7f6ccc000700 1 --2- 172.21.15.22:0/694844712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc410df80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009f90 tx=0x7f6cbc00b330 comp rx=0 tx=0).stop 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 -- 172.21.15.22:0/694844712 shutdown_connections 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 --2- 172.21.15.22:0/694844712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc410df80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 --2- 172.21.15.22:0/694844712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4107140 0x7f6cc4107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 --2- 172.21.15.22:0/694844712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4101100 0x7f6cc4101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 -- 172.21.15.22:0/694844712 >> 172.21.15.22:0/694844712 conn(0x7f6cc4078030 msgr2=0x7f6cc4078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 -- 172.21.15.22:0/694844712 shutdown_connections 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.986+0000 7f6ccc000700 1 -- 172.21.15.22:0/694844712 wait complete. 2024-03-24T06:38:34.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 Processor -- start 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 -- start start 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4107140 0x7f6cc41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc4198e80 con 0x7f6cc4107140 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc4198ff0 con 0x7f6cc4101aa0 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.987+0000 7f6ccc000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41991b0 con 0x7f6cc4101100 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55270/0 (socket says 172.21.15.22:55270) 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 -- 172.21.15.22:0/1472802639 learned_addr learned my addr 172.21.15.22:0/1472802639 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc8c00700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cca000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4107140 0x7f6cc41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 -- 172.21.15.22:0/1472802639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 msgr2=0x7f6cc419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 -- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4107140 msgr2=0x7f6cc41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4107140 0x7f6cc41986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.988+0000 7f6cc9600700 1 -- 172.21.15.22:0/1472802639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb80095f0 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6cc9600700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800ec10 tx=0x7f6cb800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800cd40 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cb8010910 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8018950 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc009c40 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.989+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc41a5580 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.991+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc4109640 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.992+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6cb8010a80 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.993+0000 7f6cc2000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 0x7f6cac0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.994+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6cb8014070 con 0x7f6cc4101100 2024-03-24T06:38:34.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.994+0000 7f6cc8c00700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 0x7f6cac0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.995+0000 7f6cc8c00700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 0x7f6cac0a4720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4199e90 tx=0x7f6cb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:34.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:33.997+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6cb8097040 con 0x7f6cc4101100 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 -- 172.21.15.22:0/3948692579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 msgr2=0x7ff6c010c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 --2- 172.21.15.22:0/3948692579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c010c170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009990 tx=0x7ff6ac009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 -- 172.21.15.22:0/3948692579 shutdown_connections 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 --2- 172.21.15.22:0/3948692579 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 0x7ff6c0103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 --2- 172.21.15.22:0/3948692579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c010eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 --2- 172.21.15.22:0/3948692579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c010c170 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 -- 172.21.15.22:0/3948692579 >> 172.21.15.22:0/3948692579 conn(0x7ff6c00fcad0 msgr2=0x7ff6c00feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 -- 172.21.15.22:0/3948692579 shutdown_connections 2024-03-24T06:38:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.130+0000 7ff6c5400700 1 -- 172.21.15.22:0/3948692579 wait complete. 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.131+0000 7ff6c5400700 1 Processor -- start 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 -- start start 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 0x7ff6c01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c019aea0 con 0x7ff6c0103360 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c019b010 con 0x7ff6c0103c80 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c5400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c019b1d0 con 0x7ff6c010c740 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6bf400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6bfe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 0x7ff6c01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6c4a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.132+0000 7ff6bf400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:56506/0 (socket says 172.21.15.22:56506) 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6bf400700 1 -- 172.21.15.22:0/772211759 learned_addr learned my addr 172.21.15.22:0/772211759 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 -- 172.21.15.22:0/772211759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 msgr2=0x7ff6c01a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 -- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 msgr2=0x7ff6c01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 0x7ff6c01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 -- 172.21.15.22:0/772211759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b8009c40 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6bf400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.133+0000 7ff6c4a00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c019a6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800efb0 tx=0x7ff6b800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.134+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8003a00 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.134+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b800f040 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.134+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80046a0 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.134+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac0095f0 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.134+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c01a75a0 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.135+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c0109cc0 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.137+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff6b8003cf0 con 0x7ff6c010c740 2024-03-24T06:38:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.138+0000 7ff6bcc00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 0x7ff6a40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.138+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff6b80cdcb0 con 0x7ff6c010c740 2024-03-24T06:38:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.139+0000 7ff6bfe00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 0x7ff6a40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.139+0000 7ff6bfe00700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 0x7ff6a40a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009c60 tx=0x7ff6b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.142+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6b80967f0 con 0x7ff6c010c740 2024-03-24T06:38:34.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.234+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6cc4199c00 con 0x7f6cc4101100 2024-03-24T06:38:34.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.234+0000 7f6cc2000700 1 -- 172.21.15.22:0/1472802639 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6cb809a700 con 0x7f6cc4101100 2024-03-24T06:38:34.234 INFO:teuthology.orchestra.run.smithi022.stdout:163208757316 2024-03-24T06:38:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 msgr2=0x7f6cac0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 0x7f6cac0a4720 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4199e90 tx=0x7f6cb0009260 comp rx=0 tx=0).stop 2024-03-24T06:38:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 msgr2=0x7f6cc419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800ec10 tx=0x7f6cb800ef20 comp rx=0 tx=0).stop 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 shutdown_connections 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cc4107140 0x7f6cc41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f6cac0a2260 0x7f6cac0a4720 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc4101100 0x7f6cc419e600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 --2- 172.21.15.22:0/1472802639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4101aa0 0x7f6cc419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.237+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 >> 172.21.15.22:0/1472802639 conn(0x7f6cc4078030 msgr2=0x7f6cc40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.238+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 shutdown_connections 2024-03-24T06:38:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.238+0000 7f6ccc000700 1 -- 172.21.15.22:0/1472802639 wait complete. 2024-03-24T06:38:34.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:34 smithi022 ceph-mon[32003]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-24T06:38:34.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:34 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1472802639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T06:38:34.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.409+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff6c0068f20 con 0x7ff6c010c740 2024-03-24T06:38:34.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.410+0000 7ff6bcc00700 1 -- 172.21.15.22:0/772211759 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff6b8099eb0 con 0x7ff6c010c740 2024-03-24T06:38:34.410 INFO:teuthology.orchestra.run.smithi022.stdout:253403070516 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.413+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 msgr2=0x7ff6a40a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.413+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 0x7ff6a40a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009c60 tx=0x7ff6b0009260 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.413+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 msgr2=0x7ff6c019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.413+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c019a6f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800efb0 tx=0x7ff6b800c3a0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 shutdown_connections 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6c0103360 0x7ff6c01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff6a40a2260 0x7ff6a40a4720 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c010c740 0x7ff6c019a6f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 --2- 172.21.15.22:0/772211759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6c0103c80 0x7ff6c01a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 >> 172.21.15.22:0/772211759 conn(0x7ff6c00fcad0 msgr2=0x7ff6c00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 shutdown_connections 2024-03-24T06:38:34.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:34.414+0000 7ff6c5400700 1 -- 172.21.15.22:0/772211759 wait complete. 2024-03-24T06:38:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:34 smithi155 ceph-mon[36262]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-24T06:38:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:34 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1472802639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T06:38:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:34 smithi143 ceph-mon[36520]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-24T06:38:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:34 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1472802639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T06:38:35.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 -- 172.21.15.22:0/2467602763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc107580 msgr2=0x7fa4fc10f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 --2- 172.21.15.22:0/2467602763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc107580 0x7fa4fc10f150 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400b0a0 tx=0x7fa4f400b3b0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 -- 172.21.15.22:0/2467602763 shutdown_connections 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 --2- 172.21.15.22:0/2467602763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 0x7fa4fc102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 --2- 172.21.15.22:0/2467602763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc107580 0x7fa4fc10f150 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 --2- 172.21.15.22:0/2467602763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc102e60 0x7fa4fc106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.477+0000 7fa502a00700 1 -- 172.21.15.22:0/2467602763 >> 172.21.15.22:0/2467602763 conn(0x7fa4fc079360 msgr2=0x7fa4fc079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.478+0000 7fa502a00700 1 -- 172.21.15.22:0/2467602763 shutdown_connections 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.478+0000 7fa502a00700 1 -- 172.21.15.22:0/2467602763 wait complete. 2024-03-24T06:38:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.479+0000 7fa502a00700 1 Processor -- start 2024-03-24T06:38:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.479+0000 7fa502a00700 1 -- start start 2024-03-24T06:38:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 0x7fa4fc074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc107580 0x7fa4fc074f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1a6d10 con 0x7fa4fc1024b0 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1a6e80 con 0x7fa4fc102e60 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa502a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc1a7040 con 0x7fa4fc107580 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa4fb400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa4fbe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 0x7fa4fc074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa4fb400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:36738/0 (socket says 172.21.15.22:36738) 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa500a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc107580 0x7fa4fc074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.480+0000 7fa4fb400700 1 -- 172.21.15.22:0/3214207421 learned_addr learned my addr 172.21.15.22:0/3214207421 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 -- 172.21.15.22:0/3214207421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc107580 msgr2=0x7fa4fc074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc107580 0x7fa4fc074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 -- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 msgr2=0x7fa4fc074500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 0x7fa4fc074500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 -- 172.21.15.22:0/3214207421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f4009d00 con 0x7fa4fc102e60 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4fb400700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800e970 tx=0x7fa4e800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8009790 con 0x7fa4fc102e60 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4e800c7e0 con 0x7fa4fc102e60 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8005580 con 0x7fa4fc102e60 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4fc1a7320 con 0x7fa4fc102e60 2024-03-24T06:38:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.481+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4fc1a7870 con 0x7fa4fc102e60 2024-03-24T06:38:35.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.483+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4fc10a8a0 con 0x7fa4fc102e60 2024-03-24T06:38:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.486+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa4e800cce0 con 0x7fa4fc102e60 2024-03-24T06:38:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.487+0000 7fa4f8c00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 0x7fa4e00a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.487+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa4e8014070 con 0x7fa4fc102e60 2024-03-24T06:38:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.487+0000 7fa4fbe00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 0x7fa4e00a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:35.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.488+0000 7fa4fbe00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 0x7fa4e00a4720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009c60 tx=0x7fa4ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.495+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa4e8096d90 con 0x7fa4fc102e60 2024-03-24T06:38:35.495 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:35 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/772211759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T06:38:35.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:35 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/772211759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T06:38:35.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:35 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/772211759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T06:38:35.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.828+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa4fc068ec0 con 0x7fa4fc102e60 2024-03-24T06:38:35.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.828+0000 7fa4f8c00700 1 -- 172.21.15.22:0/3214207421 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa4e809a450 con 0x7fa4fc102e60 2024-03-24T06:38:35.828 INFO:teuthology.orchestra.run.smithi022.stdout:34359738463 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 msgr2=0x7fa4e00a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 0x7fa4e00a4720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009c60 tx=0x7fa4ec009260 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 msgr2=0x7fa4fc074a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800e970 tx=0x7fa4e800ec80 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 shutdown_connections 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4fc1024b0 0x7fa4fc074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa4e00a2260 0x7fa4e00a4720 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4fc107580 0x7fa4fc074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 --2- 172.21.15.22:0/3214207421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc102e60 0x7fa4fc074a40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 >> 172.21.15.22:0/3214207421 conn(0x7fa4fc079360 msgr2=0x7fa4fc100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 shutdown_connections 2024-03-24T06:38:35.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:35.831+0000 7fa502a00700 1 -- 172.21.15.22:0/3214207421 wait complete. 2024-03-24T06:38:36.310 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:36 smithi022 ceph-mon[32003]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-24T06:38:36.310 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3214207421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T06:38:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:36 smithi155 ceph-mon[36262]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-24T06:38:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3214207421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T06:38:36.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:36 smithi143 ceph-mon[36520]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-24T06:38:36.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3214207421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 -- 172.21.15.22:0/2240282036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 msgr2=0x7fa2101112e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 --2- 172.21.15.22:0/2240282036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101112e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa208009990 tx=0x7fa208009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 -- 172.21.15.22:0/2240282036 shutdown_connections 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 --2- 172.21.15.22:0/2240282036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101112e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 --2- 172.21.15.22:0/2240282036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 0x7fa21010a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 --2- 172.21.15.22:0/2240282036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 0x7fa210104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 -- 172.21.15.22:0/2240282036 >> 172.21.15.22:0/2240282036 conn(0x7fa2100fff00 msgr2=0x7fa210102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 -- 172.21.15.22:0/2240282036 shutdown_connections 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.734+0000 7fa214e00700 1 -- 172.21.15.22:0/2240282036 wait complete. 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.735+0000 7fa214e00700 1 Processor -- start 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.735+0000 7fa214e00700 1 -- start start 2024-03-24T06:38:38.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 0x7fa2101a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 0x7fa21019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa21019aec0 con 0x7fa210104410 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa21019b030 con 0x7fa210104dd0 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa214e00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa21019b1f0 con 0x7fa21010a4a0 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:36758/0 (socket says 172.21.15.22:36758) 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 -- 172.21.15.22:0/2380087019 learned_addr learned my addr 172.21.15.22:0/2380087019 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa20de00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 0x7fa2101a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa20e800700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 0x7fa21019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 -- 172.21.15.22:0/2380087019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 msgr2=0x7fa21019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 0x7fa21019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 -- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 msgr2=0x7fa2101a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:38.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 0x7fa2101a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.736+0000 7fa205400700 1 -- 172.21.15.22:0/2380087019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2080095f0 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa205400700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e9a0 tx=0x7fa1f800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8009790 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1f8010430 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21019b4d0 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8018680 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.737+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2101a75c0 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.738+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa210105b50 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.740+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fa1f8010910 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.742+0000 7fa207400700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 0x7fa1f40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.742+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa1f8014070 con 0x7fa210104dd0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.742+0000 7fa20de00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 0x7fa1f40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.743+0000 7fa20de00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 0x7fa1f40a46a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa20000f540 tx=0x7fa20000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:38.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.744+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa1f8096e60 con 0x7fa210104dd0 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.977+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa2101144d0 con 0x7fa210104dd0 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.978+0000 7fa207400700 1 -- 172.21.15.22:0/2380087019 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa1f80105a0 con 0x7fa210104dd0 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.980+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 msgr2=0x7fa1f40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.980+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 0x7fa1f40a46a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa20000f540 tx=0x7fa20000b040 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.980+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 msgr2=0x7fa2101a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.980+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e9a0 tx=0x7fa1f800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 shutdown_connections 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa210104410 0x7fa2101a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa1f40a21e0 0x7fa1f40a46a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa21010a4a0 0x7fa21019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 --2- 172.21.15.22:0/2380087019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa210104dd0 0x7fa2101a0b80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 >> 172.21.15.22:0/2380087019 conn(0x7fa2100fff00 msgr2=0x7fa2101003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 shutdown_connections 2024-03-24T06:38:38.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:37.981+0000 7fa214e00700 1 -- 172.21.15.22:0/2380087019 wait complete. 2024-03-24T06:38:38.495 INFO:teuthology.orchestra.run.smithi022.stdout:219043332152 2024-03-24T06:38:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:38 smithi155 ceph-mon[36262]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:38 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2380087019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T06:38:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:38 smithi022 ceph-mon[32003]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:38 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2380087019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T06:38:38.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:38 smithi143 ceph-mon[36520]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:38.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:38 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2380087019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T06:38:39.291 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757313 got 163208757316 for osd.5 2024-03-24T06:38:39.291 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 -- 172.21.15.22:0/2404190808 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d00745d0 msgr2=0x7fc5d00749b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 --2- 172.21.15.22:0/2404190808 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d00745d0 0x7fc5d00749b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009990 tx=0x7fc5c0009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 -- 172.21.15.22:0/2404190808 shutdown_connections 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 --2- 172.21.15.22:0/2404190808 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d00745d0 0x7fc5d00749b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 --2- 172.21.15.22:0/2404190808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d010e980 0x7fc5d0110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 --2- 172.21.15.22:0/2404190808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d0074ef0 0x7fc5d010e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 -- 172.21.15.22:0/2404190808 >> 172.21.15.22:0/2404190808 conn(0x7fc5d00fdbd0 msgr2=0x7fc5d00ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 -- 172.21.15.22:0/2404190808 shutdown_connections 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.035+0000 7fc5d5400700 1 -- 172.21.15.22:0/2404190808 wait complete. 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.036+0000 7fc5d5400700 1 Processor -- start 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 -- start start 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 0x7fc5d01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d010e980 0x7fc5d019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019aeb0 con 0x7fc5d010e980 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019b020 con 0x7fc5d0074ef0 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.037+0000 7fc5d5400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d019b1e0 con 0x7fc5d00745d0 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5cfe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 0x7fc5d01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:36776/0 (socket says 172.21.15.22:36776) 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5cfe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 0x7fc5d01a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42884/0 (socket says 172.21.15.22:42884) 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 -- 172.21.15.22:0/1210204393 learned_addr learned my addr 172.21.15.22:0/1210204393 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 -- 172.21.15.22:0/1210204393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 msgr2=0x7fc5d01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 0x7fc5d01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 -- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d010e980 msgr2=0x7fc5d019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d010e980 0x7fc5d019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.038+0000 7fc5c7e00700 1 -- 172.21.15.22:0/1210204393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c00095f0 con 0x7fc5d0074ef0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5c7e00700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e9a0 tx=0x7fc5b800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8009790 con 0x7fc5d0074ef0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b800cbd0 con 0x7fc5d0074ef0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d019b4c0 con 0x7fc5d0074ef0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8018680 con 0x7fc5d0074ef0 2024-03-24T06:38:40.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.039+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d01a7560 con 0x7fc5d0074ef0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.041+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac005330 con 0x7fc5d0074ef0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.042+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc5b8010430 con 0x7fc5d0074ef0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.044+0000 7fc5cd600700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 0x7fc5b40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.044+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc5b8014070 con 0x7fc5d0074ef0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.044+0000 7fc5cfe00700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 0x7fc5b40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.045+0000 7fc5cfe00700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 0x7fc5b40a46a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006010 tx=0x7fc5c000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:40.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.052+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5b8053280 con 0x7fc5d0074ef0 2024-03-24T06:38:40.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 -- 172.21.15.22:0/4224023578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102e30 msgr2=0x7f2fa4106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/4224023578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102e30 0x7f2fa4106e80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009990 tx=0x7f2f90009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 -- 172.21.15.22:0/4224023578 shutdown_connections 2024-03-24T06:38:40.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/4224023578 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa410f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/4224023578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102480 0x7f2fa4102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/4224023578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102e30 0x7f2fa4106e80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.141+0000 7f2fa9c00700 1 -- 172.21.15.22:0/4224023578 >> 172.21.15.22:0/4224023578 conn(0x7f2fa40fdcd0 msgr2=0x7f2fa41000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.142+0000 7f2fa9c00700 1 -- 172.21.15.22:0/4224023578 shutdown_connections 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.142+0000 7f2fa9c00700 1 -- 172.21.15.22:0/4224023578 wait complete. 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 Processor -- start 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 -- start start 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 0x7f2fa419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 0x7f2fa419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4198c00 con 0x7f2fa4107560 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4198d40 con 0x7f2fa4102480 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.143+0000 7f2fa9c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4198f00 con 0x7f2fa4102e30 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 0x7f2fa419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 0x7f2fa419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:36804/0 (socket says 172.21.15.22:36804) 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2a00700 1 -- 172.21.15.22:0/2845136264 learned_addr learned my addr 172.21.15.22:0/2845136264 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 0x7f2fa419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa3400700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 -- 172.21.15.22:0/2845136264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 msgr2=0x7f2fa419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 0x7f2fa419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 -- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 msgr2=0x7f2fa41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa2000700 1 -- 172.21.15.22:0/2845136264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f900095f0 con 0x7f2fa4102e30 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.144+0000 7f2fa3400700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2fa2000700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 0x7f2fa419e920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098e0 tx=0x7f2f90005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90021070 con 0x7f2fa4102e30 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9000b610 con 0x7f2fa4102e30 2024-03-24T06:38:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa4199180 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90027680 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.145+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa4074600 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.147+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa410a880 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.151+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f2f90025070 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.153+0000 7f2f9b400700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 0x7f2f880a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.154+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2f900d3590 con 0x7f2fa4102e30 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.154+0000 7f2fa2a00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 0x7f2f880a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.154+0000 7f2fa2a00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 0x7f2f880a4720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005950 tx=0x7f2f9400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:40.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.156+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f9009c0d0 con 0x7f2fa4102e30 2024-03-24T06:38:40.346 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:40 smithi022 ceph-mon[32003]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:40.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.412+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc5ac0058b0 con 0x7fc5d0074ef0 2024-03-24T06:38:40.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.413+0000 7fc5cd600700 1 -- 172.21.15.22:0/1210204393 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc5b809a940 con 0x7fc5d0074ef0 2024-03-24T06:38:40.413 INFO:teuthology.orchestra.run.smithi022.stdout:55834574939 2024-03-24T06:38:40.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.416+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 msgr2=0x7fc5b40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.416+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 0x7fc5b40a46a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0006010 tx=0x7fc5c000b260 comp rx=0 tx=0).stop 2024-03-24T06:38:40.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.416+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 msgr2=0x7fc5d01a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.416+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e9a0 tx=0x7fc5b800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 shutdown_connections 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc5d010e980 0x7fc5d019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc5b40a21e0 0x7fc5b40a46a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5d00745d0 0x7fc5d01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 --2- 172.21.15.22:0/1210204393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5d0074ef0 0x7fc5d01a0b20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 >> 172.21.15.22:0/1210204393 conn(0x7fc5d00fdbd0 msgr2=0x7fc5d00fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 shutdown_connections 2024-03-24T06:38:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.417+0000 7fc5d5400700 1 -- 172.21.15.22:0/1210204393 wait complete. 2024-03-24T06:38:40.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.522+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f2fa4199ad0 con 0x7f2fa4102e30 2024-03-24T06:38:40.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.522+0000 7f2f9b400700 1 -- 172.21.15.22:0/2845136264 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2f9009f790 con 0x7f2fa4102e30 2024-03-24T06:38:40.522 INFO:teuthology.orchestra.run.smithi022.stdout:287762808877 2024-03-24T06:38:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 msgr2=0x7f2f880a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 0x7f2f880a4720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f94005950 tx=0x7f2f9400b500 comp rx=0 tx=0).stop 2024-03-24T06:38:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 msgr2=0x7f2fa419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 0x7f2fa419e920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2f900098e0 tx=0x7f2f90005740 comp rx=0 tx=0).stop 2024-03-24T06:38:40.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 shutdown_connections 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fa4107560 0x7f2fa41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f2f880a2260 0x7f2f880a4720 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fa4102480 0x7f2fa419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 --2- 172.21.15.22:0/2845136264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fa4102e30 0x7f2fa419e920 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 >> 172.21.15.22:0/2845136264 conn(0x7f2fa40fdcd0 msgr2=0x7f2fa40fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 shutdown_connections 2024-03-24T06:38:40.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:40.526+0000 7f2fa9c00700 1 -- 172.21.15.22:0/2845136264 wait complete. 2024-03-24T06:38:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:40 smithi155 ceph-mon[36262]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:40 smithi143 ceph-mon[36520]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:41.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:41 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1210204393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T06:38:41.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:41 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2845136264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T06:38:41.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:41 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1210204393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T06:38:41.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:41 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2845136264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T06:38:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:41 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1210204393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T06:38:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:41 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2845136264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T06:38:42.500 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:42 smithi022 ceph-mon[32003]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:42 smithi155 ceph-mon[36262]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:42.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:42 smithi143 ceph-mon[36520]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:42.700 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332147 got 219043332152 for osd.7 2024-03-24T06:38:42.701 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:43.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 -- 172.21.15.22:0/2367255643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060102e60 msgr2=0x7fb060106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 --2- 172.21.15.22:0/2367255643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060102e60 0x7fb060106ea0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009990 tx=0x7fb04c009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 -- 172.21.15.22:0/2367255643 shutdown_connections 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 --2- 172.21.15.22:0/2367255643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060107580 0x7fb06010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 --2- 172.21.15.22:0/2367255643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0601024b0 0x7fb060102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 --2- 172.21.15.22:0/2367255643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060102e60 0x7fb060106ea0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 -- 172.21.15.22:0/2367255643 >> 172.21.15.22:0/2367255643 conn(0x7fb060079360 msgr2=0x7fb060079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 -- 172.21.15.22:0/2367255643 shutdown_connections 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.002+0000 7fb065c00700 1 -- 172.21.15.22:0/2367255643 wait complete. 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.003+0000 7fb065c00700 1 Processor -- start 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 -- start start 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060102e60 0x7fb0601a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060107580 0x7fb06019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb06019af30 con 0x7fb060102e60 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb06019b0a0 con 0x7fb060107580 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.004+0000 7fb065c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb06019b260 con 0x7fb0601024b0 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42920/0 (socket says 172.21.15.22:42920) 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 -- 172.21.15.22:0/673007685 learned_addr learned my addr 172.21.15.22:0/673007685 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05e000700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060102e60 0x7fb0601a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05f400700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060107580 0x7fb06019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 -- 172.21.15.22:0/673007685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060107580 msgr2=0x7fb06019a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060107580 0x7fb06019a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 -- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060102e60 msgr2=0x7fb0601a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060102e60 0x7fb0601a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 -- 172.21.15.22:0/673007685 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb050009520 con 0x7fb0601024b0 2024-03-24T06:38:43.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.005+0000 7fb05ea00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb05000eb30 tx=0x7fb05000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.006+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050009880 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.006+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb050004500 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.006+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050003e70 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.006+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c0095f0 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.006+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06019b730 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.007+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb060103980 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.010+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb05000cbd0 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.012+0000 7fb057400700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 0x7fb0440a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.012+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb050014070 con 0x7fb0601024b0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.012+0000 7fb05e000700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 0x7fb0440a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.013+0000 7fb05e000700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 0x7fb0440a4720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb06010e540 tx=0x7fb04c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:43.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.016+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb050097010 con 0x7fb0601024b0 2024-03-24T06:38:43.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.246+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb060112300 con 0x7fb0601024b0 2024-03-24T06:38:43.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.247+0000 7fb057400700 1 -- 172.21.15.22:0/673007685 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb05009a6d0 con 0x7fb0601024b0 2024-03-24T06:38:43.247 INFO:teuthology.orchestra.run.smithi022.stdout:141733920844 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.249+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 msgr2=0x7fb0440a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.249+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 0x7fb0440a4720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb06010e540 tx=0x7fb04c00b260 comp rx=0 tx=0).stop 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.249+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 msgr2=0x7fb0601a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.249+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb05000eb30 tx=0x7fb05000ee40 comp rx=0 tx=0).stop 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 shutdown_connections 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb0440a2260 0x7fb0440a4720 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb060107580 0x7fb06019a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0601024b0 0x7fb0601a06b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 --2- 172.21.15.22:0/673007685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb060102e60 0x7fb0601a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 >> 172.21.15.22:0/673007685 conn(0x7fb060079360 msgr2=0x7fb06010ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:43.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 shutdown_connections 2024-03-24T06:38:43.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.250+0000 7fb065c00700 1 -- 172.21.15.22:0/673007685 wait complete. 2024-03-24T06:38:43.335 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/673007685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T06:38:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 -- 172.21.15.22:0/3722384307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 msgr2=0x7fc928106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3722384307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc928106d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc914009990 tx=0x7fc914009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 -- 172.21.15.22:0/3722384307 shutdown_connections 2024-03-24T06:38:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3722384307 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 0x7fc928102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3722384307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 0x7fc92810eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3722384307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc928106d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 -- 172.21.15.22:0/3722384307 >> 172.21.15.22:0/3722384307 conn(0x7fc9280fdbd0 msgr2=0x7fc9280ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 -- 172.21.15.22:0/3722384307 shutdown_connections 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.496+0000 7fc92ca00700 1 -- 172.21.15.22:0/3722384307 wait complete. 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 Processor -- start 2024-03-24T06:38:43.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 -- start start 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 0x7fc9281a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 0x7fc92819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc92819ae30 con 0x7fc928102340 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc92819afa0 con 0x7fc928107420 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.498+0000 7fc92ca00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc92819b160 con 0x7fc928102cf0 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42946/0 (socket says 172.21.15.22:42946) 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 -- 172.21.15.22:0/3295926386 learned_addr learned my addr 172.21.15.22:0/3295926386 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc927200700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 0x7fc92819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc926800700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 0x7fc9281a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 -- 172.21.15.22:0/3295926386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 msgr2=0x7fc92819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 0x7fc92819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 -- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 msgr2=0x7fc9281a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 0x7fc9281a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 -- 172.21.15.22:0/3295926386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc918009520 con 0x7fc928102cf0 2024-03-24T06:38:43.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.499+0000 7fc925e00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc914000c00 tx=0x7fc914005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.500+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91401d070 con 0x7fc928102cf0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.500+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc91400b610 con 0x7fc928102cf0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.500+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9140095f0 con 0x7fc928102cf0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.500+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9140173f0 con 0x7fc928102cf0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.500+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9281a7360 con 0x7fc928102cf0 2024-03-24T06:38:43.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.501+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc928103820 con 0x7fc928102cf0 2024-03-24T06:38:43.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.503+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc914022690 con 0x7fc928102cf0 2024-03-24T06:38:43.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.504+0000 7fc91f400700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 0x7fc90c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:43.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.505+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc9140cf360 con 0x7fc928102cf0 2024-03-24T06:38:43.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.505+0000 7fc926800700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 0x7fc90c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:43.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.506+0000 7fc926800700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 0x7fc90c0a4720 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc918009d80 tx=0x7fc918009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:43.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.513+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc914097ea0 con 0x7fc928102cf0 2024-03-24T06:38:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:43 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/673007685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T06:38:43.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.764+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc92819bc10 con 0x7fc928102cf0 2024-03-24T06:38:43.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.764+0000 7fc91f400700 1 -- 172.21.15.22:0/3295926386 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc91409b560 con 0x7fc928102cf0 2024-03-24T06:38:43.764 INFO:teuthology.orchestra.run.smithi022.stdout:77309411416 2024-03-24T06:38:43.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 msgr2=0x7fc90c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 0x7fc90c0a4720 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc918009d80 tx=0x7fc918009260 comp rx=0 tx=0).stop 2024-03-24T06:38:43.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 msgr2=0x7fc9281a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:43.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc914000c00 tx=0x7fc914005740 comp rx=0 tx=0).stop 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 shutdown_connections 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc928102340 0x7fc9281a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc90c0a2260 0x7fc90c0a4720 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc928107420 0x7fc92819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 --2- 172.21.15.22:0/3295926386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc928102cf0 0x7fc9281a0af0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 >> 172.21.15.22:0/3295926386 conn(0x7fc9280fdbd0 msgr2=0x7fc9280feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.767+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 shutdown_connections 2024-03-24T06:38:43.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:43.768+0000 7fc92ca00700 1 -- 172.21.15.22:0/3295926386 wait complete. 2024-03-24T06:38:43.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:43 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/673007685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T06:38:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:44 smithi022 ceph-mon[32003]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:44 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3295926386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T06:38:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:44 smithi143 ceph-mon[36520]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:44 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3295926386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T06:38:44.762 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920844 for osd.4 2024-03-24T06:38:44.762 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:44 smithi155 ceph-mon[36262]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:44.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:44 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3295926386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T06:38:45.039 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070512 got 253403070516 for osd.8 2024-03-24T06:38:45.039 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:45.359 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738463 for osd.0 2024-03-24T06:38:45.360 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:45.578 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574939 for osd.1 2024-03-24T06:38:45.578 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 -- 172.21.15.22:0/1034720600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808102c10 msgr2=0x7fd80810f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 --2- 172.21.15.22:0/1034720600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808102c10 0x7fd80810f0b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd800009990 tx=0x7fd800009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 -- 172.21.15.22:0/1034720600 shutdown_connections 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 --2- 172.21.15.22:0/1034720600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808108300 0x7fd8081086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 --2- 172.21.15.22:0/1034720600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808102c10 0x7fd80810f0b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 --2- 172.21.15.22:0/1034720600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102250 0x7fd8081026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 -- 172.21.15.22:0/1034720600 >> 172.21.15.22:0/1034720600 conn(0x7fd808079330 msgr2=0x7fd808079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 -- 172.21.15.22:0/1034720600 shutdown_connections 2024-03-24T06:38:46.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.115+0000 7fd80fe00700 1 -- 172.21.15.22:0/1034720600 wait complete. 2024-03-24T06:38:46.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.117+0000 7fd80fe00700 1 Processor -- start 2024-03-24T06:38:46.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.117+0000 7fd80fe00700 1 -- start start 2024-03-24T06:38:46.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.117+0000 7fd80fe00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 0x7fd8081a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.117+0000 7fd80fe00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.117+0000 7fd80fe00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 0x7fd80819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80fe00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd80819af10 con 0x7fd808102250 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80fe00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd80819b080 con 0x7fd808102c10 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80fe00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd80819b240 con 0x7fd808108300 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80d400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 0x7fd8081a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49792/0 (socket says 172.21.15.22:49792) 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 -- 172.21.15.22:0/3937684978 learned_addr learned my addr 172.21.15.22:0/3937684978 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80de00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 0x7fd80819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 -- 172.21.15.22:0/3937684978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 msgr2=0x7fd80819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 0x7fd80819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 -- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 msgr2=0x7fd8081a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 0x7fd8081a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80ca00700 1 -- 172.21.15.22:0/3937684978 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8000095f0 con 0x7fd808102c10 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80d400700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 0x7fd8081a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.118+0000 7fd80de00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 0x7fd80819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd80ca00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400e9a0 tx=0x7fd7f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4009790 con 0x7fd808102c10 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f4010430 con 0x7fd808102c10 2024-03-24T06:38:46.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80819b520 con 0x7fd808102c10 2024-03-24T06:38:46.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4018680 con 0x7fd808102c10 2024-03-24T06:38:46.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.119+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8081a7660 con 0x7fd808102c10 2024-03-24T06:38:46.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.120+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd808103940 con 0x7fd808102c10 2024-03-24T06:38:46.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.124+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fd7f4010910 con 0x7fd808102c10 2024-03-24T06:38:46.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.126+0000 7fd806000700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 0x7fd7f00a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.126+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd7f4014070 con 0x7fd808102c10 2024-03-24T06:38:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.126+0000 7fd80d400700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 0x7fd7f00a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.127+0000 7fd80d400700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 0x7fd7f00a4720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009480 tx=0x7fd7fc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:46.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.130+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd7f4053280 con 0x7fd808102c10 2024-03-24T06:38:46.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.385+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd8081122e0 con 0x7fd808102c10 2024-03-24T06:38:46.386 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:46 smithi022 ceph-mon[32003]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:46.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.386+0000 7fd806000700 1 -- 172.21.15.22:0/3937684978 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd7f4096e30 con 0x7fd808102c10 2024-03-24T06:38:46.386 INFO:teuthology.orchestra.run.smithi022.stdout:343597383716 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 msgr2=0x7fd7f00a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 0x7fd7f00a4720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009480 tx=0x7fd7fc00b040 comp rx=0 tx=0).stop 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 msgr2=0x7fd8081a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400e9a0 tx=0x7fd7f400ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 shutdown_connections 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd808102250 0x7fd8081a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd7f00a2260 0x7fd7f00a4720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd808108300 0x7fd80819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 --2- 172.21.15.22:0/3937684978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd808102c10 0x7fd8081a0bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 >> 172.21.15.22:0/3937684978 conn(0x7fd808079330 msgr2=0x7fd808100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:46.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 shutdown_connections 2024-03-24T06:38:46.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.390+0000 7fd80fe00700 1 -- 172.21.15.22:0/3937684978 wait complete. 2024-03-24T06:38:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 -- 172.21.15.22:0/1788994200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 msgr2=0x7fc49c108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/1788994200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c108630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009990 tx=0x7fc48c009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 -- 172.21.15.22:0/1788994200 shutdown_connections 2024-03-24T06:38:46.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/1788994200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c06a850 0x7fc49c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/1788994200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c108630 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/1788994200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c069eb0 0x7fc49c06a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 -- 172.21.15.22:0/1788994200 >> 172.21.15.22:0/1788994200 conn(0x7fc49c078030 msgr2=0x7fc49c078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 -- 172.21.15.22:0/1788994200 shutdown_connections 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.594+0000 7fc4a1a00700 1 -- 172.21.15.22:0/1788994200 wait complete. 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.595+0000 7fc4a1a00700 1 Processor -- start 2024-03-24T06:38:46.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 -- start start 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c069eb0 0x7fc49c1011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1a6af0 con 0x7fc49c069eb0 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1a6c60 con 0x7fc49c108250 2024-03-24T06:38:46.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.596+0000 7fc4a1a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c1a6e20 con 0x7fc49c06a850 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49aa00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c069eb0 0x7fc49c1011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38948/0 (socket says 172.21.15.22:38948) 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 -- 172.21.15.22:0/2829462624 learned_addr learned my addr 172.21.15.22:0/2829462624 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49b400700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 -- 172.21.15.22:0/2829462624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 msgr2=0x7fc49c101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 -- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c069eb0 msgr2=0x7fc49c1011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c069eb0 0x7fc49c1011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 -- 172.21.15.22:0/2829462624 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48c0095f0 con 0x7fc49c06a850 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.597+0000 7fc49a000700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc48800e9a0 tx=0x7fc48800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.598+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488009790 con 0x7fc49c06a850 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.598+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc48800cbd0 con 0x7fc49c06a850 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.598+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49c1a7100 con 0x7fc49c06a850 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.598+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488018680 con 0x7fc49c06a850 2024-03-24T06:38:46.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.598+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49c1a7650 con 0x7fc49c06a850 2024-03-24T06:38:46.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.599+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc49c10a770 con 0x7fc49c06a850 2024-03-24T06:38:46.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.601+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fc48800cd40 con 0x7fc49c06a850 2024-03-24T06:38:46.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.603+0000 7fc493400700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 0x7fc4800a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:46.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.603+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc488014070 con 0x7fc49c06a850 2024-03-24T06:38:46.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.604+0000 7fc49aa00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 0x7fc4800a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:46.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.604+0000 7fc49aa00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 0x7fc4800a4720 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00b2e0 tx=0x7fc48c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:46.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.610+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc488053280 con 0x7fc49c06a850 2024-03-24T06:38:46.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:46 smithi143 ceph-mon[36520]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:46.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:46 smithi155 ceph-mon[36262]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:46.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.858+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc49c068ec0 con 0x7fc49c06a850 2024-03-24T06:38:46.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.858+0000 7fc493400700 1 -- 172.21.15.22:0/2829462624 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc488096ed0 con 0x7fc49c06a850 2024-03-24T06:38:46.858 INFO:teuthology.orchestra.run.smithi022.stdout:107374182480 2024-03-24T06:38:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 msgr2=0x7fc4800a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 0x7fc4800a4720 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00b2e0 tx=0x7fc48c005fd0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 msgr2=0x7fc49c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc48800e9a0 tx=0x7fc48800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 shutdown_connections 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc49c069eb0 0x7fc49c1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc4800a2260 0x7fc4800a4720 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc49c108250 0x7fc49c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 --2- 172.21.15.22:0/2829462624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc49c06a850 0x7fc49c101710 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 >> 172.21.15.22:0/2829462624 conn(0x7fc49c078030 msgr2=0x7fc49c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.861+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 shutdown_connections 2024-03-24T06:38:46.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:46.862+0000 7fc4a1a00700 1 -- 172.21.15.22:0/2829462624 wait complete. 2024-03-24T06:38:47.542 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383709 got 343597383716 for osd.11 2024-03-24T06:38:47.543 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:47.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3937684978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T06:38:47.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2829462624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T06:38:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3937684978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T06:38:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2829462624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T06:38:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3937684978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T06:38:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2829462624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T06:38:48.503 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411411 got 77309411416 for osd.2 2024-03-24T06:38:48.503 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:48.573 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:48 smithi022 ceph-mon[32003]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:48.605 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:48.656 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808877 for osd.9 2024-03-24T06:38:48.657 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:48 smithi143 ceph-mon[36520]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:48.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:48 smithi155 ceph-mon[36262]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:38:49.101 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:49.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182480 for osd.3 2024-03-24T06:38:49.140 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:49.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:49.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:49 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:49 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:50.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:50.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:49 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:50 smithi155 ceph-mon[36262]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:38:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:38:51.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:50 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:51.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:50 smithi022 ceph-mon[32003]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:51.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:38:51.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:38:51.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:50 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:50 smithi143 ceph-mon[36520]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:38:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:38:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:38:51.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:50 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' 2024-03-24T06:38:52.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 -- 172.21.15.22:0/2442680922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 msgr2=0x7fe904102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 --2- 172.21.15.22:0/2442680922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe904102820 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009990 tx=0x7fe8f4009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 -- 172.21.15.22:0/2442680922 shutdown_connections 2024-03-24T06:38:52.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 --2- 172.21.15.22:0/2442680922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe904102820 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 --2- 172.21.15.22:0/2442680922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe90410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 --2- 172.21.15.22:0/2442680922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe904106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.200+0000 7fe909200700 1 -- 172.21.15.22:0/2442680922 >> 172.21.15.22:0/2442680922 conn(0x7fe9040fdcb0 msgr2=0x7fe9041000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.201+0000 7fe909200700 1 -- 172.21.15.22:0/2442680922 shutdown_connections 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.201+0000 7fe909200700 1 -- 172.21.15.22:0/2442680922 wait complete. 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 Processor -- start 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 -- start start 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe90419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe9041985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe904198c20 con 0x7fe904102440 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe904198d60 con 0x7fe904107520 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.202+0000 7fe909200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe904198f20 con 0x7fe904102df0 2024-03-24T06:38:52.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38958/0 (socket says 172.21.15.22:38958) 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 -- 172.21.15.22:0/3813527845 learned_addr learned my addr 172.21.15.22:0/3813527845 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901e00700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe90419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe902800700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe9041985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 -- 172.21.15.22:0/3813527845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 msgr2=0x7fe9041985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe9041985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 -- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 msgr2=0x7fe90419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe90419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe901400700 1 -- 172.21.15.22:0/3813527845 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8fc009c40 con 0x7fe904102df0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.203+0000 7fe902800700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe9041985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe901400700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e970 tx=0x7fe8f000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0009790 con 0x7fe904102df0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f000c7c0 con 0x7fe904102df0 2024-03-24T06:38:52.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f40095f0 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f00055c0 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.204+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9041a52d0 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.206+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90410a7e0 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.210+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe8f000c930 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.211+0000 7fe8faa00700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 0x7fe8e80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.211+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe8f0014070 con 0x7fe904102df0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.211+0000 7fe901e00700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 0x7fe8e80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.212+0000 7fe901e00700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 0x7fe8e80a4720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4006010 tx=0x7fe8f400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:52.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.212+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe8f0096d80 con 0x7fe904102df0 2024-03-24T06:38:52.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.457+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe904068ec0 con 0x7fe904102df0 2024-03-24T06:38:52.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.458+0000 7fe8faa00700 1 -- 172.21.15.22:0/3813527845 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe8f009a440 con 0x7fe904102df0 2024-03-24T06:38:52.458 INFO:teuthology.orchestra.run.smithi022.stdout:309237645354 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 msgr2=0x7fe8e80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 0x7fe8e80a4720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4006010 tx=0x7fe8f400b260 comp rx=0 tx=0).stop 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 msgr2=0x7fe90419e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e970 tx=0x7fe8f000ec80 comp rx=0 tx=0).stop 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 shutdown_connections 2024-03-24T06:38:52.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe904102440 0x7fe90419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe8e80a2260 0x7fe8e80a4720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe904107520 0x7fe9041985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 --2- 172.21.15.22:0/3813527845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe904102df0 0x7fe90419e940 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 >> 172.21.15.22:0/3813527845 conn(0x7fe9040fdcb0 msgr2=0x7fe904100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 shutdown_connections 2024-03-24T06:38:52.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.462+0000 7fe909200700 1 -- 172.21.15.22:0/3813527845 wait complete. 2024-03-24T06:38:52.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 -- 172.21.15.22:0/1849327394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950101aa0 msgr2=0x7fb95010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 --2- 172.21.15.22:0/1849327394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950101aa0 0x7fb95010df80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00b0a0 tx=0x7fb94c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 -- 172.21.15.22:0/1849327394 shutdown_connections 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 --2- 172.21.15.22:0/1849327394 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950107140 0x7fb950107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 --2- 172.21.15.22:0/1849327394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950101aa0 0x7fb95010df80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 --2- 172.21.15.22:0/1849327394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101100 0x7fb950101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.501+0000 7fb958800700 1 -- 172.21.15.22:0/1849327394 >> 172.21.15.22:0/1849327394 conn(0x7fb950078030 msgr2=0x7fb950078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.502+0000 7fb958800700 1 -- 172.21.15.22:0/1849327394 shutdown_connections 2024-03-24T06:38:52.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.502+0000 7fb958800700 1 -- 172.21.15.22:0/1849327394 wait complete. 2024-03-24T06:38:52.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.503+0000 7fb958800700 1 Processor -- start 2024-03-24T06:38:52.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.503+0000 7fb958800700 1 -- start start 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950101100 0x7fb9501a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 0x7fb95019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb95019af10 con 0x7fb950101100 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb95019b050 con 0x7fb950101aa0 2024-03-24T06:38:52.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb958800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb95019b210 con 0x7fb950107140 2024-03-24T06:38:52.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb955400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb955e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950101100 0x7fb9501a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb955400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49864/0 (socket says 172.21.15.22:49864) 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.504+0000 7fb955400700 1 -- 172.21.15.22:0/2262140659 learned_addr learned my addr 172.21.15.22:0/2262140659 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb956800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 0x7fb95019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 -- 172.21.15.22:0/2262140659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 msgr2=0x7fb95019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 0x7fb95019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 -- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950101100 msgr2=0x7fb9501a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950101100 0x7fb9501a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 -- 172.21.15.22:0/2262140659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb940009520 con 0x7fb950101aa0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb956800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 0x7fb95019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb955400700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb94000eb30 tx=0x7fb94000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.505+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940009880 con 0x7fb950101aa0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.506+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb940004500 con 0x7fb950101aa0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.506+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940010430 con 0x7fb950101aa0 2024-03-24T06:38:52.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.506+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94c009d00 con 0x7fb950101aa0 2024-03-24T06:38:52.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.506+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9501a75c0 con 0x7fb950101aa0 2024-03-24T06:38:52.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.507+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb950068ec0 con 0x7fb950101aa0 2024-03-24T06:38:52.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.509+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb94000cc90 con 0x7fb950101aa0 2024-03-24T06:38:52.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.511+0000 7fb94aa00700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 0x7fb9380a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:52.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.511+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb940014070 con 0x7fb950101aa0 2024-03-24T06:38:52.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.512+0000 7fb955e00700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 0x7fb9380a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:52.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.513+0000 7fb955e00700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 0x7fb9380a4720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb95019b9f0 tx=0x7fb944009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:52.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.514+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb940096fe0 con 0x7fb950101aa0 2024-03-24T06:38:52.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.747+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb95006a450 con 0x7fb950101aa0 2024-03-24T06:38:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.747+0000 7fb94aa00700 1 -- 172.21.15.22:0/2262140659 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb94009a6a0 con 0x7fb950101aa0 2024-03-24T06:38:52.747 INFO:teuthology.orchestra.run.smithi022.stdout:197568495682 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 msgr2=0x7fb9380a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 0x7fb9380a4720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb95019b9f0 tx=0x7fb944009260 comp rx=0 tx=0).stop 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 msgr2=0x7fb9501a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb94000eb30 tx=0x7fb94000ee40 comp rx=0 tx=0).stop 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 shutdown_connections 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb950101100 0x7fb9501a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9380a2260 0x7fb9380a4720 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb950107140 0x7fb95019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 --2- 172.21.15.22:0/2262140659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb950101aa0 0x7fb9501a0c30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 >> 172.21.15.22:0/2262140659 conn(0x7fb950078030 msgr2=0x7fb950110bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 shutdown_connections 2024-03-24T06:38:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:52.751+0000 7fb958800700 1 -- 172.21.15.22:0/2262140659 wait complete. 2024-03-24T06:38:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:52 smithi155 ceph-mon[36262]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:53.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:52 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3813527845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T06:38:53.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:52 smithi022 ceph-mon[32003]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:53.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:52 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3813527845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T06:38:53.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:52 smithi143 ceph-mon[36520]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:53.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:52 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3813527845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T06:38:53.523 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645348 got 309237645354 for osd.10 2024-03-24T06:38:53.523 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:53.977 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:53 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2262140659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T06:38:54.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:53 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2262140659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T06:38:54.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495676 got 197568495682 for osd.6 2024-03-24T06:38:54.037 DEBUG:teuthology.parallel:result is None 2024-03-24T06:38:54.037 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T06:38:54.037 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T06:38:54.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:53 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2262140659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T06:38:54.510 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:54.990 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:54 smithi022 ceph-mon[32003]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:55.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:54 smithi155 ceph-mon[36262]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:55.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:54 smithi143 ceph-mon[36520]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 -- 172.21.15.22:0/2671473144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102340 msgr2=0x7f3348102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 --2- 172.21.15.22:0/2671473144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102340 0x7f3348102720 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3338009990 tx=0x7f3338009ca0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 -- 172.21.15.22:0/2671473144 shutdown_connections 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 --2- 172.21.15.22:0/2671473144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334810eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 --2- 172.21.15.22:0/2671473144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102340 0x7f3348102720 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 --2- 172.21.15.22:0/2671473144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102cf0 0x7f3348106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 -- 172.21.15.22:0/2671473144 >> 172.21.15.22:0/2671473144 conn(0x7f33480fdbd0 msgr2=0x7f33480ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.708+0000 7f334d200700 1 -- 172.21.15.22:0/2671473144 shutdown_connections 2024-03-24T06:38:56.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.709+0000 7f334d200700 1 -- 172.21.15.22:0/2671473144 wait complete. 2024-03-24T06:38:56.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 Processor -- start 2024-03-24T06:38:56.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 -- start start 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102340 0x7f33481a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102cf0 0x7f33481a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f334819ae90 con 0x7f3348102340 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f334819b000 con 0x7f3348107420 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.710+0000 7f334d200700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f334819b1c0 con 0x7f3348102cf0 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102340 0x7f33481a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:53918/0 (socket says 172.21.15.22:53918) 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3346a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102cf0 0x7f33481a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:56.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 -- 172.21.15.22:0/898208963 learned_addr learned my addr 172.21.15.22:0/898208963 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 -- 172.21.15.22:0/898208963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102cf0 msgr2=0x7f33481a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102cf0 0x7f33481a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 -- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102340 msgr2=0x7f33481a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102340 0x7f33481a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.711+0000 7f3347e00700 1 -- 172.21.15.22:0/898208963 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3340009c40 con 0x7f3348107420 2024-03-24T06:38:56.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f3347e00700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f334000c370 tx=0x7f334000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340017070 con 0x7f3348107420 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f334000f040 con 0x7f3348107420 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33380095f0 con 0x7f3348107420 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334000e050 con 0x7f3348107420 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.712+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33481a7540 con 0x7f3348107420 2024-03-24T06:38:56.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.713+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3348103820 con 0x7f3348107420 2024-03-24T06:38:56.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.715+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f334000cb60 con 0x7f3348107420 2024-03-24T06:38:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.717+0000 7f333fe00700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 0x7f332c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:38:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.717+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f33400cd900 con 0x7f3348107420 2024-03-24T06:38:56.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.717+0000 7f3347400700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 0x7f332c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:38:56.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.718+0000 7f3347400700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 0x7f332c0a4690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f333800b2e0 tx=0x7f3338005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:38:56.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.721+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3340096510 con 0x7f3348107420 2024-03-24T06:38:56.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:56 smithi022 ceph-mon[32003]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:56.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:56 smithi022 ceph-mon[32003]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-24T06:38:56.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:56 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-24T06:38:56.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.963+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33481121c0 con 0x7f332c0a21d0 2024-03-24T06:38:56.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:56.994+0000 7f333fe00700 1 -- 172.21.15.22:0/898208963 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340545 (secure 0 0 0) 0x7f33481121c0 con 0x7f332c0a21d0 2024-03-24T06:38:57.001 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 msgr2=0x7f332c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 0x7f332c0a4690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f333800b2e0 tx=0x7f3338005fd0 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 msgr2=0x7f334819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f334000c370 tx=0x7f334000c680 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 shutdown_connections 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3348102340 0x7f33481a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f332c0a21d0 0x7f332c0a4690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3348107420 0x7f334819a6e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 --2- 172.21.15.22:0/898208963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3348102cf0 0x7f33481a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 >> 172.21.15.22:0/898208963 conn(0x7f33480fdbd0 msgr2=0x7f33480ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.001+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 shutdown_connections 2024-03-24T06:38:57.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:38:57.002+0000 7f334d200700 1 -- 172.21.15.22:0/898208963 wait complete. 2024-03-24T06:38:57.005 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:38:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:56 smithi155 ceph-mon[36262]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:56 smithi155 ceph-mon[36262]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-24T06:38:57.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:56 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-24T06:38:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:56 smithi143 ceph-mon[36520]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:38:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:56 smithi143 ceph-mon[36520]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-24T06:38:57.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:56 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2024-03-24T06:38:57.846 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-03-24T06:38:56.766444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658246,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":964,"num_read_kb":613,"num_write":424,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181380,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":554,"ondisk_log_size":554,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388384,"kb_used_data":14228,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553440,"statfs":{"total":1151940427776,"available":1151542722560,"internally_reserved":0,"allocated":14569472,"data_stored":7960793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.627207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825507+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:37:56.825507+0000","last_peered":"2024-03-24T06:37:56.825507+0000","last_clean":"2024-03-24T06:37:56.825507+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:37:56.825507+0000","last_undegraded":"2024-03-24T06:37:56.825507+0000","last_fullsized":"2024-03-24T06:37:56.825507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.140324+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:38:55.140324+0000","last_peered":"2024-03-24T06:38:55.140324+0000","last_clean":"2024-03-24T06:38:55.140324+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:38:55.140324+0000","last_undegraded":"2024-03-24T06:38:55.140324+0000","last_fullsized":"2024-03-24T06:38:55.140324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.133754+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:38:55.133754+0000","last_peered":"2024-03-24T06:38:55.133754+0000","last_clean":"2024-03-24T06:38:55.133754+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:38:55.133754+0000","last_undegraded":"2024-03-24T06:38:55.133754+0000","last_fullsized":"2024-03-24T06:38:55.133754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820894+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:37:56.820894+0000","last_peered":"2024-03-24T06:37:56.820894+0000","last_clean":"2024-03-24T06:37:56.820894+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:37:56.820894+0000","last_undegraded":"2024-03-24T06:37:56.820894+0000","last_fullsized":"2024-03-24T06:37:56.820894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.134071+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:38:55.134071+0000","last_peered":"2024-03-24T06:38:55.134071+0000","last_clean":"2024-03-24T06:38:55.134071+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:38:55.134071+0000","last_undegraded":"2024-03-24T06:38:55.134071+0000","last_fullsized":"2024-03-24T06:38:55.134071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054998+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:37:57.054998+0000","last_peered":"2024-03-24T06:37:57.054998+0000","last_clean":"2024-03-24T06:37:57.054998+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:37:57.054998+0000","last_undegraded":"2024-03-24T06:37:57.054998+0000","last_fullsized":"2024-03-24T06:37:57.054998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831055+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:37:56.831055+0000","last_peered":"2024-03-24T06:37:56.831055+0000","last_clean":"2024-03-24T06:37:56.831055+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:37:56.831055+0000","last_undegraded":"2024-03-24T06:37:56.831055+0000","last_fullsized":"2024-03-24T06:37:56.831055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.119631+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:38:55.119631+0000","last_peered":"2024-03-24T06:38:55.119631+0000","last_clean":"2024-03-24T06:38:55.119631+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:38:55.119631+0000","last_undegraded":"2024-03-24T06:38:55.119631+0000","last_fullsized":"2024-03-24T06:38:55.119631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.106832+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:38:55.106832+0000","last_peered":"2024-03-24T06:38:55.106832+0000","last_clean":"2024-03-24T06:38:55.106832+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:38:55.106832+0000","last_undegraded":"2024-03-24T06:38:55.106832+0000","last_fullsized":"2024-03-24T06:38:55.106832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.055615+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:37:57.055615+0000","last_peered":"2024-03-24T06:37:57.055615+0000","last_clean":"2024-03-24T06:37:57.055615+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:37:57.055615+0000","last_undegraded":"2024-03-24T06:37:57.055615+0000","last_fullsized":"2024-03-24T06:37:57.055615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831591+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:37:56.831591+0000","last_peered":"2024-03-24T06:37:56.831591+0000","last_clean":"2024-03-24T06:37:56.831591+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:37:56.831591+0000","last_undegraded":"2024-03-24T06:37:56.831591+0000","last_fullsized":"2024-03-24T06:37:56.831591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820794+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:37:56.820794+0000","last_peered":"2024-03-24T06:37:56.820794+0000","last_clean":"2024-03-24T06:37:56.820794+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:37:56.820794+0000","last_undegraded":"2024-03-24T06:37:56.820794+0000","last_fullsized":"2024-03-24T06:37:56.820794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.828652+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:37:56.828652+0000","last_peered":"2024-03-24T06:37:56.828652+0000","last_clean":"2024-03-24T06:37:56.828652+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:37:56.828652+0000","last_undegraded":"2024-03-24T06:37:56.828652+0000","last_fullsized":"2024-03-24T06:37:56.828652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.132073+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:38:55.132073+0000","last_peered":"2024-03-24T06:38:55.132073+0000","last_clean":"2024-03-24T06:38:55.132073+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:38:55.132073+0000","last_undegraded":"2024-03-24T06:38:55.132073+0000","last_fullsized":"2024-03-24T06:38:55.132073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.120753+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:38:55.120753+0000","last_peered":"2024-03-24T06:38:55.120753+0000","last_clean":"2024-03-24T06:38:55.120753+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:38:55.120753+0000","last_undegraded":"2024-03-24T06:38:55.120753+0000","last_fullsized":"2024-03-24T06:38:55.120753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.138870+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:38:55.138870+0000","last_peered":"2024-03-24T06:38:55.138870+0000","last_clean":"2024-03-24T06:38:55.138870+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:38:55.138870+0000","last_undegraded":"2024-03-24T06:38:55.138870+0000","last_fullsized":"2024-03-24T06:38:55.138870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825414+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:37:56.825414+0000","last_peered":"2024-03-24T06:37:56.825414+0000","last_clean":"2024-03-24T06:37:56.825414+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:37:56.825414+0000","last_undegraded":"2024-03-24T06:37:56.825414+0000","last_fullsized":"2024-03-24T06:37:56.825414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.101324+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:38:55.101324+0000","last_peered":"2024-03-24T06:38:55.101324+0000","last_clean":"2024-03-24T06:38:55.101324+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:38:55.101324+0000","last_undegraded":"2024-03-24T06:38:55.101324+0000","last_fullsized":"2024-03-24T06:38:55.101324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826100+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:37:56.826100+0000","last_peered":"2024-03-24T06:37:56.826100+0000","last_clean":"2024-03-24T06:37:56.826100+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:37:56.826100+0000","last_undegraded":"2024-03-24T06:37:56.826100+0000","last_fullsized":"2024-03-24T06:37:56.826100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826549+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:37:56.826549+0000","last_peered":"2024-03-24T06:37:56.826549+0000","last_clean":"2024-03-24T06:37:56.826549+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:37:56.826549+0000","last_undegraded":"2024-03-24T06:37:56.826549+0000","last_fullsized":"2024-03-24T06:37:56.826549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:54.556281+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:38:54.556281+0000","last_peered":"2024-03-24T06:38:54.556281+0000","last_clean":"2024-03-24T06:38:54.556281+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:38:54.556281+0000","last_undegraded":"2024-03-24T06:38:54.556281+0000","last_fullsized":"2024-03-24T06:38:54.556281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.116237+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:38:55.116237+0000","last_peered":"2024-03-24T06:38:55.116237+0000","last_clean":"2024-03-24T06:38:55.116237+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:38:55.116237+0000","last_undegraded":"2024-03-24T06:38:55.116237+0000","last_fullsized":"2024-03-24T06:38:55.116237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820353+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:37:56.820353+0000","last_peered":"2024-03-24T06:37:56.820353+0000","last_clean":"2024-03-24T06:37:56.820353+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:37:56.820353+0000","last_undegraded":"2024-03-24T06:37:56.820353+0000","last_fullsized":"2024-03-24T06:37:56.820353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.740472+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:37:57.740472+0000","last_peered":"2024-03-24T06:37:57.740472+0000","last_clean":"2024-03-24T06:37:57.740472+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:37:57.740472+0000","last_undegraded":"2024-03-24T06:37:57.740472+0000","last_fullsized":"2024-03-24T06:37:57.740472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.146503+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:38:55.146503+0000","last_peered":"2024-03-24T06:38:55.146503+0000","last_clean":"2024-03-24T06:38:55.146503+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:38:55.146503+0000","last_undegraded":"2024-03-24T06:38:55.146503+0000","last_fullsized":"2024-03-24T06:38:55.146503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820572+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:37:56.820572+0000","last_peered":"2024-03-24T06:37:56.820572+0000","last_clean":"2024-03-24T06:37:56.820572+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:37:56.820572+0000","last_undegraded":"2024-03-24T06:37:56.820572+0000","last_fullsized":"2024-03-24T06:37:56.820572+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824288+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:37:56.824288+0000","last_peered":"2024-03-24T06:37:56.824288+0000","last_clean":"2024-03-24T06:37:56.824288+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:37:56.824288+0000","last_undegraded":"2024-03-24T06:37:56.824288+0000","last_fullsized":"2024-03-24T06:37:56.824288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.140509+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:38:55.140509+0000","last_peered":"2024-03-24T06:38:55.140509+0000","last_clean":"2024-03-24T06:38:55.140509+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:38:55.140509+0000","last_undegraded":"2024-03-24T06:38:55.140509+0000","last_fullsized":"2024-03-24T06:38:55.140509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820473+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:37:56.820473+0000","last_peered":"2024-03-24T06:37:56.820473+0000","last_clean":"2024-03-24T06:37:56.820473+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:37:56.820473+0000","last_undegraded":"2024-03-24T06:37:56.820473+0000","last_fullsized":"2024-03-24T06:37:56.820473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823346+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:37:56.823346+0000","last_peered":"2024-03-24T06:37:56.823346+0000","last_clean":"2024-03-24T06:37:56.823346+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:37:56.823346+0000","last_undegraded":"2024-03-24T06:37:56.823346+0000","last_fullsized":"2024-03-24T06:37:56.823346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831339+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:37:56.831339+0000","last_peered":"2024-03-24T06:37:56.831339+0000","last_clean":"2024-03-24T06:37:56.831339+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:37:56.831339+0000","last_undegraded":"2024-03-24T06:37:56.831339+0000","last_fullsized":"2024-03-24T06:37:56.831339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825919+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:37:56.825919+0000","last_peered":"2024-03-24T06:37:56.825919+0000","last_clean":"2024-03-24T06:37:56.825919+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:37:56.825919+0000","last_undegraded":"2024-03-24T06:37:56.825919+0000","last_fullsized":"2024-03-24T06:37:56.825919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:51.886275+0000","last_change":"2024-03-24T06:37:56.839629+0000","last_active":"2024-03-24T06:38:51.886275+0000","last_peered":"2024-03-24T06:38:51.886275+0000","last_clean":"2024-03-24T06:38:51.886275+0000","last_became_active":"2024-03-24T06:37:56.832648+0000","last_became_peered":"2024-03-24T06:37:56.832648+0000","last_unstale":"2024-03-24T06:38:51.886275+0000","last_undegraded":"2024-03-24T06:38:51.886275+0000","last_fullsized":"2024-03-24T06:38:51.886275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:57.270140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.091467+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:38:55.091467+0000","last_peered":"2024-03-24T06:38:55.091467+0000","last_clean":"2024-03-24T06:38:55.091467+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:38:55.091467+0000","last_undegraded":"2024-03-24T06:38:55.091467+0000","last_fullsized":"2024-03-24T06:38:55.091467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825965+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:37:56.825965+0000","last_peered":"2024-03-24T06:37:56.825965+0000","last_clean":"2024-03-24T06:37:56.825965+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:37:56.825965+0000","last_undegraded":"2024-03-24T06:37:56.825965+0000","last_fullsized":"2024-03-24T06:37:56.825965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054963+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:37:57.054963+0000","last_peered":"2024-03-24T06:37:57.054963+0000","last_clean":"2024-03-24T06:37:57.054963+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:37:57.054963+0000","last_undegraded":"2024-03-24T06:37:57.054963+0000","last_fullsized":"2024-03-24T06:37:57.054963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830943+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:37:56.830943+0000","last_peered":"2024-03-24T06:37:56.830943+0000","last_clean":"2024-03-24T06:37:56.830943+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:37:56.830943+0000","last_undegraded":"2024-03-24T06:37:56.830943+0000","last_fullsized":"2024-03-24T06:37:56.830943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825190+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:37:56.825190+0000","last_peered":"2024-03-24T06:37:56.825190+0000","last_clean":"2024-03-24T06:37:56.825190+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:37:56.825190+0000","last_undegraded":"2024-03-24T06:37:56.825190+0000","last_fullsized":"2024-03-24T06:37:56.825190+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825414+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:37:56.825414+0000","last_peered":"2024-03-24T06:37:56.825414+0000","last_clean":"2024-03-24T06:37:56.825414+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:37:56.825414+0000","last_undegraded":"2024-03-24T06:37:56.825414+0000","last_fullsized":"2024-03-24T06:37:56.825414+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.131038+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:38:55.131038+0000","last_peered":"2024-03-24T06:38:55.131038+0000","last_clean":"2024-03-24T06:38:55.131038+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:38:55.131038+0000","last_undegraded":"2024-03-24T06:38:55.131038+0000","last_fullsized":"2024-03-24T06:38:55.131038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830501+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:37:56.830501+0000","last_peered":"2024-03-24T06:37:56.830501+0000","last_clean":"2024-03-24T06:37:56.830501+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:37:56.830501+0000","last_undegraded":"2024-03-24T06:37:56.830501+0000","last_fullsized":"2024-03-24T06:37:56.830501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830320+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:37:56.830320+0000","last_peered":"2024-03-24T06:37:56.830320+0000","last_clean":"2024-03-24T06:37:56.830320+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:37:56.830320+0000","last_undegraded":"2024-03-24T06:37:56.830320+0000","last_fullsized":"2024-03-24T06:37:56.830320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.102608+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:38:55.102608+0000","last_peered":"2024-03-24T06:38:55.102608+0000","last_clean":"2024-03-24T06:38:55.102608+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:38:55.102608+0000","last_undegraded":"2024-03-24T06:38:55.102608+0000","last_fullsized":"2024-03-24T06:38:55.102608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.738862+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:37:57.738862+0000","last_peered":"2024-03-24T06:37:57.738862+0000","last_clean":"2024-03-24T06:37:57.738862+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:37:57.738862+0000","last_undegraded":"2024-03-24T06:37:57.738862+0000","last_fullsized":"2024-03-24T06:37:57.738862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":190,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:52.519633+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:38:52.519633+0000","last_peered":"2024-03-24T06:38:52.519633+0000","last_clean":"2024-03-24T06:38:52.519633+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:38:52.519633+0000","last_undegraded":"2024-03-24T06:38:52.519633+0000","last_fullsized":"2024-03-24T06:38:52.519633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.105521+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:38:55.105521+0000","last_peered":"2024-03-24T06:38:55.105521+0000","last_clean":"2024-03-24T06:38:55.105521+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:38:55.105521+0000","last_undegraded":"2024-03-24T06:38:55.105521+0000","last_fullsized":"2024-03-24T06:38:55.105521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.834155+0000","last_change":"2024-03-24T06:37:56.831960+0000","last_active":"2024-03-24T06:37:56.834155+0000","last_peered":"2024-03-24T06:37:56.834155+0000","last_clean":"2024-03-24T06:37:56.834155+0000","last_became_active":"2024-03-24T06:37:56.831524+0000","last_became_peered":"2024-03-24T06:37:56.831524+0000","last_unstale":"2024-03-24T06:37:56.834155+0000","last_undegraded":"2024-03-24T06:37:56.834155+0000","last_fullsized":"2024-03-24T06:37:56.834155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:28.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831567+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:37:56.831567+0000","last_peered":"2024-03-24T06:37:56.831567+0000","last_clean":"2024-03-24T06:37:56.831567+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:37:56.831567+0000","last_undegraded":"2024-03-24T06:37:56.831567+0000","last_fullsized":"2024-03-24T06:37:56.831567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.146440+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:38:55.146440+0000","last_peered":"2024-03-24T06:38:55.146440+0000","last_clean":"2024-03-24T06:38:55.146440+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:38:55.146440+0000","last_undegraded":"2024-03-24T06:38:55.146440+0000","last_fullsized":"2024-03-24T06:38:55.146440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.056640+0000","last_change":"2024-03-24T06:37:57.056640+0000","last_active":"2024-03-24T06:37:57.056640+0000","last_peered":"2024-03-24T06:37:57.056640+0000","last_clean":"2024-03-24T06:37:57.056640+0000","last_became_active":"2024-03-24T06:37:57.056087+0000","last_became_peered":"2024-03-24T06:37:57.056087+0000","last_unstale":"2024-03-24T06:37:57.056640+0000","last_undegraded":"2024-03-24T06:37:57.056640+0000","last_fullsized":"2024-03-24T06:37:57.056640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:46.933123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831226+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:37:56.831226+0000","last_peered":"2024-03-24T06:37:56.831226+0000","last_clean":"2024-03-24T06:37:56.831226+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:37:56.831226+0000","last_undegraded":"2024-03-24T06:37:56.831226+0000","last_fullsized":"2024-03-24T06:37:56.831226+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.104391+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:38:55.104391+0000","last_peered":"2024-03-24T06:38:55.104391+0000","last_clean":"2024-03-24T06:38:55.104391+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:38:55.104391+0000","last_undegraded":"2024-03-24T06:38:55.104391+0000","last_fullsized":"2024-03-24T06:38:55.104391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.821501+0000","last_change":"2024-03-24T06:37:01.248655+0000","last_active":"2024-03-24T06:37:56.821501+0000","last_peered":"2024-03-24T06:37:56.821501+0000","last_clean":"2024-03-24T06:37:56.821501+0000","last_became_active":"2024-03-24T06:37:01.245755+0000","last_became_peered":"2024-03-24T06:37:01.245755+0000","last_unstale":"2024-03-24T06:37:56.821501+0000","last_undegraded":"2024-03-24T06:37:56.821501+0000","last_fullsized":"2024-03-24T06:37:56.821501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:29.632351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":224,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:52.886850+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:38:52.886850+0000","last_peered":"2024-03-24T06:38:52.886850+0000","last_clean":"2024-03-24T06:38:52.886850+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:38:52.886850+0000","last_undegraded":"2024-03-24T06:38:52.886850+0000","last_fullsized":"2024-03-24T06:38:52.886850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61222,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":165,"num_read_kb":142,"num_write":35,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.098744+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:38:55.098744+0000","last_peered":"2024-03-24T06:38:55.098744+0000","last_clean":"2024-03-24T06:38:55.098744+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:38:55.098744+0000","last_undegraded":"2024-03-24T06:38:55.098744+0000","last_fullsized":"2024-03-24T06:38:55.098744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831605+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:37:56.831605+0000","last_peered":"2024-03-24T06:37:56.831605+0000","last_clean":"2024-03-24T06:37:56.831605+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:37:56.831605+0000","last_undegraded":"2024-03-24T06:37:56.831605+0000","last_fullsized":"2024-03-24T06:37:56.831605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826594+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:37:56.826594+0000","last_peered":"2024-03-24T06:37:56.826594+0000","last_clean":"2024-03-24T06:37:56.826594+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:37:56.826594+0000","last_undegraded":"2024-03-24T06:37:56.826594+0000","last_fullsized":"2024-03-24T06:37:56.826594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.827224+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:37:56.827224+0000","last_peered":"2024-03-24T06:37:56.827224+0000","last_clean":"2024-03-24T06:37:56.827224+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:37:56.827224+0000","last_undegraded":"2024-03-24T06:37:56.827224+0000","last_fullsized":"2024-03-24T06:37:56.827224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825802+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:37:56.825802+0000","last_peered":"2024-03-24T06:37:56.825802+0000","last_clean":"2024-03-24T06:37:56.825802+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:37:56.825802+0000","last_undegraded":"2024-03-24T06:37:56.825802+0000","last_fullsized":"2024-03-24T06:37:56.825802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830866+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:37:56.830866+0000","last_peered":"2024-03-24T06:37:56.830866+0000","last_clean":"2024-03-24T06:37:56.830866+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:37:56.830866+0000","last_undegraded":"2024-03-24T06:37:56.830866+0000","last_fullsized":"2024-03-24T06:37:56.830866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831359+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:37:56.831359+0000","last_peered":"2024-03-24T06:37:56.831359+0000","last_clean":"2024-03-24T06:37:56.831359+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:37:56.831359+0000","last_undegraded":"2024-03-24T06:37:56.831359+0000","last_fullsized":"2024-03-24T06:37:56.831359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823559+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:37:56.823559+0000","last_peered":"2024-03-24T06:37:56.823559+0000","last_clean":"2024-03-24T06:37:56.823559+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:37:56.823559+0000","last_undegraded":"2024-03-24T06:37:56.823559+0000","last_fullsized":"2024-03-24T06:37:56.823559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:51.588950+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:38:51.588950+0000","last_peered":"2024-03-24T06:38:51.588950+0000","last_clean":"2024-03-24T06:38:51.588950+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:38:51.588950+0000","last_undegraded":"2024-03-24T06:38:51.588950+0000","last_fullsized":"2024-03-24T06:38:51.588950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831325+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:37:56.831325+0000","last_peered":"2024-03-24T06:37:56.831325+0000","last_clean":"2024-03-24T06:37:56.831325+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:37:56.831325+0000","last_undegraded":"2024-03-24T06:37:56.831325+0000","last_fullsized":"2024-03-24T06:37:56.831325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825268+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:37:56.825268+0000","last_peered":"2024-03-24T06:37:56.825268+0000","last_clean":"2024-03-24T06:37:56.825268+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:37:56.825268+0000","last_undegraded":"2024-03-24T06:37:56.825268+0000","last_fullsized":"2024-03-24T06:37:56.825268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824434+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:37:56.824434+0000","last_peered":"2024-03-24T06:37:56.824434+0000","last_clean":"2024-03-24T06:37:56.824434+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:37:56.824434+0000","last_undegraded":"2024-03-24T06:37:56.824434+0000","last_fullsized":"2024-03-24T06:37:56.824434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830357+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:37:56.830357+0000","last_peered":"2024-03-24T06:37:56.830357+0000","last_clean":"2024-03-24T06:37:56.830357+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:37:56.830357+0000","last_undegraded":"2024-03-24T06:37:56.830357+0000","last_fullsized":"2024-03-24T06:37:56.830357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830234+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:37:56.830234+0000","last_peered":"2024-03-24T06:37:56.830234+0000","last_clean":"2024-03-24T06:37:56.830234+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:37:56.830234+0000","last_undegraded":"2024-03-24T06:37:56.830234+0000","last_fullsized":"2024-03-24T06:37:56.830234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.837857+0000","last_change":"2024-03-24T06:37:56.837415+0000","last_active":"2024-03-24T06:37:56.837857+0000","last_peered":"2024-03-24T06:37:56.837857+0000","last_clean":"2024-03-24T06:37:56.837857+0000","last_became_active":"2024-03-24T06:37:56.832240+0000","last_became_peered":"2024-03-24T06:37:56.832240+0000","last_unstale":"2024-03-24T06:37:56.837857+0000","last_undegraded":"2024-03-24T06:37:56.837857+0000","last_fullsized":"2024-03-24T06:37:56.837857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:44.867297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826934+0000","last_change":"2024-03-24T06:37:56.826934+0000","last_active":"2024-03-24T06:37:56.826934+0000","last_peered":"2024-03-24T06:37:56.826934+0000","last_clean":"2024-03-24T06:37:56.826934+0000","last_became_active":"2024-03-24T06:37:56.826574+0000","last_became_peered":"2024-03-24T06:37:56.826574+0000","last_unstale":"2024-03-24T06:37:56.826934+0000","last_undegraded":"2024-03-24T06:37:56.826934+0000","last_fullsized":"2024-03-24T06:37:56.826934+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:52.406936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823017+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:37:56.823017+0000","last_peered":"2024-03-24T06:37:56.823017+0000","last_clean":"2024-03-24T06:37:56.823017+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:37:56.823017+0000","last_undegraded":"2024-03-24T06:37:56.823017+0000","last_fullsized":"2024-03-24T06:37:56.823017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.832165+0000","last_change":"2024-03-24T06:37:56.832165+0000","last_active":"2024-03-24T06:37:56.832165+0000","last_peered":"2024-03-24T06:37:56.832165+0000","last_clean":"2024-03-24T06:37:56.832165+0000","last_became_active":"2024-03-24T06:37:56.831670+0000","last_became_peered":"2024-03-24T06:37:56.831670+0000","last_unstale":"2024-03-24T06:37:56.832165+0000","last_undegraded":"2024-03-24T06:37:56.832165+0000","last_fullsized":"2024-03-24T06:37:56.832165+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.201188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.055247+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:37:57.055247+0000","last_peered":"2024-03-24T06:37:57.055247+0000","last_clean":"2024-03-24T06:37:57.055247+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:37:57.055247+0000","last_undegraded":"2024-03-24T06:37:57.055247+0000","last_fullsized":"2024-03-24T06:37:57.055247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826048+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:37:56.826048+0000","last_peered":"2024-03-24T06:37:56.826048+0000","last_clean":"2024-03-24T06:37:56.826048+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:37:56.826048+0000","last_undegraded":"2024-03-24T06:37:56.826048+0000","last_fullsized":"2024-03-24T06:37:56.826048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:52.553971+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:38:52.553971+0000","last_peered":"2024-03-24T06:38:52.553971+0000","last_clean":"2024-03-24T06:38:52.553971+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:38:52.553971+0000","last_undegraded":"2024-03-24T06:38:52.553971+0000","last_fullsized":"2024-03-24T06:38:52.553971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.055192+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:37:57.055192+0000","last_peered":"2024-03-24T06:37:57.055192+0000","last_clean":"2024-03-24T06:37:57.055192+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:37:57.055192+0000","last_undegraded":"2024-03-24T06:37:57.055192+0000","last_fullsized":"2024-03-24T06:37:57.055192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826377+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:37:56.826377+0000","last_peered":"2024-03-24T06:37:56.826377+0000","last_clean":"2024-03-24T06:37:56.826377+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:37:56.826377+0000","last_undegraded":"2024-03-24T06:37:56.826377+0000","last_fullsized":"2024-03-24T06:37:56.826377+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830489+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:37:56.830489+0000","last_peered":"2024-03-24T06:37:56.830489+0000","last_clean":"2024-03-24T06:37:56.830489+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:37:56.830489+0000","last_undegraded":"2024-03-24T06:37:56.830489+0000","last_fullsized":"2024-03-24T06:37:56.830489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:51.588226+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:38:51.588226+0000","last_peered":"2024-03-24T06:38:51.588226+0000","last_clean":"2024-03-24T06:38:51.588226+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:38:51.588226+0000","last_undegraded":"2024-03-24T06:38:51.588226+0000","last_fullsized":"2024-03-24T06:38:51.588226+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831456+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:37:56.831456+0000","last_peered":"2024-03-24T06:37:56.831456+0000","last_clean":"2024-03-24T06:37:56.831456+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:37:56.831456+0000","last_undegraded":"2024-03-24T06:37:56.831456+0000","last_fullsized":"2024-03-24T06:37:56.831456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.128952+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:38:55.128952+0000","last_peered":"2024-03-24T06:38:55.128952+0000","last_clean":"2024-03-24T06:38:55.128952+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:38:55.128952+0000","last_undegraded":"2024-03-24T06:38:55.128952+0000","last_fullsized":"2024-03-24T06:38:55.128952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820629+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:37:56.820629+0000","last_peered":"2024-03-24T06:37:56.820629+0000","last_clean":"2024-03-24T06:37:56.820629+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:37:56.820629+0000","last_undegraded":"2024-03-24T06:37:56.820629+0000","last_fullsized":"2024-03-24T06:37:56.820629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831027+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:37:56.831027+0000","last_peered":"2024-03-24T06:37:56.831027+0000","last_clean":"2024-03-24T06:37:56.831027+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:37:56.831027+0000","last_undegraded":"2024-03-24T06:37:56.831027+0000","last_fullsized":"2024-03-24T06:37:56.831027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831324+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:37:56.831324+0000","last_peered":"2024-03-24T06:37:56.831324+0000","last_clean":"2024-03-24T06:37:56.831324+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:37:56.831324+0000","last_undegraded":"2024-03-24T06:37:56.831324+0000","last_fullsized":"2024-03-24T06:37:56.831324+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831122+0000","last_change":"2024-03-24T06:37:56.831122+0000","last_active":"2024-03-24T06:37:56.831122+0000","last_peered":"2024-03-24T06:37:56.831122+0000","last_clean":"2024-03-24T06:37:56.831122+0000","last_became_active":"2024-03-24T06:37:56.830642+0000","last_became_peered":"2024-03-24T06:37:56.830642+0000","last_unstale":"2024-03-24T06:37:56.831122+0000","last_undegraded":"2024-03-24T06:37:56.831122+0000","last_fullsized":"2024-03-24T06:37:56.831122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.204542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831022+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:37:56.831022+0000","last_peered":"2024-03-24T06:37:56.831022+0000","last_clean":"2024-03-24T06:37:56.831022+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:37:56.831022+0000","last_undegraded":"2024-03-24T06:37:56.831022+0000","last_fullsized":"2024-03-24T06:37:56.831022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824498+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:37:56.824498+0000","last_peered":"2024-03-24T06:37:56.824498+0000","last_clean":"2024-03-24T06:37:56.824498+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:37:56.824498+0000","last_undegraded":"2024-03-24T06:37:56.824498+0000","last_fullsized":"2024-03-24T06:37:56.824498+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824121+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:37:56.824121+0000","last_peered":"2024-03-24T06:37:56.824121+0000","last_clean":"2024-03-24T06:37:56.824121+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:37:56.824121+0000","last_undegraded":"2024-03-24T06:37:56.824121+0000","last_fullsized":"2024-03-24T06:37:56.824121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826234+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:37:56.826234+0000","last_peered":"2024-03-24T06:37:56.826234+0000","last_clean":"2024-03-24T06:37:56.826234+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:37:56.826234+0000","last_undegraded":"2024-03-24T06:37:56.826234+0000","last_fullsized":"2024-03-24T06:37:56.826234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.738772+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:37:57.738772+0000","last_peered":"2024-03-24T06:37:57.738772+0000","last_clean":"2024-03-24T06:37:57.738772+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:37:57.738772+0000","last_undegraded":"2024-03-24T06:37:57.738772+0000","last_fullsized":"2024-03-24T06:37:57.738772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.822847+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:37:56.822847+0000","last_peered":"2024-03-24T06:37:56.822847+0000","last_clean":"2024-03-24T06:37:56.822847+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:37:56.822847+0000","last_undegraded":"2024-03-24T06:37:56.822847+0000","last_fullsized":"2024-03-24T06:37:56.822847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.130621+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:38:55.130621+0000","last_peered":"2024-03-24T06:38:55.130621+0000","last_clean":"2024-03-24T06:38:55.130621+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:38:55.130621+0000","last_undegraded":"2024-03-24T06:38:55.130621+0000","last_fullsized":"2024-03-24T06:38:55.130621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826427+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:37:56.826427+0000","last_peered":"2024-03-24T06:37:56.826427+0000","last_clean":"2024-03-24T06:37:56.826427+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:37:56.826427+0000","last_undegraded":"2024-03-24T06:37:56.826427+0000","last_fullsized":"2024-03-24T06:37:56.826427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820955+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:37:56.820955+0000","last_peered":"2024-03-24T06:37:56.820955+0000","last_clean":"2024-03-24T06:37:56.820955+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:37:56.820955+0000","last_undegraded":"2024-03-24T06:37:56.820955+0000","last_fullsized":"2024-03-24T06:37:56.820955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.141644+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:38:55.141644+0000","last_peered":"2024-03-24T06:38:55.141644+0000","last_clean":"2024-03-24T06:38:55.141644+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:38:55.141644+0000","last_undegraded":"2024-03-24T06:38:55.141644+0000","last_fullsized":"2024-03-24T06:38:55.141644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.056479+0000","last_change":"2024-03-24T06:37:57.056479+0000","last_active":"2024-03-24T06:37:57.056479+0000","last_peered":"2024-03-24T06:37:57.056479+0000","last_clean":"2024-03-24T06:37:57.056479+0000","last_became_active":"2024-03-24T06:37:57.055993+0000","last_became_peered":"2024-03-24T06:37:57.055993+0000","last_unstale":"2024-03-24T06:37:57.056479+0000","last_undegraded":"2024-03-24T06:37:57.056479+0000","last_fullsized":"2024-03-24T06:37:57.056479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:12.473494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054208+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:37:57.054208+0000","last_peered":"2024-03-24T06:37:57.054208+0000","last_clean":"2024-03-24T06:37:57.054208+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:37:57.054208+0000","last_undegraded":"2024-03-24T06:37:57.054208+0000","last_fullsized":"2024-03-24T06:37:57.054208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.062142+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:38:55.062142+0000","last_peered":"2024-03-24T06:38:55.062142+0000","last_clean":"2024-03-24T06:38:55.062142+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:38:55.062142+0000","last_undegraded":"2024-03-24T06:38:55.062142+0000","last_fullsized":"2024-03-24T06:38:55.062142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823636+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:37:56.823636+0000","last_peered":"2024-03-24T06:37:56.823636+0000","last_clean":"2024-03-24T06:37:56.823636+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:37:56.823636+0000","last_undegraded":"2024-03-24T06:37:56.823636+0000","last_fullsized":"2024-03-24T06:37:56.823636+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.739104+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:37:57.739104+0000","last_peered":"2024-03-24T06:37:57.739104+0000","last_clean":"2024-03-24T06:37:57.739104+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:37:57.739104+0000","last_undegraded":"2024-03-24T06:37:57.739104+0000","last_fullsized":"2024-03-24T06:37:57.739104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824216+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:37:56.824216+0000","last_peered":"2024-03-24T06:37:56.824216+0000","last_clean":"2024-03-24T06:37:56.824216+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:37:56.824216+0000","last_undegraded":"2024-03-24T06:37:56.824216+0000","last_fullsized":"2024-03-24T06:37:56.824216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820629+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:37:56.820629+0000","last_peered":"2024-03-24T06:37:56.820629+0000","last_clean":"2024-03-24T06:37:56.820629+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:37:56.820629+0000","last_undegraded":"2024-03-24T06:37:56.820629+0000","last_fullsized":"2024-03-24T06:37:56.820629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825809+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:37:56.825809+0000","last_peered":"2024-03-24T06:37:56.825809+0000","last_clean":"2024-03-24T06:37:56.825809+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:37:56.825809+0000","last_undegraded":"2024-03-24T06:37:56.825809+0000","last_fullsized":"2024-03-24T06:37:56.825809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.142252+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:38:55.142252+0000","last_peered":"2024-03-24T06:38:55.142252+0000","last_clean":"2024-03-24T06:38:55.142252+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:38:55.142252+0000","last_undegraded":"2024-03-24T06:38:55.142252+0000","last_fullsized":"2024-03-24T06:38:55.142252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.820689+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:37:56.820689+0000","last_peered":"2024-03-24T06:37:56.820689+0000","last_clean":"2024-03-24T06:37:56.820689+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:37:56.820689+0000","last_undegraded":"2024-03-24T06:37:56.820689+0000","last_fullsized":"2024-03-24T06:37:56.820689+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823718+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:37:56.823718+0000","last_peered":"2024-03-24T06:37:56.823718+0000","last_clean":"2024-03-24T06:37:56.823718+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:37:56.823718+0000","last_undegraded":"2024-03-24T06:37:56.823718+0000","last_fullsized":"2024-03-24T06:37:56.823718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831251+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:37:56.831251+0000","last_peered":"2024-03-24T06:37:56.831251+0000","last_clean":"2024-03-24T06:37:56.831251+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:37:56.831251+0000","last_undegraded":"2024-03-24T06:37:56.831251+0000","last_fullsized":"2024-03-24T06:37:56.831251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054403+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:37:57.054403+0000","last_peered":"2024-03-24T06:37:57.054403+0000","last_clean":"2024-03-24T06:37:57.054403+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:37:57.054403+0000","last_undegraded":"2024-03-24T06:37:57.054403+0000","last_fullsized":"2024-03-24T06:37:57.054403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:53.520929+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:38:53.520929+0000","last_peered":"2024-03-24T06:38:53.520929+0000","last_clean":"2024-03-24T06:38:53.520929+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:38:53.520929+0000","last_undegraded":"2024-03-24T06:38:53.520929+0000","last_fullsized":"2024-03-24T06:38:53.520929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831668+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:37:56.831668+0000","last_peered":"2024-03-24T06:37:56.831668+0000","last_clean":"2024-03-24T06:37:56.831668+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:37:56.831668+0000","last_undegraded":"2024-03-24T06:37:56.831668+0000","last_fullsized":"2024-03-24T06:37:56.831668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.829630+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:37:56.829630+0000","last_peered":"2024-03-24T06:37:56.829630+0000","last_clean":"2024-03-24T06:37:56.829630+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:37:56.829630+0000","last_undegraded":"2024-03-24T06:37:56.829630+0000","last_fullsized":"2024-03-24T06:37:56.829630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831439+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:37:56.831439+0000","last_peered":"2024-03-24T06:37:56.831439+0000","last_clean":"2024-03-24T06:37:56.831439+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:37:56.831439+0000","last_undegraded":"2024-03-24T06:37:56.831439+0000","last_fullsized":"2024-03-24T06:37:56.831439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831399+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:37:56.831399+0000","last_peered":"2024-03-24T06:37:56.831399+0000","last_clean":"2024-03-24T06:37:56.831399+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:37:56.831399+0000","last_undegraded":"2024-03-24T06:37:56.831399+0000","last_fullsized":"2024-03-24T06:37:56.831399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.828912+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:37:56.828912+0000","last_peered":"2024-03-24T06:37:56.828912+0000","last_clean":"2024-03-24T06:37:56.828912+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:37:56.828912+0000","last_undegraded":"2024-03-24T06:37:56.828912+0000","last_fullsized":"2024-03-24T06:37:56.828912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825127+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:37:56.825127+0000","last_peered":"2024-03-24T06:37:56.825127+0000","last_clean":"2024-03-24T06:37:56.825127+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:37:56.825127+0000","last_undegraded":"2024-03-24T06:37:56.825127+0000","last_fullsized":"2024-03-24T06:37:56.825127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.831665+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:37:56.831665+0000","last_peered":"2024-03-24T06:37:56.831665+0000","last_clean":"2024-03-24T06:37:56.831665+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:37:56.831665+0000","last_undegraded":"2024-03-24T06:37:56.831665+0000","last_fullsized":"2024-03-24T06:37:56.831665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824321+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:37:56.824321+0000","last_peered":"2024-03-24T06:37:56.824321+0000","last_clean":"2024-03-24T06:37:56.824321+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:37:56.824321+0000","last_undegraded":"2024-03-24T06:37:56.824321+0000","last_fullsized":"2024-03-24T06:37:56.824321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.832171+0000","last_change":"2024-03-24T06:37:56.832171+0000","last_active":"2024-03-24T06:37:56.832171+0000","last_peered":"2024-03-24T06:37:56.832171+0000","last_clean":"2024-03-24T06:37:56.832171+0000","last_became_active":"2024-03-24T06:37:56.831734+0000","last_became_peered":"2024-03-24T06:37:56.831734+0000","last_unstale":"2024-03-24T06:37:56.832171+0000","last_undegraded":"2024-03-24T06:37:56.832171+0000","last_fullsized":"2024-03-24T06:37:56.832171+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:19.788831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825693+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:37:56.825693+0000","last_peered":"2024-03-24T06:37:56.825693+0000","last_clean":"2024-03-24T06:37:56.825693+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:37:56.825693+0000","last_undegraded":"2024-03-24T06:37:56.825693+0000","last_fullsized":"2024-03-24T06:37:56.825693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.145970+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:38:55.145970+0000","last_peered":"2024-03-24T06:38:55.145970+0000","last_clean":"2024-03-24T06:38:55.145970+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:38:55.145970+0000","last_undegraded":"2024-03-24T06:38:55.145970+0000","last_fullsized":"2024-03-24T06:38:55.145970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826353+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:37:56.826353+0000","last_peered":"2024-03-24T06:37:56.826353+0000","last_clean":"2024-03-24T06:37:56.826353+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:37:56.826353+0000","last_undegraded":"2024-03-24T06:37:56.826353+0000","last_fullsized":"2024-03-24T06:37:56.826353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824883+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:37:56.824883+0000","last_peered":"2024-03-24T06:37:56.824883+0000","last_clean":"2024-03-24T06:37:56.824883+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:37:56.824883+0000","last_undegraded":"2024-03-24T06:37:56.824883+0000","last_fullsized":"2024-03-24T06:37:56.824883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826534+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:37:56.826534+0000","last_peered":"2024-03-24T06:37:56.826534+0000","last_clean":"2024-03-24T06:37:56.826534+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:37:56.826534+0000","last_undegraded":"2024-03-24T06:37:56.826534+0000","last_fullsized":"2024-03-24T06:37:56.826534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.112672+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:38:55.112672+0000","last_peered":"2024-03-24T06:38:55.112672+0000","last_clean":"2024-03-24T06:38:55.112672+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:38:55.112672+0000","last_undegraded":"2024-03-24T06:38:55.112672+0000","last_fullsized":"2024-03-24T06:38:55.112672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830373+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:37:56.830373+0000","last_peered":"2024-03-24T06:37:56.830373+0000","last_clean":"2024-03-24T06:37:56.830373+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:37:56.830373+0000","last_undegraded":"2024-03-24T06:37:56.830373+0000","last_fullsized":"2024-03-24T06:37:56.830373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.118651+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:38:55.118651+0000","last_peered":"2024-03-24T06:38:55.118651+0000","last_clean":"2024-03-24T06:38:55.118651+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:38:55.118651+0000","last_undegraded":"2024-03-24T06:38:55.118651+0000","last_fullsized":"2024-03-24T06:38:55.118651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.135232+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:38:55.135232+0000","last_peered":"2024-03-24T06:38:55.135232+0000","last_clean":"2024-03-24T06:38:55.135232+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:38:55.135232+0000","last_undegraded":"2024-03-24T06:38:55.135232+0000","last_fullsized":"2024-03-24T06:38:55.135232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.828468+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:37:56.828468+0000","last_peered":"2024-03-24T06:37:56.828468+0000","last_clean":"2024-03-24T06:37:56.828468+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:37:56.828468+0000","last_undegraded":"2024-03-24T06:37:56.828468+0000","last_fullsized":"2024-03-24T06:37:56.828468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.828731+0000","last_change":"2024-03-24T06:37:56.828731+0000","last_active":"2024-03-24T06:37:56.828731+0000","last_peered":"2024-03-24T06:37:56.828731+0000","last_clean":"2024-03-24T06:37:56.828731+0000","last_became_active":"2024-03-24T06:37:56.828384+0000","last_became_peered":"2024-03-24T06:37:56.828384+0000","last_unstale":"2024-03-24T06:37:56.828731+0000","last_undegraded":"2024-03-24T06:37:56.828731+0000","last_fullsized":"2024-03-24T06:37:56.828731+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:26.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67644,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":828,"num_read_kb":494,"num_write":241,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":501455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1724416,"data_stored":1149471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":501719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1097847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":559368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":558717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":501563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:38:57.852 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T06:38:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:57 smithi155 ceph-mon[36262]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-24T06:38:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:57 smithi155 ceph-mon[36262]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T06:38:58.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:57 smithi155 ceph-mon[36262]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:38:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:57 smithi022 ceph-mon[32003]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-24T06:38:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:57 smithi022 ceph-mon[32003]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T06:38:58.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:57 smithi022 ceph-mon[32003]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:38:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:57 smithi143 ceph-mon[36520]: from='mgr.14152 172.21.15.22:0/254899648' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2024-03-24T06:38:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:57 smithi143 ceph-mon[36520]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T06:38:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:57 smithi143 ceph-mon[36520]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:38:58.369 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:38:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:58 smithi022 ceph-mon[32003]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T06:38:59.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:38:58 smithi022 ceph-mon[32003]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T06:38:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:58 smithi143 ceph-mon[36520]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T06:38:59.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:38:58 smithi143 ceph-mon[36520]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T06:38:59.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:58 smithi155 ceph-mon[36262]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T06:38:59.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:38:58 smithi155 ceph-mon[36262]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 -- 172.21.15.22:0/2647607042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc108300 msgr2=0x7f8acc1086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 --2- 172.21.15.22:0/2647607042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc108300 0x7f8acc1086e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009990 tx=0x7f8ac0009ca0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 -- 172.21.15.22:0/2647607042 shutdown_connections 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 --2- 172.21.15.22:0/2647607042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 0x7f8acc10f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 --2- 172.21.15.22:0/2647607042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc108300 0x7f8acc1086e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 --2- 172.21.15.22:0/2647607042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc102250 0x7f8acc1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 -- 172.21.15.22:0/2647607042 >> 172.21.15.22:0/2647607042 conn(0x7f8acc079330 msgr2=0x7f8acc079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:00.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 -- 172.21.15.22:0/2647607042 shutdown_connections 2024-03-24T06:39:00.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.728+0000 7f8ad4600700 1 -- 172.21.15.22:0/2647607042 wait complete. 2024-03-24T06:39:00.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.729+0000 7f8ad4600700 1 Processor -- start 2024-03-24T06:39:00.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.729+0000 7f8ad4600700 1 -- start start 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 0x7f8acc074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc108300 0x7f8acc074f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc1a6b50 con 0x7f8acc102c10 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc1a6cc0 con 0x7f8acc102250 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad4600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc1a6e80 con 0x7f8acc108300 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad1c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:00.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad1c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:53930/0 (socket says 172.21.15.22:53930) 2024-03-24T06:39:00.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad1c00700 1 -- 172.21.15.22:0/3183895632 learned_addr learned my addr 172.21.15.22:0/3183895632 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:00.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad2600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc108300 0x7f8acc074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:00.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.730+0000 7f8ad1200700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 0x7f8acc074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 -- 172.21.15.22:0/3183895632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc108300 msgr2=0x7f8acc074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc108300 0x7f8acc074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 -- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 msgr2=0x7f8acc074a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 0x7f8acc074a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 -- 172.21.15.22:0/3183895632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac00095f0 con 0x7f8acc102250 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad1c00700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac001d070 con 0x7f8acc102250 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ac000b610 con 0x7f8acc102250 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8acc1a7100 con 0x7f8acc102250 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.731+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00173f0 con 0x7f8acc102250 2024-03-24T06:39:00.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.732+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8acc1a75f0 con 0x7f8acc102250 2024-03-24T06:39:00.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.733+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8acc10a650 con 0x7f8acc102250 2024-03-24T06:39:00.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.735+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f8ac000fb00 con 0x7f8acc102250 2024-03-24T06:39:00.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.737+0000 7f8aca000700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 0x7f8ab40a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:00.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.737+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f8ac00cf3c0 con 0x7f8acc102250 2024-03-24T06:39:00.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.738+0000 7f8ad1200700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 0x7f8ab40a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:00.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.738+0000 7f8ad1200700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 0x7f8ab40a4690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8acc079a70 tx=0x7f8ab8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:00.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.738+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ac00980a0 con 0x7f8acc102250 2024-03-24T06:39:00.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.964+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8acc075aa0 con 0x7f8ab40a21d0 2024-03-24T06:39:00.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.991+0000 7f8aca000700 1 -- 172.21.15.22:0/3183895632 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340671 (secure 0 0 0) 0x7f8acc075aa0 con 0x7f8ab40a21d0 2024-03-24T06:39:00.997 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.997+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 msgr2=0x7f8ab40a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 0x7f8ab40a4690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8acc079a70 tx=0x7f8ab8009260 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 msgr2=0x7f8acc074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0005740 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 shutdown_connections 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ab40a21d0 0x7f8ab40a4690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8acc108300 0x7f8acc074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8acc102250 0x7f8acc074500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 --2- 172.21.15.22:0/3183895632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8acc102c10 0x7f8acc074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 >> 172.21.15.22:0/3183895632 conn(0x7f8acc079330 msgr2=0x7f8acc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.998+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 shutdown_connections 2024-03-24T06:39:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:00.999+0000 7f8ad4600700 1 -- 172.21.15.22:0/3183895632 wait complete. 2024-03-24T06:39:01.001 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:39:01.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:00 smithi022 ceph-mon[32003]: pgmap v385: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-24T06:39:01.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:00 smithi143 ceph-mon[36520]: pgmap v385: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-24T06:39:01.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:00 smithi155 ceph-mon[36262]: pgmap v385: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-03-24T06:39:01.840 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-24T06:38:59.144940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658495,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1005,"num_read_kb":634,"num_write":427,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181380,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":808,"ondisk_log_size":808,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388488,"kb_used_data":14340,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553336,"statfs":{"total":1151940427776,"available":1151542616064,"internally_reserved":0,"allocated":14684160,"data_stored":8026471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":91,"num_read_kb":46,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002945"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773528+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:38:56.773528+0000","last_peered":"2024-03-24T06:38:56.773528+0000","last_clean":"2024-03-24T06:38:56.773528+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:38:56.773528+0000","last_undegraded":"2024-03-24T06:38:56.773528+0000","last_fullsized":"2024-03-24T06:38:56.773528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.140324+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:38:55.140324+0000","last_peered":"2024-03-24T06:38:55.140324+0000","last_clean":"2024-03-24T06:38:55.140324+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:38:55.140324+0000","last_undegraded":"2024-03-24T06:38:55.140324+0000","last_fullsized":"2024-03-24T06:38:55.140324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.133754+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:38:55.133754+0000","last_peered":"2024-03-24T06:38:55.133754+0000","last_clean":"2024-03-24T06:38:55.133754+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:38:55.133754+0000","last_undegraded":"2024-03-24T06:38:55.133754+0000","last_fullsized":"2024-03-24T06:38:55.133754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781667+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:38:57.781667+0000","last_peered":"2024-03-24T06:38:57.781667+0000","last_clean":"2024-03-24T06:38:57.781667+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:38:57.781667+0000","last_undegraded":"2024-03-24T06:38:57.781667+0000","last_fullsized":"2024-03-24T06:38:57.781667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.134071+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:38:55.134071+0000","last_peered":"2024-03-24T06:38:55.134071+0000","last_clean":"2024-03-24T06:38:55.134071+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:38:55.134071+0000","last_undegraded":"2024-03-24T06:38:55.134071+0000","last_fullsized":"2024-03-24T06:38:55.134071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.893720+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:38:57.893720+0000","last_peered":"2024-03-24T06:38:57.893720+0000","last_clean":"2024-03-24T06:38:57.893720+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:38:57.893720+0000","last_undegraded":"2024-03-24T06:38:57.893720+0000","last_fullsized":"2024-03-24T06:38:57.893720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564059+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:38:58.564059+0000","last_peered":"2024-03-24T06:38:58.564059+0000","last_clean":"2024-03-24T06:38:58.564059+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:38:58.564059+0000","last_undegraded":"2024-03-24T06:38:58.564059+0000","last_fullsized":"2024-03-24T06:38:58.564059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.119631+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:38:55.119631+0000","last_peered":"2024-03-24T06:38:55.119631+0000","last_clean":"2024-03-24T06:38:55.119631+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:38:55.119631+0000","last_undegraded":"2024-03-24T06:38:55.119631+0000","last_fullsized":"2024-03-24T06:38:55.119631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.106832+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:38:55.106832+0000","last_peered":"2024-03-24T06:38:55.106832+0000","last_clean":"2024-03-24T06:38:55.106832+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:38:55.106832+0000","last_undegraded":"2024-03-24T06:38:55.106832+0000","last_fullsized":"2024-03-24T06:38:55.106832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.894297+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:38:57.894297+0000","last_peered":"2024-03-24T06:38:57.894297+0000","last_clean":"2024-03-24T06:38:57.894297+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:38:57.894297+0000","last_undegraded":"2024-03-24T06:38:57.894297+0000","last_fullsized":"2024-03-24T06:38:57.894297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564464+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:38:58.564464+0000","last_peered":"2024-03-24T06:38:58.564464+0000","last_clean":"2024-03-24T06:38:58.564464+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:38:58.564464+0000","last_undegraded":"2024-03-24T06:38:58.564464+0000","last_fullsized":"2024-03-24T06:38:58.564464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781498+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:38:57.781498+0000","last_peered":"2024-03-24T06:38:57.781498+0000","last_clean":"2024-03-24T06:38:57.781498+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:38:57.781498+0000","last_undegraded":"2024-03-24T06:38:57.781498+0000","last_fullsized":"2024-03-24T06:38:57.781498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817989+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:38:56.817989+0000","last_peered":"2024-03-24T06:38:56.817989+0000","last_clean":"2024-03-24T06:38:56.817989+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:38:56.817989+0000","last_undegraded":"2024-03-24T06:38:56.817989+0000","last_fullsized":"2024-03-24T06:38:56.817989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.132073+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:38:55.132073+0000","last_peered":"2024-03-24T06:38:55.132073+0000","last_clean":"2024-03-24T06:38:55.132073+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:38:55.132073+0000","last_undegraded":"2024-03-24T06:38:55.132073+0000","last_fullsized":"2024-03-24T06:38:55.132073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.120753+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:38:55.120753+0000","last_peered":"2024-03-24T06:38:55.120753+0000","last_clean":"2024-03-24T06:38:55.120753+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:38:55.120753+0000","last_undegraded":"2024-03-24T06:38:55.120753+0000","last_fullsized":"2024-03-24T06:38:55.120753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.138870+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:38:55.138870+0000","last_peered":"2024-03-24T06:38:55.138870+0000","last_clean":"2024-03-24T06:38:55.138870+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:38:55.138870+0000","last_undegraded":"2024-03-24T06:38:55.138870+0000","last_fullsized":"2024-03-24T06:38:55.138870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.774228+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:38:56.774228+0000","last_peered":"2024-03-24T06:38:56.774228+0000","last_clean":"2024-03-24T06:38:56.774228+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:38:56.774228+0000","last_undegraded":"2024-03-24T06:38:56.774228+0000","last_fullsized":"2024-03-24T06:38:56.774228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.101324+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:38:55.101324+0000","last_peered":"2024-03-24T06:38:55.101324+0000","last_clean":"2024-03-24T06:38:55.101324+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:38:55.101324+0000","last_undegraded":"2024-03-24T06:38:55.101324+0000","last_fullsized":"2024-03-24T06:38:55.101324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826100+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:37:56.826100+0000","last_peered":"2024-03-24T06:37:56.826100+0000","last_clean":"2024-03-24T06:37:56.826100+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:37:56.826100+0000","last_undegraded":"2024-03-24T06:37:56.826100+0000","last_fullsized":"2024-03-24T06:37:56.826100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826549+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:37:56.826549+0000","last_peered":"2024-03-24T06:37:56.826549+0000","last_clean":"2024-03-24T06:37:56.826549+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:37:56.826549+0000","last_undegraded":"2024-03-24T06:37:56.826549+0000","last_fullsized":"2024-03-24T06:37:56.826549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:54.556281+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:38:54.556281+0000","last_peered":"2024-03-24T06:38:54.556281+0000","last_clean":"2024-03-24T06:38:54.556281+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:38:54.556281+0000","last_undegraded":"2024-03-24T06:38:54.556281+0000","last_fullsized":"2024-03-24T06:38:54.556281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.116237+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:38:55.116237+0000","last_peered":"2024-03-24T06:38:55.116237+0000","last_clean":"2024-03-24T06:38:55.116237+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:38:55.116237+0000","last_undegraded":"2024-03-24T06:38:55.116237+0000","last_fullsized":"2024-03-24T06:38:55.116237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781144+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:38:57.781144+0000","last_peered":"2024-03-24T06:38:57.781144+0000","last_clean":"2024-03-24T06:38:57.781144+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:38:57.781144+0000","last_undegraded":"2024-03-24T06:38:57.781144+0000","last_fullsized":"2024-03-24T06:38:57.781144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.740472+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:37:57.740472+0000","last_peered":"2024-03-24T06:37:57.740472+0000","last_clean":"2024-03-24T06:37:57.740472+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:37:57.740472+0000","last_undegraded":"2024-03-24T06:37:57.740472+0000","last_fullsized":"2024-03-24T06:37:57.740472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.146503+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:38:55.146503+0000","last_peered":"2024-03-24T06:38:55.146503+0000","last_clean":"2024-03-24T06:38:55.146503+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:38:55.146503+0000","last_undegraded":"2024-03-24T06:38:55.146503+0000","last_fullsized":"2024-03-24T06:38:55.146503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781316+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:38:57.781316+0000","last_peered":"2024-03-24T06:38:57.781316+0000","last_clean":"2024-03-24T06:38:57.781316+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:38:57.781316+0000","last_undegraded":"2024-03-24T06:38:57.781316+0000","last_fullsized":"2024-03-24T06:38:57.781316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824288+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:37:56.824288+0000","last_peered":"2024-03-24T06:37:56.824288+0000","last_clean":"2024-03-24T06:37:56.824288+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:37:56.824288+0000","last_undegraded":"2024-03-24T06:37:56.824288+0000","last_fullsized":"2024-03-24T06:37:56.824288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.140509+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:38:55.140509+0000","last_peered":"2024-03-24T06:38:55.140509+0000","last_clean":"2024-03-24T06:38:55.140509+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:38:55.140509+0000","last_undegraded":"2024-03-24T06:38:55.140509+0000","last_fullsized":"2024-03-24T06:38:55.140509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781163+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:38:57.781163+0000","last_peered":"2024-03-24T06:38:57.781163+0000","last_clean":"2024-03-24T06:38:57.781163+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:38:57.781163+0000","last_undegraded":"2024-03-24T06:38:57.781163+0000","last_fullsized":"2024-03-24T06:38:57.781163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823346+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:37:56.823346+0000","last_peered":"2024-03-24T06:37:56.823346+0000","last_clean":"2024-03-24T06:37:56.823346+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:37:56.823346+0000","last_undegraded":"2024-03-24T06:37:56.823346+0000","last_fullsized":"2024-03-24T06:37:56.823346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.782127+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:38:57.782127+0000","last_peered":"2024-03-24T06:38:57.782127+0000","last_clean":"2024-03-24T06:38:57.782127+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:38:57.782127+0000","last_undegraded":"2024-03-24T06:38:57.782127+0000","last_fullsized":"2024-03-24T06:38:57.782127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825919+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:37:56.825919+0000","last_peered":"2024-03-24T06:37:56.825919+0000","last_clean":"2024-03-24T06:37:56.825919+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:37:56.825919+0000","last_undegraded":"2024-03-24T06:37:56.825919+0000","last_fullsized":"2024-03-24T06:37:56.825919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781883+0000","last_change":"2024-03-24T06:37:56.839629+0000","last_active":"2024-03-24T06:38:57.781883+0000","last_peered":"2024-03-24T06:38:57.781883+0000","last_clean":"2024-03-24T06:38:57.781883+0000","last_became_active":"2024-03-24T06:37:56.832648+0000","last_became_peered":"2024-03-24T06:37:56.832648+0000","last_unstale":"2024-03-24T06:38:57.781883+0000","last_undegraded":"2024-03-24T06:38:57.781883+0000","last_fullsized":"2024-03-24T06:38:57.781883+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:57.270140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.091467+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:38:55.091467+0000","last_peered":"2024-03-24T06:38:55.091467+0000","last_clean":"2024-03-24T06:38:55.091467+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:38:55.091467+0000","last_undegraded":"2024-03-24T06:38:55.091467+0000","last_fullsized":"2024-03-24T06:38:55.091467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825965+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:37:56.825965+0000","last_peered":"2024-03-24T06:37:56.825965+0000","last_clean":"2024-03-24T06:37:56.825965+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:37:56.825965+0000","last_undegraded":"2024-03-24T06:37:56.825965+0000","last_fullsized":"2024-03-24T06:37:56.825965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.893681+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:38:57.893681+0000","last_peered":"2024-03-24T06:38:57.893681+0000","last_clean":"2024-03-24T06:38:57.893681+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:38:57.893681+0000","last_undegraded":"2024-03-24T06:38:57.893681+0000","last_fullsized":"2024-03-24T06:38:57.893681+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781898+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:38:57.781898+0000","last_peered":"2024-03-24T06:38:57.781898+0000","last_clean":"2024-03-24T06:38:57.781898+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:38:57.781898+0000","last_undegraded":"2024-03-24T06:38:57.781898+0000","last_fullsized":"2024-03-24T06:38:57.781898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.774027+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:38:56.774027+0000","last_peered":"2024-03-24T06:38:56.774027+0000","last_clean":"2024-03-24T06:38:56.774027+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:38:56.774027+0000","last_undegraded":"2024-03-24T06:38:56.774027+0000","last_fullsized":"2024-03-24T06:38:56.774027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817481+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:38:56.817481+0000","last_peered":"2024-03-24T06:38:56.817481+0000","last_clean":"2024-03-24T06:38:56.817481+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:38:56.817481+0000","last_undegraded":"2024-03-24T06:38:56.817481+0000","last_fullsized":"2024-03-24T06:38:56.817481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.131038+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:38:55.131038+0000","last_peered":"2024-03-24T06:38:55.131038+0000","last_clean":"2024-03-24T06:38:55.131038+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:38:55.131038+0000","last_undegraded":"2024-03-24T06:38:55.131038+0000","last_fullsized":"2024-03-24T06:38:55.131038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.830501+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:37:56.830501+0000","last_peered":"2024-03-24T06:37:56.830501+0000","last_clean":"2024-03-24T06:37:56.830501+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:37:56.830501+0000","last_undegraded":"2024-03-24T06:37:56.830501+0000","last_fullsized":"2024-03-24T06:37:56.830501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781494+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:38:57.781494+0000","last_peered":"2024-03-24T06:38:57.781494+0000","last_clean":"2024-03-24T06:38:57.781494+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:38:57.781494+0000","last_undegraded":"2024-03-24T06:38:57.781494+0000","last_fullsized":"2024-03-24T06:38:57.781494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.102608+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:38:55.102608+0000","last_peered":"2024-03-24T06:38:55.102608+0000","last_clean":"2024-03-24T06:38:55.102608+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:38:55.102608+0000","last_undegraded":"2024-03-24T06:38:55.102608+0000","last_fullsized":"2024-03-24T06:38:55.102608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.738862+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:37:57.738862+0000","last_peered":"2024-03-24T06:37:57.738862+0000","last_clean":"2024-03-24T06:37:57.738862+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:37:57.738862+0000","last_undegraded":"2024-03-24T06:37:57.738862+0000","last_fullsized":"2024-03-24T06:37:57.738862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":204,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.894072+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:38:57.894072+0000","last_peered":"2024-03-24T06:38:57.894072+0000","last_clean":"2024-03-24T06:38:57.894072+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:38:57.894072+0000","last_undegraded":"2024-03-24T06:38:57.894072+0000","last_fullsized":"2024-03-24T06:38:57.894072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":170,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.105521+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:38:55.105521+0000","last_peered":"2024-03-24T06:38:55.105521+0000","last_clean":"2024-03-24T06:38:55.105521+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:38:55.105521+0000","last_undegraded":"2024-03-24T06:38:55.105521+0000","last_fullsized":"2024-03-24T06:38:55.105521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.834155+0000","last_change":"2024-03-24T06:37:56.831960+0000","last_active":"2024-03-24T06:37:56.834155+0000","last_peered":"2024-03-24T06:37:56.834155+0000","last_clean":"2024-03-24T06:37:56.834155+0000","last_became_active":"2024-03-24T06:37:56.831524+0000","last_became_peered":"2024-03-24T06:37:56.831524+0000","last_unstale":"2024-03-24T06:37:56.834155+0000","last_undegraded":"2024-03-24T06:37:56.834155+0000","last_fullsized":"2024-03-24T06:37:56.834155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:28.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564475+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:38:58.564475+0000","last_peered":"2024-03-24T06:38:58.564475+0000","last_clean":"2024-03-24T06:38:58.564475+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:38:58.564475+0000","last_undegraded":"2024-03-24T06:38:58.564475+0000","last_fullsized":"2024-03-24T06:38:58.564475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.146440+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:38:55.146440+0000","last_peered":"2024-03-24T06:38:55.146440+0000","last_clean":"2024-03-24T06:38:55.146440+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:38:55.146440+0000","last_undegraded":"2024-03-24T06:38:55.146440+0000","last_fullsized":"2024-03-24T06:38:55.146440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781422+0000","last_change":"2024-03-24T06:37:57.056640+0000","last_active":"2024-03-24T06:38:57.781422+0000","last_peered":"2024-03-24T06:38:57.781422+0000","last_clean":"2024-03-24T06:38:57.781422+0000","last_became_active":"2024-03-24T06:37:57.056087+0000","last_became_peered":"2024-03-24T06:37:57.056087+0000","last_unstale":"2024-03-24T06:38:57.781422+0000","last_undegraded":"2024-03-24T06:38:57.781422+0000","last_fullsized":"2024-03-24T06:38:57.781422+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:46.933123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.782069+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:38:57.782069+0000","last_peered":"2024-03-24T06:38:57.782069+0000","last_clean":"2024-03-24T06:38:57.782069+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:38:57.782069+0000","last_undegraded":"2024-03-24T06:38:57.782069+0000","last_fullsized":"2024-03-24T06:38:57.782069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.104391+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:38:55.104391+0000","last_peered":"2024-03-24T06:38:55.104391+0000","last_clean":"2024-03-24T06:38:55.104391+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:38:55.104391+0000","last_undegraded":"2024-03-24T06:38:55.104391+0000","last_fullsized":"2024-03-24T06:38:55.104391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.821501+0000","last_change":"2024-03-24T06:37:01.248655+0000","last_active":"2024-03-24T06:37:56.821501+0000","last_peered":"2024-03-24T06:37:56.821501+0000","last_clean":"2024-03-24T06:37:56.821501+0000","last_became_active":"2024-03-24T06:37:01.245755+0000","last_became_peered":"2024-03-24T06:37:01.245755+0000","last_unstale":"2024-03-24T06:37:56.821501+0000","last_undegraded":"2024-03-24T06:37:56.821501+0000","last_fullsized":"2024-03-24T06:37:56.821501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:29.632351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":242,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.894077+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:38:57.894077+0000","last_peered":"2024-03-24T06:38:57.894077+0000","last_clean":"2024-03-24T06:38:57.894077+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:38:57.894077+0000","last_undegraded":"2024-03-24T06:38:57.894077+0000","last_fullsized":"2024-03-24T06:38:57.894077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":176,"num_read_kb":148,"num_write":38,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.098744+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:38:55.098744+0000","last_peered":"2024-03-24T06:38:55.098744+0000","last_clean":"2024-03-24T06:38:55.098744+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:38:55.098744+0000","last_undegraded":"2024-03-24T06:38:55.098744+0000","last_fullsized":"2024-03-24T06:38:55.098744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.782310+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:38:57.782310+0000","last_peered":"2024-03-24T06:38:57.782310+0000","last_clean":"2024-03-24T06:38:57.782310+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:38:57.782310+0000","last_undegraded":"2024-03-24T06:38:57.782310+0000","last_fullsized":"2024-03-24T06:38:57.782310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826594+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:37:56.826594+0000","last_peered":"2024-03-24T06:37:56.826594+0000","last_clean":"2024-03-24T06:37:56.826594+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:37:56.826594+0000","last_undegraded":"2024-03-24T06:37:56.826594+0000","last_fullsized":"2024-03-24T06:37:56.826594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.818544+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:38:56.818544+0000","last_peered":"2024-03-24T06:38:56.818544+0000","last_clean":"2024-03-24T06:38:56.818544+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:38:56.818544+0000","last_undegraded":"2024-03-24T06:38:56.818544+0000","last_fullsized":"2024-03-24T06:38:56.818544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773645+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:38:56.773645+0000","last_peered":"2024-03-24T06:38:56.773645+0000","last_clean":"2024-03-24T06:38:56.773645+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:38:56.773645+0000","last_undegraded":"2024-03-24T06:38:56.773645+0000","last_fullsized":"2024-03-24T06:38:56.773645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781817+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:38:57.781817+0000","last_peered":"2024-03-24T06:38:57.781817+0000","last_clean":"2024-03-24T06:38:57.781817+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:38:57.781817+0000","last_undegraded":"2024-03-24T06:38:57.781817+0000","last_fullsized":"2024-03-24T06:38:57.781817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564312+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:38:58.564312+0000","last_peered":"2024-03-24T06:38:58.564312+0000","last_clean":"2024-03-24T06:38:58.564312+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:38:58.564312+0000","last_undegraded":"2024-03-24T06:38:58.564312+0000","last_fullsized":"2024-03-24T06:38:58.564312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823559+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:37:56.823559+0000","last_peered":"2024-03-24T06:37:56.823559+0000","last_clean":"2024-03-24T06:37:56.823559+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:37:56.823559+0000","last_undegraded":"2024-03-24T06:37:56.823559+0000","last_fullsized":"2024-03-24T06:37:56.823559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":123,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781262+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:38:57.781262+0000","last_peered":"2024-03-24T06:38:57.781262+0000","last_clean":"2024-03-24T06:38:57.781262+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:38:57.781262+0000","last_undegraded":"2024-03-24T06:38:57.781262+0000","last_fullsized":"2024-03-24T06:38:57.781262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564298+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:38:58.564298+0000","last_peered":"2024-03-24T06:38:58.564298+0000","last_clean":"2024-03-24T06:38:58.564298+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:38:58.564298+0000","last_undegraded":"2024-03-24T06:38:58.564298+0000","last_fullsized":"2024-03-24T06:38:58.564298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.818199+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:38:56.818199+0000","last_peered":"2024-03-24T06:38:56.818199+0000","last_clean":"2024-03-24T06:38:56.818199+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:38:56.818199+0000","last_undegraded":"2024-03-24T06:38:56.818199+0000","last_fullsized":"2024-03-24T06:38:56.818199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824434+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:37:56.824434+0000","last_peered":"2024-03-24T06:37:56.824434+0000","last_clean":"2024-03-24T06:37:56.824434+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:37:56.824434+0000","last_undegraded":"2024-03-24T06:37:56.824434+0000","last_fullsized":"2024-03-24T06:37:56.824434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781632+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:38:57.781632+0000","last_peered":"2024-03-24T06:38:57.781632+0000","last_clean":"2024-03-24T06:38:57.781632+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:38:57.781632+0000","last_undegraded":"2024-03-24T06:38:57.781632+0000","last_fullsized":"2024-03-24T06:38:57.781632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781537+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:38:57.781537+0000","last_peered":"2024-03-24T06:38:57.781537+0000","last_clean":"2024-03-24T06:38:57.781537+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:38:57.781537+0000","last_undegraded":"2024-03-24T06:38:57.781537+0000","last_fullsized":"2024-03-24T06:38:57.781537+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.774014+0000","last_change":"2024-03-24T06:37:56.837415+0000","last_active":"2024-03-24T06:38:56.774014+0000","last_peered":"2024-03-24T06:38:56.774014+0000","last_clean":"2024-03-24T06:38:56.774014+0000","last_became_active":"2024-03-24T06:37:56.832240+0000","last_became_peered":"2024-03-24T06:37:56.832240+0000","last_unstale":"2024-03-24T06:38:56.774014+0000","last_undegraded":"2024-03-24T06:38:56.774014+0000","last_fullsized":"2024-03-24T06:38:56.774014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:44.867297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773959+0000","last_change":"2024-03-24T06:37:56.826934+0000","last_active":"2024-03-24T06:38:56.773959+0000","last_peered":"2024-03-24T06:38:56.773959+0000","last_clean":"2024-03-24T06:38:56.773959+0000","last_became_active":"2024-03-24T06:37:56.826574+0000","last_became_peered":"2024-03-24T06:37:56.826574+0000","last_unstale":"2024-03-24T06:38:56.773959+0000","last_undegraded":"2024-03-24T06:38:56.773959+0000","last_fullsized":"2024-03-24T06:38:56.773959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:52.406936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823017+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:37:56.823017+0000","last_peered":"2024-03-24T06:37:56.823017+0000","last_clean":"2024-03-24T06:37:56.823017+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:37:56.823017+0000","last_undegraded":"2024-03-24T06:37:56.823017+0000","last_fullsized":"2024-03-24T06:37:56.823017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.832165+0000","last_change":"2024-03-24T06:37:56.832165+0000","last_active":"2024-03-24T06:37:56.832165+0000","last_peered":"2024-03-24T06:37:56.832165+0000","last_clean":"2024-03-24T06:37:56.832165+0000","last_became_active":"2024-03-24T06:37:56.831670+0000","last_became_peered":"2024-03-24T06:37:56.831670+0000","last_unstale":"2024-03-24T06:37:56.832165+0000","last_undegraded":"2024-03-24T06:37:56.832165+0000","last_fullsized":"2024-03-24T06:37:56.832165+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.201188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.893903+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:38:57.893903+0000","last_peered":"2024-03-24T06:38:57.893903+0000","last_clean":"2024-03-24T06:38:57.893903+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:38:57.893903+0000","last_undegraded":"2024-03-24T06:38:57.893903+0000","last_fullsized":"2024-03-24T06:38:57.893903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.818536+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:38:56.818536+0000","last_peered":"2024-03-24T06:38:56.818536+0000","last_clean":"2024-03-24T06:38:56.818536+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:38:56.818536+0000","last_undegraded":"2024-03-24T06:38:56.818536+0000","last_fullsized":"2024-03-24T06:38:56.818536+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":122,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564152+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:38:58.564152+0000","last_peered":"2024-03-24T06:38:58.564152+0000","last_clean":"2024-03-24T06:38:58.564152+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:38:58.564152+0000","last_undegraded":"2024-03-24T06:38:58.564152+0000","last_fullsized":"2024-03-24T06:38:58.564152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.893760+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:38:57.893760+0000","last_peered":"2024-03-24T06:38:57.893760+0000","last_clean":"2024-03-24T06:38:57.893760+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:38:57.893760+0000","last_undegraded":"2024-03-24T06:38:57.893760+0000","last_fullsized":"2024-03-24T06:38:57.893760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817934+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:38:56.817934+0000","last_peered":"2024-03-24T06:38:56.817934+0000","last_clean":"2024-03-24T06:38:56.817934+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:38:56.817934+0000","last_undegraded":"2024-03-24T06:38:56.817934+0000","last_fullsized":"2024-03-24T06:38:56.817934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781782+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:38:57.781782+0000","last_peered":"2024-03-24T06:38:57.781782+0000","last_clean":"2024-03-24T06:38:57.781782+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:38:57.781782+0000","last_undegraded":"2024-03-24T06:38:57.781782+0000","last_fullsized":"2024-03-24T06:38:57.781782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":119,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817448+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:38:56.817448+0000","last_peered":"2024-03-24T06:38:56.817448+0000","last_clean":"2024-03-24T06:38:56.817448+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:38:56.817448+0000","last_undegraded":"2024-03-24T06:38:56.817448+0000","last_fullsized":"2024-03-24T06:38:56.817448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564359+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:38:58.564359+0000","last_peered":"2024-03-24T06:38:58.564359+0000","last_clean":"2024-03-24T06:38:58.564359+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:38:58.564359+0000","last_undegraded":"2024-03-24T06:38:58.564359+0000","last_fullsized":"2024-03-24T06:38:58.564359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.128952+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:38:55.128952+0000","last_peered":"2024-03-24T06:38:55.128952+0000","last_clean":"2024-03-24T06:38:55.128952+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:38:55.128952+0000","last_undegraded":"2024-03-24T06:38:55.128952+0000","last_fullsized":"2024-03-24T06:38:55.128952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781353+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:38:57.781353+0000","last_peered":"2024-03-24T06:38:57.781353+0000","last_clean":"2024-03-24T06:38:57.781353+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:38:57.781353+0000","last_undegraded":"2024-03-24T06:38:57.781353+0000","last_fullsized":"2024-03-24T06:38:57.781353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.563984+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:38:58.563984+0000","last_peered":"2024-03-24T06:38:58.563984+0000","last_clean":"2024-03-24T06:38:58.563984+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:38:58.563984+0000","last_undegraded":"2024-03-24T06:38:58.563984+0000","last_fullsized":"2024-03-24T06:38:58.563984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.782118+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:38:57.782118+0000","last_peered":"2024-03-24T06:38:57.782118+0000","last_clean":"2024-03-24T06:38:57.782118+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:38:57.782118+0000","last_undegraded":"2024-03-24T06:38:57.782118+0000","last_fullsized":"2024-03-24T06:38:57.782118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817302+0000","last_change":"2024-03-24T06:37:56.831122+0000","last_active":"2024-03-24T06:38:56.817302+0000","last_peered":"2024-03-24T06:38:56.817302+0000","last_clean":"2024-03-24T06:38:56.817302+0000","last_became_active":"2024-03-24T06:37:56.830642+0000","last_became_peered":"2024-03-24T06:37:56.830642+0000","last_unstale":"2024-03-24T06:38:56.817302+0000","last_undegraded":"2024-03-24T06:38:56.817302+0000","last_fullsized":"2024-03-24T06:38:56.817302+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.204542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564113+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:38:58.564113+0000","last_peered":"2024-03-24T06:38:58.564113+0000","last_clean":"2024-03-24T06:38:58.564113+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:38:58.564113+0000","last_undegraded":"2024-03-24T06:38:58.564113+0000","last_fullsized":"2024-03-24T06:38:58.564113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773544+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:38:56.773544+0000","last_peered":"2024-03-24T06:38:56.773544+0000","last_clean":"2024-03-24T06:38:56.773544+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:38:56.773544+0000","last_undegraded":"2024-03-24T06:38:56.773544+0000","last_fullsized":"2024-03-24T06:38:56.773544+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.824121+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:37:56.824121+0000","last_peered":"2024-03-24T06:37:56.824121+0000","last_clean":"2024-03-24T06:37:56.824121+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:37:56.824121+0000","last_undegraded":"2024-03-24T06:37:56.824121+0000","last_fullsized":"2024-03-24T06:37:56.824121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817000+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:38:56.817000+0000","last_peered":"2024-03-24T06:38:56.817000+0000","last_clean":"2024-03-24T06:38:56.817000+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:38:56.817000+0000","last_undegraded":"2024-03-24T06:38:56.817000+0000","last_fullsized":"2024-03-24T06:38:56.817000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.738772+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:37:57.738772+0000","last_peered":"2024-03-24T06:37:57.738772+0000","last_clean":"2024-03-24T06:37:57.738772+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:37:57.738772+0000","last_undegraded":"2024-03-24T06:37:57.738772+0000","last_fullsized":"2024-03-24T06:37:57.738772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.822847+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:37:56.822847+0000","last_peered":"2024-03-24T06:37:56.822847+0000","last_clean":"2024-03-24T06:37:56.822847+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:37:56.822847+0000","last_undegraded":"2024-03-24T06:37:56.822847+0000","last_fullsized":"2024-03-24T06:37:56.822847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.130621+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:38:55.130621+0000","last_peered":"2024-03-24T06:38:55.130621+0000","last_clean":"2024-03-24T06:38:55.130621+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:38:55.130621+0000","last_undegraded":"2024-03-24T06:38:55.130621+0000","last_fullsized":"2024-03-24T06:38:55.130621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826427+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:37:56.826427+0000","last_peered":"2024-03-24T06:37:56.826427+0000","last_clean":"2024-03-24T06:37:56.826427+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:37:56.826427+0000","last_undegraded":"2024-03-24T06:37:56.826427+0000","last_fullsized":"2024-03-24T06:37:56.826427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781591+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:38:57.781591+0000","last_peered":"2024-03-24T06:38:57.781591+0000","last_clean":"2024-03-24T06:38:57.781591+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:38:57.781591+0000","last_undegraded":"2024-03-24T06:38:57.781591+0000","last_fullsized":"2024-03-24T06:38:57.781591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.141644+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:38:55.141644+0000","last_peered":"2024-03-24T06:38:55.141644+0000","last_clean":"2024-03-24T06:38:55.141644+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:38:55.141644+0000","last_undegraded":"2024-03-24T06:38:55.141644+0000","last_fullsized":"2024-03-24T06:38:55.141644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781374+0000","last_change":"2024-03-24T06:37:57.056479+0000","last_active":"2024-03-24T06:38:57.781374+0000","last_peered":"2024-03-24T06:38:57.781374+0000","last_clean":"2024-03-24T06:38:57.781374+0000","last_became_active":"2024-03-24T06:37:57.055993+0000","last_became_peered":"2024-03-24T06:37:57.055993+0000","last_unstale":"2024-03-24T06:38:57.781374+0000","last_undegraded":"2024-03-24T06:38:57.781374+0000","last_fullsized":"2024-03-24T06:38:57.781374+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:12.473494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054208+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:37:57.054208+0000","last_peered":"2024-03-24T06:37:57.054208+0000","last_clean":"2024-03-24T06:37:57.054208+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:37:57.054208+0000","last_undegraded":"2024-03-24T06:37:57.054208+0000","last_fullsized":"2024-03-24T06:37:57.054208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.062142+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:38:55.062142+0000","last_peered":"2024-03-24T06:38:55.062142+0000","last_clean":"2024-03-24T06:38:55.062142+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:38:55.062142+0000","last_undegraded":"2024-03-24T06:38:55.062142+0000","last_fullsized":"2024-03-24T06:38:55.062142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823636+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:37:56.823636+0000","last_peered":"2024-03-24T06:37:56.823636+0000","last_clean":"2024-03-24T06:37:56.823636+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:37:56.823636+0000","last_undegraded":"2024-03-24T06:37:56.823636+0000","last_fullsized":"2024-03-24T06:37:56.823636+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.739104+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:37:57.739104+0000","last_peered":"2024-03-24T06:37:57.739104+0000","last_clean":"2024-03-24T06:37:57.739104+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:37:57.739104+0000","last_undegraded":"2024-03-24T06:37:57.739104+0000","last_fullsized":"2024-03-24T06:37:57.739104+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.772850+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:38:56.772850+0000","last_peered":"2024-03-24T06:38:56.772850+0000","last_clean":"2024-03-24T06:38:56.772850+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:38:56.772850+0000","last_undegraded":"2024-03-24T06:38:56.772850+0000","last_fullsized":"2024-03-24T06:38:56.772850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781630+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:38:57.781630+0000","last_peered":"2024-03-24T06:38:57.781630+0000","last_clean":"2024-03-24T06:38:57.781630+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:38:57.781630+0000","last_undegraded":"2024-03-24T06:38:57.781630+0000","last_fullsized":"2024-03-24T06:38:57.781630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825809+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:37:56.825809+0000","last_peered":"2024-03-24T06:37:56.825809+0000","last_clean":"2024-03-24T06:37:56.825809+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:37:56.825809+0000","last_undegraded":"2024-03-24T06:37:56.825809+0000","last_fullsized":"2024-03-24T06:37:56.825809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.142252+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:38:55.142252+0000","last_peered":"2024-03-24T06:38:55.142252+0000","last_clean":"2024-03-24T06:38:55.142252+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:38:55.142252+0000","last_undegraded":"2024-03-24T06:38:55.142252+0000","last_fullsized":"2024-03-24T06:38:55.142252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781393+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:38:57.781393+0000","last_peered":"2024-03-24T06:38:57.781393+0000","last_clean":"2024-03-24T06:38:57.781393+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:38:57.781393+0000","last_undegraded":"2024-03-24T06:38:57.781393+0000","last_fullsized":"2024-03-24T06:38:57.781393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.823718+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:37:56.823718+0000","last_peered":"2024-03-24T06:37:56.823718+0000","last_clean":"2024-03-24T06:37:56.823718+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:37:56.823718+0000","last_undegraded":"2024-03-24T06:37:56.823718+0000","last_fullsized":"2024-03-24T06:37:56.823718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564229+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:38:58.564229+0000","last_peered":"2024-03-24T06:38:58.564229+0000","last_clean":"2024-03-24T06:38:58.564229+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:38:58.564229+0000","last_undegraded":"2024-03-24T06:38:58.564229+0000","last_fullsized":"2024-03-24T06:38:58.564229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:57.054403+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:37:57.054403+0000","last_peered":"2024-03-24T06:37:57.054403+0000","last_clean":"2024-03-24T06:37:57.054403+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:37:57.054403+0000","last_undegraded":"2024-03-24T06:37:57.054403+0000","last_fullsized":"2024-03-24T06:37:57.054403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:53.520929+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:38:53.520929+0000","last_peered":"2024-03-24T06:38:53.520929+0000","last_clean":"2024-03-24T06:38:53.520929+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:38:53.520929+0000","last_undegraded":"2024-03-24T06:38:53.520929+0000","last_fullsized":"2024-03-24T06:38:53.520929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564504+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:38:58.564504+0000","last_peered":"2024-03-24T06:38:58.564504+0000","last_clean":"2024-03-24T06:38:58.564504+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:38:58.564504+0000","last_undegraded":"2024-03-24T06:38:58.564504+0000","last_fullsized":"2024-03-24T06:38:58.564504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.829630+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:37:56.829630+0000","last_peered":"2024-03-24T06:37:56.829630+0000","last_clean":"2024-03-24T06:37:56.829630+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:37:56.829630+0000","last_undegraded":"2024-03-24T06:37:56.829630+0000","last_fullsized":"2024-03-24T06:37:56.829630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.782200+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:38:57.782200+0000","last_peered":"2024-03-24T06:38:57.782200+0000","last_clean":"2024-03-24T06:38:57.782200+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:38:57.782200+0000","last_undegraded":"2024-03-24T06:38:57.782200+0000","last_fullsized":"2024-03-24T06:38:57.782200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564263+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:38:58.564263+0000","last_peered":"2024-03-24T06:38:58.564263+0000","last_clean":"2024-03-24T06:38:58.564263+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:38:58.564263+0000","last_undegraded":"2024-03-24T06:38:58.564263+0000","last_fullsized":"2024-03-24T06:38:58.564263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781358+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:38:57.781358+0000","last_peered":"2024-03-24T06:38:57.781358+0000","last_clean":"2024-03-24T06:38:57.781358+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:38:57.781358+0000","last_undegraded":"2024-03-24T06:38:57.781358+0000","last_fullsized":"2024-03-24T06:38:57.781358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.818122+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:38:56.818122+0000","last_peered":"2024-03-24T06:38:56.818122+0000","last_clean":"2024-03-24T06:38:56.818122+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:38:56.818122+0000","last_undegraded":"2024-03-24T06:38:56.818122+0000","last_fullsized":"2024-03-24T06:38:56.818122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:58.564566+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:38:58.564566+0000","last_peered":"2024-03-24T06:38:58.564566+0000","last_clean":"2024-03-24T06:38:58.564566+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:38:58.564566+0000","last_undegraded":"2024-03-24T06:38:58.564566+0000","last_fullsized":"2024-03-24T06:38:58.564566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773324+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:38:56.773324+0000","last_peered":"2024-03-24T06:38:56.773324+0000","last_clean":"2024-03-24T06:38:56.773324+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:38:56.773324+0000","last_undegraded":"2024-03-24T06:38:56.773324+0000","last_fullsized":"2024-03-24T06:38:56.773324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.832171+0000","last_change":"2024-03-24T06:37:56.832171+0000","last_active":"2024-03-24T06:37:56.832171+0000","last_peered":"2024-03-24T06:37:56.832171+0000","last_clean":"2024-03-24T06:37:56.832171+0000","last_became_active":"2024-03-24T06:37:56.831734+0000","last_became_peered":"2024-03-24T06:37:56.831734+0000","last_unstale":"2024-03-24T06:37:56.832171+0000","last_undegraded":"2024-03-24T06:37:56.832171+0000","last_fullsized":"2024-03-24T06:37:56.832171+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:19.788831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.825693+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:37:56.825693+0000","last_peered":"2024-03-24T06:37:56.825693+0000","last_clean":"2024-03-24T06:37:56.825693+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:37:56.825693+0000","last_undegraded":"2024-03-24T06:37:56.825693+0000","last_fullsized":"2024-03-24T06:37:56.825693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.145970+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:38:55.145970+0000","last_peered":"2024-03-24T06:38:55.145970+0000","last_clean":"2024-03-24T06:38:55.145970+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:38:55.145970+0000","last_undegraded":"2024-03-24T06:38:55.145970+0000","last_fullsized":"2024-03-24T06:38:55.145970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.816864+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:38:56.816864+0000","last_peered":"2024-03-24T06:38:56.816864+0000","last_clean":"2024-03-24T06:38:56.816864+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:38:56.816864+0000","last_undegraded":"2024-03-24T06:38:56.816864+0000","last_fullsized":"2024-03-24T06:38:56.816864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773941+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:38:56.773941+0000","last_peered":"2024-03-24T06:38:56.773941+0000","last_clean":"2024-03-24T06:38:56.773941+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:38:56.773941+0000","last_undegraded":"2024-03-24T06:38:56.773941+0000","last_fullsized":"2024-03-24T06:38:56.773941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:37:56.826534+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:37:56.826534+0000","last_peered":"2024-03-24T06:37:56.826534+0000","last_clean":"2024-03-24T06:37:56.826534+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:37:56.826534+0000","last_undegraded":"2024-03-24T06:37:56.826534+0000","last_fullsized":"2024-03-24T06:37:56.826534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.112672+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:38:55.112672+0000","last_peered":"2024-03-24T06:38:55.112672+0000","last_clean":"2024-03-24T06:38:55.112672+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:38:55.112672+0000","last_undegraded":"2024-03-24T06:38:55.112672+0000","last_fullsized":"2024-03-24T06:38:55.112672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-24T06:38:57.781523+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:38:57.781523+0000","last_peered":"2024-03-24T06:38:57.781523+0000","last_clean":"2024-03-24T06:38:57.781523+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:38:57.781523+0000","last_undegraded":"2024-03-24T06:38:57.781523+0000","last_fullsized":"2024-03-24T06:38:57.781523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.118651+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:38:55.118651+0000","last_peered":"2024-03-24T06:38:55.118651+0000","last_clean":"2024-03-24T06:38:55.118651+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:38:55.118651+0000","last_undegraded":"2024-03-24T06:38:55.118651+0000","last_fullsized":"2024-03-24T06:38:55.118651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T06:38:55.135232+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:38:55.135232+0000","last_peered":"2024-03-24T06:38:55.135232+0000","last_clean":"2024-03-24T06:38:55.135232+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:38:55.135232+0000","last_undegraded":"2024-03-24T06:38:55.135232+0000","last_fullsized":"2024-03-24T06:38:55.135232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.817787+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:38:56.817787+0000","last_peered":"2024-03-24T06:38:56.817787+0000","last_clean":"2024-03-24T06:38:56.817787+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:38:56.817787+0000","last_undegraded":"2024-03-24T06:38:56.817787+0000","last_fullsized":"2024-03-24T06:38:56.817787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T06:38:56.773672+0000","last_change":"2024-03-24T06:37:56.828731+0000","last_active":"2024-03-24T06:38:56.773672+0000","last_peered":"2024-03-24T06:38:56.773672+0000","last_clean":"2024-03-24T06:38:56.773672+0000","last_became_active":"2024-03-24T06:37:56.828384+0000","last_became_peered":"2024-03-24T06:37:56.828384+0000","last_unstale":"2024-03-24T06:38:56.773672+0000","last_undegraded":"2024-03-24T06:38:56.773672+0000","last_fullsized":"2024-03-24T06:38:56.773672+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:26.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458,"ondisk_log_size":458,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67893,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":869,"num_read_kb":515,"num_write":244,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":510805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28720,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716432,"statfs":{"total":95995035648,"available":95965626368,"internally_reserved":0,"allocated":1740800,"data_stored":1159070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":501719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1116526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1097728,"data_stored":559368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":558717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":510913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36976,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708176,"statfs":{"total":95995035648,"available":95957172224,"internally_reserved":0,"allocated":1675264,"data_stored":1102779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:39:01.845 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T06:39:01.846 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T06:39:01.846 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T06:39:01.846 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T06:39:02.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:01 smithi022 ceph-mon[32003]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:39:02.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:01 smithi143 ceph-mon[36520]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:39:02.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:01 smithi155 ceph-mon[36262]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:39:02.350 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:03.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:02 smithi022 ceph-mon[32003]: pgmap v386: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T06:39:03.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:02 smithi143 ceph-mon[36520]: pgmap v386: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T06:39:03.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:02 smithi155 ceph-mon[36262]: pgmap v386: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 9 op/s 2024-03-24T06:39:04.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 -- 172.21.15.22:0/1037217178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 msgr2=0x7f0ab8101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 --2- 172.21.15.22:0/1037217178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab8101560 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0aa40098f0 tx=0x7f0aa4009c00 comp rx=0 tx=0).stop 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 -- 172.21.15.22:0/1037217178 shutdown_connections 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 --2- 172.21.15.22:0/1037217178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8107140 0x7f0ab8107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 --2- 172.21.15.22:0/1037217178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8101aa0 0x7f0ab810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 --2- 172.21.15.22:0/1037217178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab8101560 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 -- 172.21.15.22:0/1037217178 >> 172.21.15.22:0/1037217178 conn(0x7f0ab8078030 msgr2=0x7f0ab8078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 -- 172.21.15.22:0/1037217178 shutdown_connections 2024-03-24T06:39:04.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.491+0000 7f0abe400700 1 -- 172.21.15.22:0/1037217178 wait complete. 2024-03-24T06:39:04.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.492+0000 7f0abe400700 1 Processor -- start 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.492+0000 7f0abe400700 1 -- start start 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8101aa0 0x7f0ab819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8107140 0x7f0ab81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab8198c30 con 0x7f0ab8101aa0 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab8198d70 con 0x7f0ab8101100 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0abe400700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab8198f30 con 0x7f0ab8107140 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0ab7400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0ab7400700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:53948/0 (socket says 172.21.15.22:53948) 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0ab7400700 1 -- 172.21.15.22:0/2605558686 learned_addr learned my addr 172.21.15.22:0/2605558686 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:04.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0ab7e00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8107140 0x7f0ab81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.493+0000 7f0ab6a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8101aa0 0x7f0ab819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 -- 172.21.15.22:0/2605558686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8107140 msgr2=0x7f0ab81985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8107140 0x7f0ab81985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 -- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8101aa0 msgr2=0x7f0ab819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8101aa0 0x7f0ab819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 -- 172.21.15.22:0/2605558686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab0009d00 con 0x7f0ab8101100 2024-03-24T06:39:04.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0ab7400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800e9a0 tx=0x7f0aa800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8009790 con 0x7f0ab8101100 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aa8010430 con 0x7f0ab8101100 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8018680 con 0x7f0ab8101100 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.494+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa4009520 con 0x7f0ab8101100 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.495+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab810e880 con 0x7f0ab8101100 2024-03-24T06:39:04.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.496+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab8109520 con 0x7f0ab8101100 2024-03-24T06:39:04.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.498+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0aa8010910 con 0x7f0ab8101100 2024-03-24T06:39:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.499+0000 7f0aafe00700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 0x7f0a9c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.499+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f0aa8014070 con 0x7f0ab8101100 2024-03-24T06:39:04.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.499+0000 7f0ab6a00700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 0x7f0a9c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:04.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.500+0000 7f0ab6a00700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 0x7f0a9c0a4690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009810 tx=0x7f0aa4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:04.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.501+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0aa8096d20 con 0x7f0ab8101100 2024-03-24T06:39:04.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.908+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0ab8199b00 con 0x7f0ab8101100 2024-03-24T06:39:04.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.908+0000 7f0aafe00700 1 -- 172.21.15.22:0/2605558686 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0aa80105a0 con 0x7f0ab8101100 2024-03-24T06:39:04.908 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:04.908 INFO:teuthology.orchestra.run.smithi022.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 msgr2=0x7f0a9c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 0x7f0a9c0a4690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009810 tx=0x7f0aa4019040 comp rx=0 tx=0).stop 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 msgr2=0x7f0ab819e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800e9a0 tx=0x7f0aa800ecb0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 shutdown_connections 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0a9c0a21d0 0x7f0a9c0a4690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ab8107140 0x7f0ab81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab8101100 0x7f0ab819e410 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 --2- 172.21.15.22:0/2605558686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ab8101aa0 0x7f0ab819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 >> 172.21.15.22:0/2605558686 conn(0x7f0ab8078030 msgr2=0x7f0ab80fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.912+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 shutdown_connections 2024-03-24T06:39:04.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:04.913+0000 7f0abe400700 1 -- 172.21.15.22:0/2605558686 wait complete. 2024-03-24T06:39:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:04 smithi022 ceph-mon[32003]: pgmap v387: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T06:39:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:04 smithi143 ceph-mon[36520]: pgmap v387: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T06:39:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:04 smithi155 ceph-mon[36262]: pgmap v387: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T06:39:05.760 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T06:39:05.762 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 -- 172.21.15.22:0/4148142430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 msgr2=0x7f1a3c10bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 --2- 172.21.15.22:0/4148142430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c10bd90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009ca0 tx=0x7f1a3402c0f0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 -- 172.21.15.22:0/4148142430 shutdown_connections 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 --2- 172.21.15.22:0/4148142430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c10bd90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 --2- 172.21.15.22:0/4148142430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 --2- 172.21.15.22:0/4148142430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 0x7f1a3c103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 -- 172.21.15.22:0/4148142430 >> 172.21.15.22:0/4148142430 conn(0x7f1a3c0fef00 msgr2=0x7f1a3c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:05.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.963+0000 7f1a42c00640 1 -- 172.21.15.22:0/4148142430 shutdown_connections 2024-03-24T06:39:05.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.964+0000 7f1a42c00640 1 -- 172.21.15.22:0/4148142430 wait complete. 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.965+0000 7f1a42c00640 1 Processor -- start 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.965+0000 7f1a42c00640 1 -- start start 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 0x7f1a3c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c1a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1a1470 con 0x7f1a3c103950 2024-03-24T06:39:05.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1a15e0 con 0x7f1a3c105500 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a42c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1a17a0 con 0x7f1a3c104b40 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a33400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a3be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 0x7f1a3c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a40c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c1a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a33400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54976/0 (socket says 172.21.15.22:54976) 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.966+0000 7f1a33400640 1 -- 172.21.15.22:0/2285399567 learned_addr learned my addr 172.21.15.22:0/2285399567 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 -- 172.21.15.22:0/2285399567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 msgr2=0x7f1a3c199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:05.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 -- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 msgr2=0x7f1a3c199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 0x7f1a3c199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 -- 172.21.15.22:0/2285399567 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a34009900 con 0x7f1a3c105500 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a33400640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a40c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c1a0f30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a34037040 tx=0x7f1a34032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34034e20 con 0x7f1a3c105500 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a34003e50 con 0x7f1a3c105500 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c1a1a20 con 0x7f1a3c105500 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.967+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34004800 con 0x7f1a3c105500 2024-03-24T06:39:05.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.968+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c1a1ef0 con 0x7f1a3c105500 2024-03-24T06:39:05.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.969+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3c10ac70 con 0x7f1a3c105500 2024-03-24T06:39:05.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.970+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f1a3400a040 con 0x7f1a3c105500 2024-03-24T06:39:05.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.973+0000 7f1a39600640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 0x7f1a200a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:05.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.973+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f1a340f0a00 con 0x7f1a3c105500 2024-03-24T06:39:05.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.973+0000 7f1a3be00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 0x7f1a200a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:05.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.974+0000 7f1a3be00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 0x7f1a200a4600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1049a0 tx=0x7f1a2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:05.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:05.975+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1a340b95f0 con 0x7f1a3c105500 2024-03-24T06:39:06.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:05 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2605558686' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T06:39:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:05 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2605558686' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T06:39:06.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.227+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a3c10f350 con 0x7f1a3c105500 2024-03-24T06:39:06.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.229+0000 7f1a39600640 1 -- 172.21.15.22:0/2285399567 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7f1a340bccb0 con 0x7f1a3c105500 2024-03-24T06:39:06.229 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:06.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 msgr2=0x7f1a200a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 0x7f1a200a4600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1049a0 tx=0x7f1a2c005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:06.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 msgr2=0x7f1a3c1a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c1a0f30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a34037040 tx=0x7f1a34032f80 comp rx=0 tx=0).stop 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 shutdown_connections 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1a200a2140 0x7f1a200a4600 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a3c105500 0x7f1a3c1a0f30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c104b40 0x7f1a3c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 --2- 172.21.15.22:0/2285399567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a3c103950 0x7f1a3c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 >> 172.21.15.22:0/2285399567 conn(0x7f1a3c0fef00 msgr2=0x7f1a3c107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 shutdown_connections 2024-03-24T06:39:06.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.232+0000 7f1a42c00640 1 -- 172.21.15.22:0/2285399567 wait complete. 2024-03-24T06:39:06.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:39:06.251 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:39:06.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:05 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2605558686' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.486+0000 7f5efa800640 1 -- 172.21.15.22:0/2670918243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 msgr2=0x7f5ef410b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.486+0000 7f5efa800640 1 --2- 172.21.15.22:0/2670918243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef410b950 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009ca0 tx=0x7f5eec02c180 comp rx=0 tx=0).stop 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 -- 172.21.15.22:0/2670918243 shutdown_connections 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 --2- 172.21.15.22:0/2670918243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef410b950 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 --2- 172.21.15.22:0/2670918243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40740f0 0x7f5ef4074550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 --2- 172.21.15.22:0/2670918243 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40756d0 0x7f5ef4073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 -- 172.21.15.22:0/2670918243 >> 172.21.15.22:0/2670918243 conn(0x7f5ef40fcb80 msgr2=0x7f5ef40fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 -- 172.21.15.22:0/2670918243 shutdown_connections 2024-03-24T06:39:06.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.487+0000 7f5efa800640 1 -- 172.21.15.22:0/2670918243 wait complete. 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.488+0000 7f5efa800640 1 Processor -- start 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.488+0000 7f5efa800640 1 -- start start 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.488+0000 7f5efa800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 0x7f5ef4199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.488+0000 7f5efa800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5efa800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef41a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5efa800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41a1300 con 0x7f5ef40740f0 2024-03-24T06:39:06.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5efa800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41a1470 con 0x7f5ef40756d0 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5efa800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41a1630 con 0x7f5ef41094c0 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 0x7f5ef4199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:45766/0 (socket says 172.21.15.22:45766) 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 -- 172.21.15.22:0/4252302251 learned_addr learned my addr 172.21.15.22:0/4252302251 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef9e00640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef41a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 -- 172.21.15.22:0/4252302251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 msgr2=0x7f5ef41a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef41a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 -- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 msgr2=0x7f5ef4199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 0x7f5ef4199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef8a00640 1 -- 172.21.15.22:0/4252302251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee0009490 con 0x7f5ef40756d0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef9400640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 0x7f5ef4199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.489+0000 7f5ef9e00640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5ef8a00640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000ea60 tx=0x7f5ee000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee000cde0 con 0x7f5ef40756d0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ee00035f0 con 0x7f5ef40756d0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eec009900 con 0x7f5ef40756d0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee0010690 con 0x7f5ef40756d0 2024-03-24T06:39:06.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.490+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef41a1c90 con 0x7f5ef40756d0 2024-03-24T06:39:06.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.492+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef4108280 con 0x7f5ef40756d0 2024-03-24T06:39:06.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.493+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5ee0003af0 con 0x7f5ef40756d0 2024-03-24T06:39:06.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.494+0000 7f5ef2000640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 0x7f5edc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.494+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f5ee0014070 con 0x7f5ef40756d0 2024-03-24T06:39:06.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.494+0000 7f5ef9400640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 0x7f5edc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.495+0000 7f5ef9400640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 0x7f5edc0a4580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009fc0 tx=0x7f5ee8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:06.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.498+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5ee0097110 con 0x7f5ef40756d0 2024-03-24T06:39:06.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.696+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ef4107cf0 con 0x7f5ef40756d0 2024-03-24T06:39:06.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.696+0000 7f5ef2000640 1 -- 172.21.15.22:0/4252302251 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f5ee009a7d0 con 0x7f5ef40756d0 2024-03-24T06:39:06.695 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 msgr2=0x7f5edc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 0x7f5edc0a4580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009fc0 tx=0x7f5ee8005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 msgr2=0x7f5ef4199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000ea60 tx=0x7f5ee000ef20 comp rx=0 tx=0).stop 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 shutdown_connections 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5edc0a20c0 0x7f5edc0a4580 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ef41094c0 0x7f5ef41a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ef40756d0 0x7f5ef4199d50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 --2- 172.21.15.22:0/4252302251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ef40740f0 0x7f5ef4199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 >> 172.21.15.22:0/4252302251 conn(0x7f5ef40fcb80 msgr2=0x7f5ef4105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 shutdown_connections 2024-03-24T06:39:06.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.699+0000 7f5efa800640 1 -- 172.21.15.22:0/4252302251 wait complete. 2024-03-24T06:39:06.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 -- 172.21.15.22:0/3975544820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00fffa0 msgr2=0x7f8ae01023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 --2- 172.21.15.22:0/3975544820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00fffa0 0x7f8ae01023d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ca0 tx=0x7f8ad402c100 comp rx=0 tx=0).stop 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 -- 172.21.15.22:0/3975544820 shutdown_connections 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 --2- 172.21.15.22:0/3975544820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae01052c0 0x7f8ae0105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 --2- 172.21.15.22:0/3975544820 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae0102910 0x7f8ae0104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 --2- 172.21.15.22:0/3975544820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00fffa0 0x7f8ae01023d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 -- 172.21.15.22:0/3975544820 >> 172.21.15.22:0/3975544820 conn(0x7f8ae00fbba0 msgr2=0x7f8ae00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 -- 172.21.15.22:0/3975544820 shutdown_connections 2024-03-24T06:39:06.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.913+0000 7f8ae8000640 1 -- 172.21.15.22:0/3975544820 wait complete. 2024-03-24T06:39:06.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 Processor -- start 2024-03-24T06:39:06.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 -- start start 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae00fffa0 0x7f8ae019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae01052c0 0x7f8ae01a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01a58f0 con 0x7f8ae00fffa0 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01a5a60 con 0x7f8ae01052c0 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.914+0000 7f8ae8000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01a5c20 con 0x7f8ae0102910 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae5600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae00fffa0 0x7f8ae019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55006/0 (socket says 172.21.15.22:55006) 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 -- 172.21.15.22:0/1180324690 learned_addr learned my addr 172.21.15.22:0/1180324690 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae6000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae01052c0 0x7f8ae01a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 -- 172.21.15.22:0/1180324690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae01052c0 msgr2=0x7f8ae01a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae01052c0 0x7f8ae01a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 -- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae00fffa0 msgr2=0x7f8ae019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:06.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae00fffa0 0x7f8ae019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 -- 172.21.15.22:0/1180324690 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad4009900 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.915+0000 7f8ae4c00640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00e9f0 tx=0x7f8acc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.916+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc009700 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.916+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8acc00cd80 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.916+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae01a5f00 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.916+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc018920 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.916+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae01a63d0 con 0x7f8ae0102910 2024-03-24T06:39:06.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.917+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae00fd690 con 0x7f8ae0102910 2024-03-24T06:39:06.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.919+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f8acc018a80 con 0x7f8ae0102910 2024-03-24T06:39:06.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.920+0000 7f8ade000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 0x7f8ac80a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:06.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.920+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f8acc014070 con 0x7f8ae0102910 2024-03-24T06:39:06.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.920+0000 7f8ae5600640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 0x7f8ac80a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:06.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.921+0000 7f8ae5600640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 0x7f8ac80a4600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ad402c710 tx=0x7f8ad40326d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:06.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:06.922+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8acc096f50 con 0x7f8ae0102910 2024-03-24T06:39:07.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:06 smithi022 ceph-mon[32003]: pgmap v388: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 366 B/s wr, 11 op/s 2024-03-24T06:39:07.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:06 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2285399567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:07.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:06 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4252302251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:07.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:06 smithi143 ceph-mon[36520]: pgmap v388: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 366 B/s wr, 11 op/s 2024-03-24T06:39:07.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:06 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2285399567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:07.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:06 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4252302251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:07.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.181+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ae0113940 con 0x7f8ae0102910 2024-03-24T06:39:07.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.181+0000 7f8ade000640 1 -- 172.21.15.22:0/1180324690 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8acc09a610 con 0x7f8ae0102910 2024-03-24T06:39:07.180 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:07.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 msgr2=0x7f8ac80a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 0x7f8ac80a4600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ad402c710 tx=0x7f8ad40326d0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 msgr2=0x7f8ae019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8acc00e9f0 tx=0x7f8acc00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 shutdown_connections 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f8ac80a2140 0x7f8ac80a4600 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae01052c0 0x7f8ae01a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ae0102910 0x7f8ae019e390 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 --2- 172.21.15.22:0/1180324690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ae00fffa0 0x7f8ae019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 >> 172.21.15.22:0/1180324690 conn(0x7f8ae00fbba0 msgr2=0x7f8ae0103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.184+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 shutdown_connections 2024-03-24T06:39:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.185+0000 7f8ae8000640 1 -- 172.21.15.22:0/1180324690 wait complete. 2024-03-24T06:39:07.200 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:39:07.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:06 smithi155 ceph-mon[36262]: pgmap v388: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 366 B/s wr, 11 op/s 2024-03-24T06:39:07.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:06 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2285399567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:07.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:06 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4252302251' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 -- 172.21.15.22:0/2488745496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26980ffe60 msgr2=0x7f2698100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 --2- 172.21.15.22:0/2488745496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26980ffe60 0x7f2698100280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2688009900 tx=0x7f268802bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 -- 172.21.15.22:0/2488745496 shutdown_connections 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 --2- 172.21.15.22:0/2488745496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 0x7f26981015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 --2- 172.21.15.22:0/2488745496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26981007c0 0x7f2698100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 --2- 172.21.15.22:0/2488745496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26980ffe60 0x7f2698100280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 -- 172.21.15.22:0/2488745496 >> 172.21.15.22:0/2488745496 conn(0x7f26980fba80 msgr2=0x7f26980fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:07.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 -- 172.21.15.22:0/2488745496 shutdown_connections 2024-03-24T06:39:07.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.448+0000 7f269cc00640 1 -- 172.21.15.22:0/2488745496 wait complete. 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.449+0000 7f269cc00640 1 Processor -- start 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.449+0000 7f269cc00640 1 -- start start 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26981007c0 0x7f2698199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 0x7f26981a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f26981a1340 con 0x7f26980ffe60 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f26981a14b0 con 0x7f2698101160 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f269cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f26981a1670 con 0x7f26981007c0 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38912/0 (socket says 172.21.15.22:38912) 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 -- 172.21.15.22:0/3936092388 learned_addr learned my addr 172.21.15.22:0/3936092388 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2696a00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26981007c0 0x7f2698199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697e00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 0x7f26981a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 -- 172.21.15.22:0/3936092388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26981007c0 msgr2=0x7f2698199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26981007c0 0x7f2698199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 -- 172.21.15.22:0/3936092388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 msgr2=0x7f26981a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 0x7f26981a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.450+0000 7f2697400640 1 -- 172.21.15.22:0/3936092388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2688009560 con 0x7f26980ffe60 2024-03-24T06:39:07.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f2697400640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f268802c6b0 tx=0x7f26880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26880044e0 con 0x7f26980ffe60 2024-03-24T06:39:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26981a18f0 con 0x7f26980ffe60 2024-03-24T06:39:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26880054e0 con 0x7f26980ffe60 2024-03-24T06:39:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268803e450 con 0x7f26980ffe60 2024-03-24T06:39:07.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.451+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26981a1e50 con 0x7f26980ffe60 2024-03-24T06:39:07.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.452+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269810ada0 con 0x7f26980ffe60 2024-03-24T06:39:07.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.456+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f26880036f0 con 0x7f26980ffe60 2024-03-24T06:39:07.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.457+0000 7f268fe00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 0x7f267c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.457+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f26880f09d0 con 0x7f26980ffe60 2024-03-24T06:39:07.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.458+0000 7f2696a00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 0x7f267c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.458+0000 7f2696a00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 0x7f267c0a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f269819ad60 tx=0x7f268400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:07.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.462+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f26880b95c0 con 0x7f26980ffe60 2024-03-24T06:39:07.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.665+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f26981a1a80 con 0x7f26980ffe60 2024-03-24T06:39:07.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.666+0000 7f268fe00640 1 -- 172.21.15.22:0/3936092388 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f26981a1a80 con 0x7f26980ffe60 2024-03-24T06:39:07.665 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:39:07.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 msgr2=0x7f267c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 0x7f267c0a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f269819ad60 tx=0x7f268400b390 comp rx=0 tx=0).stop 2024-03-24T06:39:07.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 msgr2=0x7f2698199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f268802c6b0 tx=0x7f26880042f0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 shutdown_connections 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f267c0a20c0 0x7f267c0a4580 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2698101160 0x7f26981a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26981007c0 0x7f2698199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 --2- 172.21.15.22:0/3936092388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f26980ffe60 0x7f2698199850 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 >> 172.21.15.22:0/3936092388 conn(0x7f26980fba80 msgr2=0x7f26980fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.668+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 shutdown_connections 2024-03-24T06:39:07.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.669+0000 7f269cc00640 1 -- 172.21.15.22:0/3936092388 wait complete. 2024-03-24T06:39:07.687 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 -- 172.21.15.22:0/346348591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa364101610 msgr2=0x7fa364101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 --2- 172.21.15.22:0/346348591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa364101610 0x7fa364101a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa358009900 tx=0x7fa35802bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 -- 172.21.15.22:0/346348591 shutdown_connections 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 --2- 172.21.15.22:0/346348591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa3641031c0 0x7fa364109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 --2- 172.21.15.22:0/346348591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364102800 0x7fa364102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 --2- 172.21.15.22:0/346348591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa364101610 0x7fa364101a30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 -- 172.21.15.22:0/346348591 >> 172.21.15.22:0/346348591 conn(0x7fa3640fcb80 msgr2=0x7fa3640fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:07.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 -- 172.21.15.22:0/346348591 shutdown_connections 2024-03-24T06:39:07.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.908+0000 7fa36a400640 1 -- 172.21.15.22:0/346348591 wait complete. 2024-03-24T06:39:07.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.909+0000 7fa36a400640 1 Processor -- start 2024-03-24T06:39:07.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 -- start start 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa364102800 0x7fa364199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 0x7fa3641a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa3641a1450 con 0x7fa364102800 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa3641a15c0 con 0x7fa3641031c0 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa36a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa3641a1780 con 0x7fa364101610 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa369000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa369000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55052/0 (socket says 172.21.15.22:55052) 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa369000640 1 -- 172.21.15.22:0/1180957037 learned_addr learned my addr 172.21.15.22:0/1180957037 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:07.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.910+0000 7fa363e00640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa364102800 0x7fa364199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369a00640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 0x7fa3641a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 -- 172.21.15.22:0/1180957037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 msgr2=0x7fa3641a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 0x7fa3641a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 -- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa364102800 msgr2=0x7fa364199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa364102800 0x7fa364199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 -- 172.21.15.22:0/1180957037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa358009560 con 0x7fa364101610 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369a00640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 0x7fa3641a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa369000640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa3580040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.911+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3580044c0 con 0x7fa364101610 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.912+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3580052d0 con 0x7fa364101610 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.912+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3641a1a00 con 0x7fa364101610 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.912+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358003e50 con 0x7fa364101610 2024-03-24T06:39:07.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.912+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3641a1f60 con 0x7fa364101610 2024-03-24T06:39:07.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.913+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36410d040 con 0x7fa364101610 2024-03-24T06:39:07.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.915+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fa358005440 con 0x7fa364101610 2024-03-24T06:39:07.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.917+0000 7fa361600640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 0x7fa34c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:07.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.917+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa3580f08b0 con 0x7fa364101610 2024-03-24T06:39:07.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.917+0000 7fa363e00640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 0x7fa34c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:07.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.918+0000 7fa363e00640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 0x7fa34c0a4600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa36419ae70 tx=0x7fa350005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:07.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:07.924+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa3580b94a0 con 0x7fa364101610 2024-03-24T06:39:08.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.128+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa3641a2240 con 0x7fa364101610 2024-03-24T06:39:08.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.128+0000 7fa361600640 1 -- 172.21.15.22:0/1180957037 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa3580430a0 con 0x7fa364101610 2024-03-24T06:39:08.127 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 msgr2=0x7fa34c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 0x7fa34c0a4600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa36419ae70 tx=0x7fa350005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 msgr2=0x7fa3641999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa3580040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 shutdown_connections 2024-03-24T06:39:08.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa34c0a2140 0x7fa34c0a4600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3641031c0 0x7fa3641a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa364102800 0x7fa364199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 --2- 172.21.15.22:0/1180957037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa364101610 0x7fa3641999b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 >> 172.21.15.22:0/1180957037 conn(0x7fa3640fcb80 msgr2=0x7fa364073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.131+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 shutdown_connections 2024-03-24T06:39:08.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.132+0000 7fa36a400640 1 -- 172.21.15.22:0/1180957037 wait complete. 2024-03-24T06:39:08.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:07 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1180324690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.147 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:07 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3936092388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.150 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:39:08.177 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:07 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1180324690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:07 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3936092388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:07 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1180324690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:07 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3936092388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 -- 172.21.15.22:0/3270404802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 msgr2=0x7fe4e4109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 --2- 172.21.15.22:0/3270404802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e4109690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009900 tx=0x7fe4dc02bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 -- 172.21.15.22:0/3270404802 shutdown_connections 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 --2- 172.21.15.22:0/3270404802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e4109690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 --2- 172.21.15.22:0/3270404802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4104910 0x7fe4e4106d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 --2- 172.21.15.22:0/3270404802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4101fe0 0x7fe4e41043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 -- 172.21.15.22:0/3270404802 >> 172.21.15.22:0/3270404802 conn(0x7fe4e40fb9f0 msgr2=0x7fe4e40fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 -- 172.21.15.22:0/3270404802 shutdown_connections 2024-03-24T06:39:08.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.368+0000 7fe4eae00640 1 -- 172.21.15.22:0/3270404802 wait complete. 2024-03-24T06:39:08.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.369+0000 7fe4eae00640 1 Processor -- start 2024-03-24T06:39:08.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.369+0000 7fe4eae00640 1 -- start start 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 0x7fe4e40fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e41004d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4100c30 con 0x7fe4e4101fe0 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4100da0 con 0x7fe4e4104910 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4eae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41a15c0 con 0x7fe4e4107240 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 0x7fe4e40fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38940/0 (socket says 172.21.15.22:38940) 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9a00640 1 -- 172.21.15.22:0/2316171220 learned_addr learned my addr 172.21.15.22:0/2316171220 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4ea400640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e41004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 -- 172.21.15.22:0/2316171220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 msgr2=0x7fe4e41004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e41004d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 -- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 msgr2=0x7fe4e4101950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9000640 1 -- 172.21.15.22:0/2316171220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc009560 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.370+0000 7fe4e9a00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4e9000640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 0x7fe4e40fff90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000d6d0 tx=0x7fe4d000db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0005220 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d0009d00 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0004de0 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e41a1870 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.371+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e41a1d30 con 0x7fe4e4104910 2024-03-24T06:39:08.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.372+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e40fd4e0 con 0x7fe4e4104910 2024-03-24T06:39:08.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.374+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe4d0009820 con 0x7fe4e4104910 2024-03-24T06:39:08.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.375+0000 7fe4e2000640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 0x7fe4cc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.376+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fe4d00cd8f0 con 0x7fe4e4104910 2024-03-24T06:39:08.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.376+0000 7fe4e9a00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 0x7fe4cc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.376+0000 7fe4e9a00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 0x7fe4cc0a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800f7d0 tx=0x7fe4d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:08.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.380+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe4d0096590 con 0x7fe4e4104910 2024-03-24T06:39:08.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.630+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4e402d1d0 con 0x7fe4e4104910 2024-03-24T06:39:08.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.631+0000 7fe4e2000640 1 -- 172.21.15.22:0/2316171220 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7fe4d0020090 con 0x7fe4e4104910 2024-03-24T06:39:08.656 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:08.656 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:39:08.657 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 msgr2=0x7fe4cc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 0x7fe4cc0a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe4d800f7d0 tx=0x7fe4d8009500 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 msgr2=0x7fe4e40fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 0x7fe4e40fff90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000d6d0 tx=0x7fe4d000db90 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 shutdown_connections 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe4cc0a20c0 0x7fe4cc0a4580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e4107240 0x7fe4e41004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e4104910 0x7fe4e40fff90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 --2- 172.21.15.22:0/2316171220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e4101fe0 0x7fe4e4101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 >> 172.21.15.22:0/2316171220 conn(0x7fe4e40fb9f0 msgr2=0x7fe4e4102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:08.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 shutdown_connections 2024-03-24T06:39:08.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.634+0000 7fe4eae00640 1 -- 172.21.15.22:0/2316171220 wait complete. 2024-03-24T06:39:08.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.890+0000 7f44a1000640 1 -- 172.21.15.22:0/1449673397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c102770 msgr2=0x7f449c102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.890+0000 7f44a1000640 1 --2- 172.21.15.22:0/1449673397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c102770 0x7f449c102bf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4484009860 tx=0x7f448402bb70 comp rx=0 tx=0).stop 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 -- 172.21.15.22:0/1449673397 shutdown_connections 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 --2- 172.21.15.22:0/1449673397 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 0x7f449c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 --2- 172.21.15.22:0/1449673397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c102770 0x7f449c102bf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 --2- 172.21.15.22:0/1449673397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c101580 0x7f449c1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 -- 172.21.15.22:0/1449673397 >> 172.21.15.22:0/1449673397 conn(0x7f449c0fcaf0 msgr2=0x7f449c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 -- 172.21.15.22:0/1449673397 shutdown_connections 2024-03-24T06:39:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.891+0000 7f44a1000640 1 -- 172.21.15.22:0/1449673397 wait complete. 2024-03-24T06:39:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.892+0000 7f44a1000640 1 Processor -- start 2024-03-24T06:39:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 -- start start 2024-03-24T06:39:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c102770 0x7f449c073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 0x7f449c074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f449c0747a0 con 0x7f449c103130 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f449c074910 con 0x7f449c102770 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f44a1000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f449c1a1620 con 0x7f449c101580 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f449b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f449b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55104/0 (socket says 172.21.15.22:55104) 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f449b400640 1 -- 172.21.15.22:0/1517288842 learned_addr learned my addr 172.21.15.22:0/1517288842 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f4492a00640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c102770 0x7f449c073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.893+0000 7f449be00640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 0x7f449c074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 -- 172.21.15.22:0/1517288842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c102770 msgr2=0x7f449c073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c102770 0x7f449c073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 -- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 msgr2=0x7f449c074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 0x7f449c074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 -- 172.21.15.22:0/1517288842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c009560 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f449b400640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f448c00e9a0 tx=0x7f448c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c00cd80 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f448c010910 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c018980 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.894+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4484009490 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.895+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449c1a1c20 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.896+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f449c108940 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.899+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f448c010a80 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.900+0000 7f4498c00640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 0x7f44800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.901+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f448c014070 con 0x7f449c101580 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.901+0000 7f4492a00640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 0x7f44800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.901+0000 7f4492a00640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 0x7f44800a4580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4484009830 tx=0x7f4484034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:08.904+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f448c097170 con 0x7f449c101580 2024-03-24T06:39:09.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.107+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f449c02d130 con 0x7f449c101580 2024-03-24T06:39:09.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.109+0000 7f4498c00640 1 -- 172.21.15.22:0/1517288842 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7f448c09a830 con 0x7f449c101580 2024-03-24T06:39:09.108 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:09.108 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:09.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 msgr2=0x7f44800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 0x7f44800a4580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4484009830 tx=0x7f4484034be0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 msgr2=0x7f449c0754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f448c00e9a0 tx=0x7f448c00ee60 comp rx=0 tx=0).stop 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 shutdown_connections 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f44800a20c0 0x7f44800a4580 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f449c103130 0x7f449c074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f449c102770 0x7f449c073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 --2- 172.21.15.22:0/1517288842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f449c101580 0x7f449c0754c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.111+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 >> 172.21.15.22:0/1517288842 conn(0x7f449c0fcaf0 msgr2=0x7f449c105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.112+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 shutdown_connections 2024-03-24T06:39:09.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.112+0000 7f44a1000640 1 -- 172.21.15.22:0/1517288842 wait complete. 2024-03-24T06:39:09.127 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:08 smithi022 ceph-mon[32003]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T06:39:09.127 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1180957037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:09.128 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2316171220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:09.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:39:09.131 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:39:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:08 smithi143 ceph-mon[36520]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T06:39:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:08 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1180957037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:08 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2316171220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:08 smithi155 ceph-mon[36262]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T06:39:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:08 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1180957037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:08 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2316171220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:09.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 -- 172.21.15.22:0/3411284884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 msgr2=0x7f4a64100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 --2- 172.21.15.22:0/3411284884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64100280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009900 tx=0x7f4a5802bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 -- 172.21.15.22:0/3411284884 shutdown_connections 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 --2- 172.21.15.22:0/3411284884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 --2- 172.21.15.22:0/3411284884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 0x7f4a64100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 --2- 172.21.15.22:0/3411284884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64100280 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.360+0000 7f4a6a400640 1 -- 172.21.15.22:0/3411284884 >> 172.21.15.22:0/3411284884 conn(0x7f4a640fba80 msgr2=0x7f4a640fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.361+0000 7f4a6a400640 1 -- 172.21.15.22:0/3411284884 shutdown_connections 2024-03-24T06:39:09.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.361+0000 7f4a6a400640 1 -- 172.21.15.22:0/3411284884 wait complete. 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 Processor -- start 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 -- start start 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 0x7f4a64199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4a641a1360 con 0x7f4a641007c0 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4a641a14d0 con 0x7f4a64101160 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a6a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4a641a1690 con 0x7f4a640ffe60 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a69000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a69a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a63e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 0x7f4a64199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a69000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55122/0 (socket says 172.21.15.22:55122) 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a69000640 1 -- 172.21.15.22:0/1748336292 learned_addr learned my addr 172.21.15.22:0/1748336292 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.362+0000 7f4a69a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:45860/0 (socket says 172.21.15.22:45860) 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 -- 172.21.15.22:0/1748336292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 msgr2=0x7f4a641a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 -- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 msgr2=0x7f4a64199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 0x7f4a64199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 -- 172.21.15.22:0/1748336292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a58009560 con 0x7f4a640ffe60 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69a00640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.363+0000 7f4a69000640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a5802c6b0 tx=0x7f4a580042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:09.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.364+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a580044e0 con 0x7f4a640ffe60 2024-03-24T06:39:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.364+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a58004ac0 con 0x7f4a640ffe60 2024-03-24T06:39:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.364+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a641a1910 con 0x7f4a640ffe60 2024-03-24T06:39:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.364+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5803e450 con 0x7f4a640ffe60 2024-03-24T06:39:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.364+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a641a1e70 con 0x7f4a640ffe60 2024-03-24T06:39:09.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.365+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6410ad40 con 0x7f4a640ffe60 2024-03-24T06:39:09.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.368+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f4a58003680 con 0x7f4a640ffe60 2024-03-24T06:39:09.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.370+0000 7f4a61600640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 0x7f4a4c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.370+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f4a580f08f0 con 0x7f4a640ffe60 2024-03-24T06:39:09.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.370+0000 7f4a63e00640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 0x7f4a4c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.371+0000 7f4a63e00640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 0x7f4a4c0a4600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a6419ad80 tx=0x7f4a5000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:09.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.373+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a580b9600 con 0x7f4a640ffe60 2024-03-24T06:39:09.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.573+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a6410a7b0 con 0x7f4a640ffe60 2024-03-24T06:39:09.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.573+0000 7f4a61600640 1 -- 172.21.15.22:0/1748336292 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f4a58043070 con 0x7f4a640ffe60 2024-03-24T06:39:09.572 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 msgr2=0x7f4a4c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 0x7f4a4c0a4600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4a6419ad80 tx=0x7f4a5000b390 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 msgr2=0x7f4a64199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4a5802c6b0 tx=0x7f4a580042f0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 shutdown_connections 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4a4c0a2140 0x7f4a4c0a4600 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a64101160 0x7f4a641a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a641007c0 0x7f4a64199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 --2- 172.21.15.22:0/1748336292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a640ffe60 0x7f4a64199870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.576+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 >> 172.21.15.22:0/1748336292 conn(0x7f4a640fba80 msgr2=0x7f4a640fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.577+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 shutdown_connections 2024-03-24T06:39:09.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.577+0000 7f4a6a400640 1 -- 172.21.15.22:0/1748336292 wait complete. 2024-03-24T06:39:09.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:39:09.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 -- 172.21.15.22:0/2110745443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c94100750 msgr2=0x7f9c94100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/2110745443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c94100750 0x7f9c94100bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009900 tx=0x7f9c8002bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:09.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 -- 172.21.15.22:0/2110745443 shutdown_connections 2024-03-24T06:39:09.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/2110745443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 0x7f9c94101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/2110745443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c94100750 0x7f9c94100bd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/2110745443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c940ff9f0 0x7f9c940ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 -- 172.21.15.22:0/2110745443 >> 172.21.15.22:0/2110745443 conn(0x7f9c940fb300 msgr2=0x7f9c940fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 -- 172.21.15.22:0/2110745443 shutdown_connections 2024-03-24T06:39:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.814+0000 7f9c9bc00640 1 -- 172.21.15.22:0/2110745443 wait complete. 2024-03-24T06:39:09.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 Processor -- start 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 -- start start 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c94100750 0x7f9c94199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 0x7f9c941a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9c941a14e0 con 0x7f9c94101110 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9c941a1650 con 0x7f9c94100750 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.816+0000 7f9c9bc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c941a1810 con 0x7f9c940ff9f0 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55134/0 (socket says 172.21.15.22:55134) 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 -- 172.21.15.22:0/4190253066 learned_addr learned my addr 172.21.15.22:0/4190253066 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c93e00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c94100750 0x7f9c94199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99c00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 0x7f9c941a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 -- 172.21.15.22:0/4190253066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c94100750 msgr2=0x7f9c94199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c94100750 0x7f9c94199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 -- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 msgr2=0x7f9c941a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 0x7f9c941a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 -- 172.21.15.22:0/4190253066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c80009560 con 0x7f9c940ff9f0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.817+0000 7f9c99200640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800e9f0 tx=0x7f9c8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.818+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88009700 con 0x7f9c940ff9f0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.818+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c8800cd80 con 0x7f9c940ff9f0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.818+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88018920 con 0x7f9c940ff9f0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.818+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c941a1af0 con 0x7f9c940ff9f0 2024-03-24T06:39:09.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.818+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c941a1f80 con 0x7f9c940ff9f0 2024-03-24T06:39:09.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.819+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c9410b500 con 0x7f9c940ff9f0 2024-03-24T06:39:09.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.821+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9c88018a80 con 0x7f9c940ff9f0 2024-03-24T06:39:09.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.822+0000 7f9c91600640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 0x7f9c7c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:09.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.823+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f9c88014070 con 0x7f9c940ff9f0 2024-03-24T06:39:09.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.823+0000 7f9c93e00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 0x7f9c7c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:09.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.824+0000 7f9c93e00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 0x7f9c7c0a4580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c80005ab0 tx=0x7f9c80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:09.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:09.827+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c88053280 con 0x7f9c940ff9f0 2024-03-24T06:39:09.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:09 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1517288842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:09.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:09 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1748336292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:10.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.029+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9c9410f320 con 0x7f9c940ff9f0 2024-03-24T06:39:10.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.029+0000 7f9c91600640 1 -- 172.21.15.22:0/4190253066 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f9c88097090 con 0x7f9c940ff9f0 2024-03-24T06:39:10.028 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:10.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 msgr2=0x7f9c7c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 0x7f9c7c0a4580 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c80005ab0 tx=0x7f9c80005fd0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 msgr2=0x7f9c941999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800e9f0 tx=0x7f9c8800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 shutdown_connections 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f9c7c0a20c0 0x7f9c7c0a4580 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c94101110 0x7f9c941a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c94100750 0x7f9c94199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 --2- 172.21.15.22:0/4190253066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c940ff9f0 0x7f9c941999f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 >> 172.21.15.22:0/4190253066 conn(0x7f9c940fb300 msgr2=0x7f9c940fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 shutdown_connections 2024-03-24T06:39:10.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.032+0000 7f9c9bc00640 1 -- 172.21.15.22:0/4190253066 wait complete. 2024-03-24T06:39:10.051 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:39:10.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:09 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1517288842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:10.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:09 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1748336292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:09 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1517288842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:09 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1748336292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:10.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 -- 172.21.15.22:0/1334887504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 msgr2=0x7fde30101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 --2- 172.21.15.22:0/1334887504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde30101b40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde20009900 tx=0x7fde2002bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 -- 172.21.15.22:0/1334887504 shutdown_connections 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 --2- 172.21.15.22:0/1334887504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde301032d0 0x7fde30109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 --2- 172.21.15.22:0/1334887504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde30102910 0x7fde30102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 --2- 172.21.15.22:0/1334887504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde30101b40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 -- 172.21.15.22:0/1334887504 >> 172.21.15.22:0/1334887504 conn(0x7fde300fccb0 msgr2=0x7fde300ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 -- 172.21.15.22:0/1334887504 shutdown_connections 2024-03-24T06:39:10.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.279+0000 7fde35800640 1 -- 172.21.15.22:0/1334887504 wait complete. 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.280+0000 7fde35800640 1 Processor -- start 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.280+0000 7fde35800640 1 -- start start 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde30102910 0x7fde30073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 0x7fde30074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fde300747a0 con 0x7fde30102910 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fde30074910 con 0x7fde301032d0 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde35800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde301a18c0 con 0x7fde30101720 2024-03-24T06:39:10.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde2ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde2ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55148/0 (socket says 172.21.15.22:55148) 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde2f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 0x7fde30074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde2ea00640 1 -- 172.21.15.22:0/651470670 learned_addr learned my addr 172.21.15.22:0/651470670 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.281+0000 7fde2e000640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde30102910 0x7fde30073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 -- 172.21.15.22:0/651470670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 msgr2=0x7fde30074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 0x7fde30074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 -- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde30102910 msgr2=0x7fde30073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde30102910 0x7fde30073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 -- 172.21.15.22:0/651470670 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde20009560 con 0x7fde30101720 2024-03-24T06:39:10.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2f400640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 0x7fde30074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde2ea00640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fde20000c00 tx=0x7fde200040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde200044c0 con 0x7fde30101720 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde200052d0 con 0x7fde30101720 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde301a1b10 con 0x7fde30101720 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.282+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2003e450 con 0x7fde30101720 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.283+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde301a2020 con 0x7fde30101720 2024-03-24T06:39:10.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.284+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde30108ac0 con 0x7fde30101720 2024-03-24T06:39:10.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.287+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fde20005440 con 0x7fde30101720 2024-03-24T06:39:10.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.289+0000 7fde27400640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 0x7fde140a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.289+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fde200f0890 con 0x7fde30101720 2024-03-24T06:39:10.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.289+0000 7fde2e000640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 0x7fde140a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.290+0000 7fde2e000640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 0x7fde140a4580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde30075230 tx=0x7fde1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:10.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.297+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fde200b9480 con 0x7fde30101720 2024-03-24T06:39:10.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.526+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde3002d160 con 0x7fde30101720 2024-03-24T06:39:10.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.527+0000 7fde27400640 1 -- 172.21.15.22:0/651470670 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fde200430a0 con 0x7fde30101720 2024-03-24T06:39:10.525 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:39:10.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 msgr2=0x7fde140a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 0x7fde140a4580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde30075230 tx=0x7fde1c005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 msgr2=0x7fde300754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fde20000c00 tx=0x7fde200040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 shutdown_connections 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fde140a20c0 0x7fde140a4580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde301032d0 0x7fde30074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde30102910 0x7fde30073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 --2- 172.21.15.22:0/651470670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30101720 0x7fde300754c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 >> 172.21.15.22:0/651470670 conn(0x7fde300fccb0 msgr2=0x7fde30105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 shutdown_connections 2024-03-24T06:39:10.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.529+0000 7fde35800640 1 -- 172.21.15.22:0/651470670 wait complete. 2024-03-24T06:39:10.549 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:39:10.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 -- 172.21.15.22:0/1373329426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0481020d0 msgr2=0x7fd0481044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 --2- 172.21.15.22:0/1373329426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0481020d0 0x7fd0481044c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009900 tx=0x7fd03c02bc50 comp rx=0 tx=0).stop 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 -- 172.21.15.22:0/1373329426 shutdown_connections 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 --2- 172.21.15.22:0/1373329426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd048109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 --2- 172.21.15.22:0/1373329426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd048104a00 0x7fd048106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 --2- 172.21.15.22:0/1373329426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0481020d0 0x7fd0481044c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 -- 172.21.15.22:0/1373329426 >> 172.21.15.22:0/1373329426 conn(0x7fd0480fba80 msgr2=0x7fd0480fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 -- 172.21.15.22:0/1373329426 shutdown_connections 2024-03-24T06:39:10.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.775+0000 7fd04de00640 1 -- 172.21.15.22:0/1373329426 wait complete. 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.776+0000 7fd04de00640 1 Processor -- start 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.776+0000 7fd04de00640 1 -- start start 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 0x7fd048101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 0x7fd048100080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd0481005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd048100d20 con 0x7fd048107330 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd048100e90 con 0x7fd0481020d0 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04de00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0481a16d0 con 0x7fd048104a00 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 0x7fd048101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd047e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 0x7fd048100080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd047e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 0x7fd048100080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55160/0 (socket says 172.21.15.22:55160) 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd047e00640 1 -- 172.21.15.22:0/3938861962 learned_addr learned my addr 172.21.15.22:0/3938861962 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04d400640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd0481005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 -- 172.21.15.22:0/3938861962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 msgr2=0x7fd048100080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 0x7fd048100080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 -- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 msgr2=0x7fd0481005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd0481005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04ca00640 1 -- 172.21.15.22:0/3938861962 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd034009490 con 0x7fd0481020d0 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.777+0000 7fd04d400640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd0481005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:10.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd04ca00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 0x7fd048101a40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02c190 tx=0x7fd03c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:10.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0043d0 con 0x7fd0481020d0 2024-03-24T06:39:10.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03c0052d0 con 0x7fd0481020d0 2024-03-24T06:39:10.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03c009560 con 0x7fd0481020d0 2024-03-24T06:39:10.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003cd0 con 0x7fd0481020d0 2024-03-24T06:39:10.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.778+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0481a1ca0 con 0x7fd0481020d0 2024-03-24T06:39:10.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.779+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0480fd570 con 0x7fd0481020d0 2024-03-24T06:39:10.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.781+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd03c005440 con 0x7fd0481020d0 2024-03-24T06:39:10.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.783+0000 7fd045600640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 0x7fd0300a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:10.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.783+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fd03c0f0320 con 0x7fd0481020d0 2024-03-24T06:39:10.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.783+0000 7fd047e00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 0x7fd0300a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:10.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.784+0000 7fd047e00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 0x7fd0300a4580 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0481017b0 tx=0x7fd034005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:10.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.787+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd03c0b8f10 con 0x7fd0481020d0 2024-03-24T06:39:10.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.990+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd04802d130 con 0x7fd0481020d0 2024-03-24T06:39:10.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.991+0000 7fd045600640 1 -- 172.21.15.22:0/3938861962 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd03c0bc5d0 con 0x7fd0481020d0 2024-03-24T06:39:10.990 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:10.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.993+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 msgr2=0x7fd0300a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.993+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 0x7fd0300a4580 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd0481017b0 tx=0x7fd034005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:10.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.993+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 msgr2=0x7fd048101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:10.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.993+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 0x7fd048101a40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02c190 tx=0x7fd03c005e80 comp rx=0 tx=0).stop 2024-03-24T06:39:10.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 shutdown_connections 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fd0300a20c0 0x7fd0300a4580 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd048107330 0x7fd0481005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd048104a00 0x7fd048100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 --2- 172.21.15.22:0/3938861962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0481020d0 0x7fd048101a40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 >> 172.21.15.22:0/3938861962 conn(0x7fd0480fba80 msgr2=0x7fd048102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 shutdown_connections 2024-03-24T06:39:10.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:10.994+0000 7fd04de00640 1 -- 172.21.15.22:0/3938861962 wait complete. 2024-03-24T06:39:11.012 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:39:11.098 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:10 smithi022 ceph-mon[32003]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T06:39:11.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4190253066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.099 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/651470670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:10 smithi143 ceph-mon[36520]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T06:39:11.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4190253066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/651470670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 -- 172.21.15.22:0/1071214009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf480760d0 msgr2=0x7fcf480764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 --2- 172.21.15.22:0/1071214009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf480760d0 0x7fcf480764f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcf38009ca0 tx=0x7fcf3802c100 comp rx=0 tx=0).stop 2024-03-24T06:39:11.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 -- 172.21.15.22:0/1071214009 shutdown_connections 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 --2- 172.21.15.22:0/1071214009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf48077cb0 0x7fcf48078550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 --2- 172.21.15.22:0/1071214009 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077310 0x7fcf48077770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 --2- 172.21.15.22:0/1071214009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf480760d0 0x7fcf480764f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 -- 172.21.15.22:0/1071214009 >> 172.21.15.22:0/1071214009 conn(0x7fcf480feec0 msgr2=0x7fcf48101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 -- 172.21.15.22:0/1071214009 shutdown_connections 2024-03-24T06:39:11.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.241+0000 7fcf4e800640 1 -- 172.21.15.22:0/1071214009 wait complete. 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.242+0000 7fcf4e800640 1 Processor -- start 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.242+0000 7fcf4e800640 1 -- start start 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 0x7fcf4819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077cb0 0x7fcf481a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcf481a58a0 con 0x7fcf48077cb0 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcf481a5a10 con 0x7fcf480760d0 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf4e800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcf481a5bd0 con 0x7fcf48077310 2024-03-24T06:39:11.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf3ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077cb0 0x7fcf481a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf47400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 0x7fcf4819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf3ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55190/0 (socket says 172.21.15.22:55190) 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.243+0000 7fcf3ea00640 1 -- 172.21.15.22:0/4030067231 learned_addr learned my addr 172.21.15.22:0/4030067231 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 -- 172.21.15.22:0/4030067231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 msgr2=0x7fcf4819e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 -- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077cb0 msgr2=0x7fcf481a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077cb0 0x7fcf481a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 -- 172.21.15.22:0/4030067231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf38009900 con 0x7fcf480760d0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf3ea00640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf47400640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 0x7fcf4819ddc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcf38032600 tx=0x7fcf38034bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf38034850 con 0x7fcf480760d0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf38003e50 con 0x7fcf480760d0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf38040450 con 0x7fcf480760d0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.244+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf481a5e50 con 0x7fcf480760d0 2024-03-24T06:39:11.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.245+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf481a61c0 con 0x7fcf480760d0 2024-03-24T06:39:11.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.246+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf48110550 con 0x7fcf480760d0 2024-03-24T06:39:11.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.249+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fcf3800a040 con 0x7fcf480760d0 2024-03-24T06:39:11.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.250+0000 7fcf44c00640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 0x7fcf2c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.251+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fcf380f0b90 con 0x7fcf480760d0 2024-03-24T06:39:11.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.251+0000 7fcf3ea00640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 0x7fcf2c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.252+0000 7fcf3ea00640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 0x7fcf2c0a4580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcf30011550 tx=0x7fcf30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:11.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.254+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf380b96f0 con 0x7fcf480760d0 2024-03-24T06:39:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:10 smithi155 ceph-mon[36262]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 270 B/s wr, 12 op/s 2024-03-24T06:39:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4190253066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/651470670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:11.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.501+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf48113710 con 0x7fcf480760d0 2024-03-24T06:39:11.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.502+0000 7fcf44c00640 1 -- 172.21.15.22:0/4030067231 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7fcf380bb620 con 0x7fcf480760d0 2024-03-24T06:39:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:11.501 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:39:11.502 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 msgr2=0x7fcf2c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 0x7fcf2c0a4580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcf30011550 tx=0x7fcf30005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 msgr2=0x7fcf4819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 0x7fcf4819ddc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcf38032600 tx=0x7fcf38034bf0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 shutdown_connections 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf2c0a20c0 0x7fcf2c0a4580 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf48077cb0 0x7fcf481a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf48077310 0x7fcf4819e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 --2- 172.21.15.22:0/4030067231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf480760d0 0x7fcf4819ddc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 >> 172.21.15.22:0/4030067231 conn(0x7fcf480feec0 msgr2=0x7fcf48100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 shutdown_connections 2024-03-24T06:39:11.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.505+0000 7fcf4e800640 1 -- 172.21.15.22:0/4030067231 wait complete. 2024-03-24T06:39:11.528 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:11.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 -- 172.21.15.22:0/2576578573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 msgr2=0x7f095c073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 --2- 172.21.15.22:0/2576578573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c073ba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f094c009930 tx=0x7f094c02bc20 comp rx=0 tx=0).stop 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 -- 172.21.15.22:0/2576578573 shutdown_connections 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 --2- 172.21.15.22:0/2576578573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c10db50 0x7f095c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 --2- 172.21.15.22:0/2576578573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c074170 0x7f095c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 --2- 172.21.15.22:0/2576578573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c073ba0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.772+0000 7f095b200640 1 -- 172.21.15.22:0/2576578573 >> 172.21.15.22:0/2576578573 conn(0x7f095c0fcaf0 msgr2=0x7f095c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.773+0000 7f095b200640 1 -- 172.21.15.22:0/2576578573 shutdown_connections 2024-03-24T06:39:11.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.773+0000 7f095b200640 1 -- 172.21.15.22:0/2576578573 wait complete. 2024-03-24T06:39:11.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.773+0000 7f095b200640 1 Processor -- start 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 -- start start 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c10db50 0x7f095c1a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a5740 con 0x7f095c10db50 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a58b0 con 0x7f095c075750 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f095c1a5a70 con 0x7f095c074170 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f0959e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f0959400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f0959e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55208/0 (socket says 172.21.15.22:55208) 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f0959400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c19e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:45980/0 (socket says 172.21.15.22:45980) 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f0959e00640 1 -- 172.21.15.22:0/1123663702 learned_addr learned my addr 172.21.15.22:0/1123663702 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:11.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.774+0000 7f095a800640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c10db50 0x7f095c1a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 -- 172.21.15.22:0/1123663702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 msgr2=0x7f095c19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 -- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c10db50 msgr2=0x7f095c1a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c10db50 0x7f095c1a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 -- 172.21.15.22:0/1123663702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094c009560 con 0x7f095c074170 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0959e00640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f094c000c00 tx=0x7f094c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c02dea0 con 0x7f095c074170 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f094c0043d0 con 0x7f095c074170 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c1a5cf0 con 0x7f095c074170 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.775+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c003c20 con 0x7f095c074170 2024-03-24T06:39:11.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.776+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c1a61a0 con 0x7f095c074170 2024-03-24T06:39:11.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.777+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c103d20 con 0x7f095c074170 2024-03-24T06:39:11.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.779+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f094c004df0 con 0x7f095c074170 2024-03-24T06:39:11.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.780+0000 7f0952a00640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 0x7f09400a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:11.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.780+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f094c0f0690 con 0x7f095c074170 2024-03-24T06:39:11.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.780+0000 7f0959400640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 0x7f09400a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:11.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.781+0000 7f0959400640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 0x7f09400a4600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0948009bd0 tx=0x7f0948005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:11.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.784+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f094c0b9230 con 0x7f095c074170 2024-03-24T06:39:11.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.994+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f095c02d2f0 con 0x7f095c074170 2024-03-24T06:39:11.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:11.997+0000 7f0952a00640 1 -- 172.21.15.22:0/1123663702 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7f094c0bb160 con 0x7f095c074170 2024-03-24T06:39:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:11.997 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 msgr2=0x7f09400a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 0x7f09400a4600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0948009bd0 tx=0x7f0948005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 msgr2=0x7f095c19dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f094c000c00 tx=0x7f094c005e80 comp rx=0 tx=0).stop 2024-03-24T06:39:11.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 shutdown_connections 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f09400a2140 0x7f09400a4600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f095c10db50 0x7f095c1a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f095c075750 0x7f095c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 --2- 172.21.15.22:0/1123663702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f095c074170 0x7f095c19dc60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 >> 172.21.15.22:0/1123663702 conn(0x7f095c0fcaf0 msgr2=0x7f095c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 shutdown_connections 2024-03-24T06:39:12.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.000+0000 7f095b200640 1 -- 172.21.15.22:0/1123663702 wait complete. 2024-03-24T06:39:12.018 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:39:12.089 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:11 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3938861962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:12.089 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:11 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4030067231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:12.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:11 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3938861962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:12.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:11 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4030067231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:12.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:11 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3938861962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:12.178 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:11 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4030067231' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 -- 172.21.15.22:0/3370403126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0103760 msgr2=0x7f4bf0103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/3370403126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0103760 0x7f4bf0103b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009ca0 tx=0x7f4be002c100 comp rx=0 tx=0).stop 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 -- 172.21.15.22:0/3370403126 shutdown_connections 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/3370403126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0105310 0x7f4bf010bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/3370403126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0104950 0x7f4bf0104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/3370403126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0103760 0x7f4bf0103b80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 -- 172.21.15.22:0/3370403126 >> 172.21.15.22:0/3370403126 conn(0x7f4bf00fecf0 msgr2=0x7f4bf0101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 -- 172.21.15.22:0/3370403126 shutdown_connections 2024-03-24T06:39:12.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.225+0000 7f4bf5c00640 1 -- 172.21.15.22:0/3370403126 wait complete. 2024-03-24T06:39:12.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 Processor -- start 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 -- start start 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0103760 0x7f4bf01997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0105310 0x7f4bf01a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf01a12d0 con 0x7f4bf0105310 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf01a1440 con 0x7f4bf0104950 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.226+0000 7f4bf5c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf01a1600 con 0x7f4bf0103760 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4befe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0103760 0x7f4bf01997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:45992/0 (socket says 172.21.15.22:45992) 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 -- 172.21.15.22:0/938807972 learned_addr learned my addr 172.21.15.22:0/938807972 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bf5200640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0105310 0x7f4bf01a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 -- 172.21.15.22:0/938807972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0103760 msgr2=0x7f4bf01997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0103760 0x7f4bf01997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 -- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0105310 msgr2=0x7f4bf01a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0105310 0x7f4bf01a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 -- 172.21.15.22:0/938807972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be0009900 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.227+0000 7f4bef400640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800d950 tx=0x7f4bd800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.228+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8009c30 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.228+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd8009d90 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.228+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800f7e0 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.228+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf01a18e0 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.228+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf01a1e40 con 0x7f4bf0104950 2024-03-24T06:39:12.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.229+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf010ab00 con 0x7f4bf0104950 2024-03-24T06:39:12.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.231+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f4bd8010ad0 con 0x7f4bf0104950 2024-03-24T06:39:12.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.232+0000 7f4becc00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 0x7f4bd40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.232+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f4bd80cdb50 con 0x7f4bf0104950 2024-03-24T06:39:12.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.232+0000 7f4befe00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 0x7f4bd40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.233+0000 7f4befe00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 0x7f4bd40a4600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4be002c640 tx=0x7f4be0009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.236+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4bd8096850 con 0x7f4bf0104950 2024-03-24T06:39:12.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.478+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bf002d1d0 con 0x7f4bf0104950 2024-03-24T06:39:12.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.479+0000 7f4becc00640 1 -- 172.21.15.22:0/938807972 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7f4bd8099f10 con 0x7f4bf0104950 2024-03-24T06:39:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:12.478 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:39:12.478 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:12.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 msgr2=0x7f4bd40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 0x7f4bd40a4600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4be002c640 tx=0x7f4be0009800 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 msgr2=0x7f4bf0199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800d950 tx=0x7f4bd800de10 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 shutdown_connections 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4bd40a2140 0x7f4bd40a4600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4bf0105310 0x7f4bf01a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bf0104950 0x7f4bf0199d20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 --2- 172.21.15.22:0/938807972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bf0103760 0x7f4bf01997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 >> 172.21.15.22:0/938807972 conn(0x7f4bf00fecf0 msgr2=0x7f4bf0107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 shutdown_connections 2024-03-24T06:39:12.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.481+0000 7f4bf5c00640 1 -- 172.21.15.22:0/938807972 wait complete. 2024-03-24T06:39:12.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T06:36:56.216020+0000', 'modified': '2024-03-24T06:37:33.427816+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24601, 'mds_1': 24584, 'mds_2': 24565, 'mds_3': 24544, 'mds_4': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6833/3570497194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 3570497194}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 3570497194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6833/715907814', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6832', 'nonce': 715907814}, {'type': 'v1', 'addr': '172.21.15.155:6833', 'nonce': 715907814}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.22:6835/298241489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6834', 'nonce': 298241489}, {'type': 'v1', 'addr': '172.21.15.22:6835', 'nonce': 298241489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6835/1050031463', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1050031463}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1050031463}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/988781970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 988781970}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 988781970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.155:6835/1202752642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 1202752642}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 1202752642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.155:6837/1157072254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1157072254}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1157072254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24565': {'gid': 24565, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.155:6839/2828760693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 2828760693}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 2828760693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24584': {'gid': 24584, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/2325215622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 2325215622}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 2325215622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24601': {'gid': 24601, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.22:6841/857116957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6840', 'nonce': 857116957}, {'type': 'v1', 'addr': '172.21.15.22:6841', 'nonce': 857116957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24601, 'qdb_cluster': [24584, 24601, 24532, 24544, 24565]} 2024-03-24T06:39:12.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T06:39:12.501 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T06:39:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 -- 172.21.15.22:0/237713940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 msgr2=0x7f4b5c073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 --2- 172.21.15.22:0/237713940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c073ba0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c009ca0 tx=0x7f4b4c02c100 comp rx=0 tx=0).stop 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 -- 172.21.15.22:0/237713940 shutdown_connections 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 --2- 172.21.15.22:0/237713940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c1094d0 0x7f4b5c10b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 --2- 172.21.15.22:0/237713940 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c074170 0x7f4b5c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.714+0000 7f4b5be00640 1 --2- 172.21.15.22:0/237713940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c073ba0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.715+0000 7f4b5be00640 1 -- 172.21.15.22:0/237713940 >> 172.21.15.22:0/237713940 conn(0x7f4b5c0fcb80 msgr2=0x7f4b5c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.715+0000 7f4b5be00640 1 -- 172.21.15.22:0/237713940 shutdown_connections 2024-03-24T06:39:12.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.715+0000 7f4b5be00640 1 -- 172.21.15.22:0/237713940 wait complete. 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 Processor -- start 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 -- start start 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c1094d0 0x7f4b5c1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c1a1350 con 0x7f4b5c1094d0 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c1a14c0 con 0x7f4b5c075750 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.716+0000 7f4b5be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c1a1680 con 0x7f4b5c074170 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55240/0 (socket says 172.21.15.22:55240) 2024-03-24T06:39:12.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b52000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 -- 172.21.15.22:0/1434299135 learned_addr learned my addr 172.21.15.22:0/1434299135 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5b400640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c1094d0 0x7f4b5c1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 -- 172.21.15.22:0/1434299135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 msgr2=0x7f4b5c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 -- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c1094d0 msgr2=0x7f4b5c1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c1094d0 0x7f4b5c1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 -- 172.21.15.22:0/1434299135 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c009900 con 0x7f4b5c074170 2024-03-24T06:39:12.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.717+0000 7f4b5aa00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c032730 tx=0x7f4b4c034e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.718+0000 7f4b53e00640 1 -- 172.21.15.22:0/1434299135 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c034850 con 0x7f4b5c074170 2024-03-24T06:39:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.718+0000 7f4b53e00640 1 -- 172.21.15.22:0/1434299135 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b4c003e50 con 0x7f4b5c074170 2024-03-24T06:39:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.718+0000 7f4b53e00640 1 -- 172.21.15.22:0/1434299135 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c0406b0 con 0x7f4b5c074170 2024-03-24T06:39:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.718+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5c1a1900 con 0x7f4b5c074170 2024-03-24T06:39:12.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.718+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b5c1a1d30 con 0x7f4b5c074170 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b53e00640 1 -- 172.21.15.22:0/1434299135 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f4b4c00a040 con 0x7f4b5c074170 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 Processor -- start 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 -- start start 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a88778d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a8876f4050 con 0x55a887786110 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a8876f1580 con 0x55a887785720 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.721+0000 7f4b64322740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a8876ecd60 con 0x55a88778b280 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46034/0 (socket says 172.21.15.22:46034) 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 -- 172.21.15.22:0/2941850484 learned_addr learned my addr 172.21.15.22:0/2941850484 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b52000640 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a88778d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 -- 172.21.15.22:0/2941850484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 msgr2=0x55a88778d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a88778d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 -- 172.21.15.22:0/2941850484 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 msgr2=0x55a88778ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 -- 172.21.15.22:0/2941850484 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8876a5620 con 0x55a887785720 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.722+0000 7f4b5b400640 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009c10 tx=0x7f4b540117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e34d6133cd1f2ead server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b4be00640 1 -- 172.21.15.22:0/2941850484 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54009760 con 0x55a887785720 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b4be00640 1 -- 172.21.15.22:0/2941850484 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b54012930 con 0x55a887785720 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b4be00640 1 -- 172.21.15.22:0/2941850484 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400fa90 con 0x55a887785720 2024-03-24T06:39:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b53e00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 0x7f4b400a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b53e00640 1 -- 172.21.15.22:0/1434299135 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f4b4c0f0b10 con 0x7f4b5c074170 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b5aa00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 0x7f4b400a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b64322740 1 -- 172.21.15.22:0/2941850484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 msgr2=0x55a887785b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.723+0000 7f4b64322740 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009c10 tx=0x7f4b540117b0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 -- 172.21.15.22:0/2941850484 shutdown_connections 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a88778d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 --2- 172.21.15.22:0/2941850484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887785b40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 -- 172.21.15.22:0/2941850484 >> 172.21.15.22:0/2941850484 conn(0x55a887780c90 msgr2=0x55a887781090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 -- 172.21.15.22:0/2941850484 shutdown_connections 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 -- 172.21.15.22:0/2941850484 wait complete. 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b5aa00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 0x7f4b400a4600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0bd1f0 tx=0x7f4b4c0f5040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 -- start start 2024-03-24T06:39:12.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.724+0000 7f4b64322740 1 Processor -- start 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a88778f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a887796f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a887691fb0 con 0x55a887786110 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a887691c50 con 0x55a88778b280 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a887434f20 con 0x55a887785720 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b64322740 10 client.0 _ll_register_callbacks cb 0x55a8873fdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a88778f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b52000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a88778f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55258/0 (socket says 172.21.15.22:55258) 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b5b400640 1 -- 172.21.15.22:0/93908409 learned_addr learned my addr 172.21.15.22:0/93908409 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.725+0000 7f4b5b400640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a887796f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 -- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 msgr2=0x55a88778f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a88778f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 -- 172.21.15.22:0/93908409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 msgr2=0x55a887796f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a887796f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 -- 172.21.15.22:0/93908409 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b540098c0 con 0x55a887786110 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b5b400640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a887796f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b52000640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778fee0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c19ad70 tx=0x7f4b4400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8256dde620ab27b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b48c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:39:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44016070 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f4b44003dc0 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.726+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a8873ec6c0 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.727+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b44005020 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.727+0000 7f4b64322740 10 client.15036 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.727+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a8876eb3e0 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.727+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a8873ec6c0 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.727+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b4401b470 con 0x55a887786110 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 10 client.15036 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 10 client.15036 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 20 client.15036 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a88779c630 0x55a88779daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 4 client.15036 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a88743d270 con 0x55a88779c630 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.728+0000 7f4b5b400640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a88779c630 0x55a88779daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.729+0000 7f4b5b400640 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a88779c630 0x55a88779daa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1b272827e2de7111 server_cookie=41a3b39c3e3c0905 in_seq=0 out_seq=0 2024-03-24T06:39:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.729+0000 7f4b48c00640 10 client.15036 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.729+0000 7f4b48c00640 1 -- 172.21.15.22:0/93908409 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a88743d270 con 0x55a88779c630 2024-03-24T06:39:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.729+0000 7f4b48c00640 10 client.15036 handle_command_reply: tid=0 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.729+0000 7f4b64322740 1 client.15036 shutdown 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 msgr2=0x55a88778fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778fee0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c19ad70 tx=0x7f4b4400bf30 comp rx=0 tx=0).stop 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 shutdown_connections 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a88779c630 0x55a88779daa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a887796f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a88778fee0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 --2- 172.21.15.22:0/93908409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a88778f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 >> 172.21.15.22:0/93908409 conn(0x55a887780c90 msgr2=0x55a88777fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b48c00640 0 client.15036 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 shutdown_connections 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 1 -- 172.21.15.22:0/93908409 wait complete. 2024-03-24T06:39:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.730+0000 7f4b64322740 20 client.15036 trim_cache size 0 max 16384 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 Processor -- start 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 -- start start 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a8877a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a88778af10 con 0x55a887786110 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a88779ca10 con 0x55a88778b280 2024-03-24T06:39:12.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b64322740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a88743d270 con 0x55a887785720 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.750+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55268/0 (socket says 172.21.15.22:55268) 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 -- 172.21.15.22:0/3900686665 learned_addr learned my addr 172.21.15.22:0/3900686665 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b52000640 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a8877a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 -- 172.21.15.22:0/3900686665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 msgr2=0x55a8877a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a8877a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 -- 172.21.15.22:0/3900686665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 msgr2=0x55a887796ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 -- 172.21.15.22:0/3900686665 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a887687900 con 0x55a887785720 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b5b400640 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009be0 tx=0x7f4b54009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d782576afe6c737a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b4b400640 1 -- 172.21.15.22:0/3900686665 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54012aa0 con 0x55a887785720 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b4b400640 1 -- 172.21.15.22:0/3900686665 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b54035020 con 0x55a887785720 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.751+0000 7f4b4b400640 1 -- 172.21.15.22:0/3900686665 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5402e4b0 con 0x55a887785720 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 -- 172.21.15.22:0/3900686665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 msgr2=0x55a887781070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4b54009be0 tx=0x7f4b54009800 comp rx=0 tx=0).stop 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 -- 172.21.15.22:0/3900686665 shutdown_connections 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a88778b280 0x55a8877a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 --2- 172.21.15.22:0/3900686665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a887785720 0x55a887781070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 -- 172.21.15.22:0/3900686665 >> 172.21.15.22:0/3900686665 conn(0x55a887780c90 msgr2=0x55a887786bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 -- 172.21.15.22:0/3900686665 shutdown_connections 2024-03-24T06:39:12.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.752+0000 7f4b64322740 1 -- 172.21.15.22:0/3900686665 wait complete. 2024-03-24T06:39:12.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 -- start start 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 Processor -- start 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887796880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a887794e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a887795560 con 0x55a887786110 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a8877956d0 con 0x55a887785720 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a887795890 con 0x55a88778b280 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b64322740 10 client.0 _ll_register_callbacks cb 0x7f4b44004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887796880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.753+0000 7f4b5b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887796880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46058/0 (socket says 172.21.15.22:46058) 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b5b400640 1 -- 172.21.15.22:0/1421844581 learned_addr learned my addr 172.21.15.22:0/1421844581 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b5b400640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a887794e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 -- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 msgr2=0x55a887794e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a887794e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 -- 172.21.15.22:0/1421844581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 msgr2=0x55a887796880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887796880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 -- 172.21.15.22:0/1421844581 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8876a5620 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b52000640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796dc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b44000c00 tx=0x7f4b44005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda870b804629383 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b48c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.754+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44016040 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f4b44000e40 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b54012930 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b44003ec0 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 10 client.15042 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a8876ad560 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b54012930 con 0x55a887786110 2024-03-24T06:39:12.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b44004670 con 0x55a887786110 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 10 client.15042 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 10 client.15042 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 20 client.15042 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a887807640 0x55a887806350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 4 client.15042 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.755+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "status"}) -- 0x55a887429610 con 0x55a887807640 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b5b400640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a887807640 0x55a887806350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b5b400640 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a887807640 0x55a887806350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46df7612d604e824 server_cookie=f7fc604b543e48d5 in_seq=0 out_seq=0 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b48c00640 10 client.15042 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b48c00640 1 -- 172.21.15.22:0/1421844581 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a887429610 con 0x55a887807640 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b48c00640 10 client.15042 handle_command_reply: tid=0 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.756+0000 7f4b64322740 1 client.15042 shutdown 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 msgr2=0x55a887796dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796dc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b44000c00 tx=0x7f4b44005cc0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 shutdown_connections 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55a887807640 0x55a887806350 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a88778b280 0x55a887794e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a887786110 0x55a887796dc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 --2- 172.21.15.22:0/1421844581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a887785720 0x55a887796880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 >> 172.21.15.22:0/1421844581 conn(0x55a887780c90 msgr2=0x55a887781070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b48c00640 0 client.15042 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 shutdown_connections 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 1 -- 172.21.15.22:0/1421844581 wait complete. 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.757+0000 7f4b64322740 20 client.15042 trim_cache size 0 max 16384 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:12.759 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 4, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24532, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 100.34931269499999, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 162.962947192 2024-03-24T06:39:12.760 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 msgr2=0x7f4b400a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 0x7f4b400a4600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0bd1f0 tx=0x7f4b4c0f5040 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 msgr2=0x7f4b5c199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c032730 tx=0x7f4b4c034e20 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 shutdown_connections 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f4b400a2140 0x7f4b400a4600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b5c1094d0 0x7f4b5c1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b5c075750 0x7f4b5c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 --2- 172.21.15.22:0/1434299135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b5c074170 0x7f4b5c199860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 >> 172.21.15.22:0/1434299135 conn(0x7f4b5c0fcb80 msgr2=0x7f4b5c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 shutdown_connections 2024-03-24T06:39:12.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.758+0000 7f4b5be00640 1 -- 172.21.15.22:0/1434299135 wait complete. 2024-03-24T06:39:12.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T06:39:12.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 -- 172.21.15.22:0/1907281316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424100610 msgr2=0x7f5424100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 --2- 172.21.15.22:0/1907281316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424100610 0x7f5424100a30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5414009900 tx=0x7f541402bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 -- 172.21.15.22:0/1907281316 shutdown_connections 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 --2- 172.21.15.22:0/1907281316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f5424108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 --2- 172.21.15.22:0/1907281316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424101800 0x7f5424101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 --2- 172.21.15.22:0/1907281316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424100610 0x7f5424100a30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.986+0000 7f5429800640 1 -- 172.21.15.22:0/1907281316 >> 172.21.15.22:0/1907281316 conn(0x7f54240fbba0 msgr2=0x7f54240fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.987+0000 7f5429800640 1 -- 172.21.15.22:0/1907281316 shutdown_connections 2024-03-24T06:39:12.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.987+0000 7f5429800640 1 -- 172.21.15.22:0/1907281316 wait complete. 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 Processor -- start 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 -- start start 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424101800 0x7f5424199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f54241a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.988+0000 7f5429800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f54241a1520 con 0x7f54241021c0 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5429800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f54241a1690 con 0x7f5424101800 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5429800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f54241a1850 con 0x7f5424100610 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55294/0 (socket says 172.21.15.22:55294) 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 -- 172.21.15.22:0/3213844833 learned_addr learned my addr 172.21.15.22:0/3213844833 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5423400640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f54241a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422000640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424101800 0x7f5424199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 -- 172.21.15.22:0/3213844833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424101800 msgr2=0x7f5424199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424101800 0x7f5424199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 -- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 msgr2=0x7f54241a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f54241a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5422a00640 1 -- 172.21.15.22:0/3213844833 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5410009490 con 0x7f5424100610 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.989+0000 7f5423400640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f54241a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f5422a00640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5414000c00 tx=0x7f54140040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f541b400640 1 -- 172.21.15.22:0/3213844833 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140044c0 con 0x7f5424100610 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f541b400640 1 -- 172.21.15.22:0/3213844833 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54140052d0 con 0x7f5424100610 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5414009560 con 0x7f5424100610 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f541b400640 1 -- 172.21.15.22:0/3213844833 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414003e80 con 0x7f5424100610 2024-03-24T06:39:12.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.990+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54241a1e50 con 0x7f5424100610 2024-03-24T06:39:12.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.994+0000 7f542c5bc740 1 Processor -- start 2024-03-24T06:39:12.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.994+0000 7f542c5bc740 1 -- start start 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f541b400640 1 -- 172.21.15.22:0/3213844833 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5414005440 con 0x7f5424100610 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37f40 0x556d6cd3ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd3f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556d6cca6050 con 0x556d6cd37f40 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556d6cca3580 con 0x556d6cd3d0b0 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f542c5bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556d6cc9ed60 con 0x556d6cd37550 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f5423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f5422a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37f40 0x556d6cd3ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f5423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55304/0 (socket says 172.21.15.22:55304) 2024-03-24T06:39:12.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f5423400640 1 -- 172.21.15.22:0/272106663 learned_addr learned my addr 172.21.15.22:0/272106663 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.995+0000 7f5422000640 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd3f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 -- 172.21.15.22:0/272106663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 msgr2=0x556d6cd3f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd3f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 -- 172.21.15.22:0/272106663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37f40 msgr2=0x556d6cd3ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37f40 0x556d6cd3ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 -- 172.21.15.22:0/272106663 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6cc57620 con 0x556d6cd37550 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f5423400640 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fb0 tx=0x7f541c010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40ca65a5cc865b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f540fe00640 1 -- 172.21.15.22:0/272106663 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c012710 con 0x556d6cd37550 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f540fe00640 1 -- 172.21.15.22:0/272106663 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c012d50 con 0x556d6cd37550 2024-03-24T06:39:12.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.996+0000 7f540fe00640 1 -- 172.21.15.22:0/272106663 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c01bea0 con 0x556d6cd37550 2024-03-24T06:39:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f541b400640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 0x7f54080a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f541b400640 1 -- 172.21.15.22:0/3213844833 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f54140f0aa0 con 0x7f5424100610 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f5422a00640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 0x7f54080a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 -- 172.21.15.22:0/272106663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 msgr2=0x556d6cd37970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fb0 tx=0x7f541c010800 comp rx=0 tx=0).stop 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 -- 172.21.15.22:0/272106663 shutdown_connections 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd3f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37f40 0x556d6cd3ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 --2- 172.21.15.22:0/272106663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37550 0x556d6cd37970 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 -- 172.21.15.22:0/272106663 >> 172.21.15.22:0/272106663 conn(0x556d6cd32ac0 msgr2=0x556d6cd32ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 -- 172.21.15.22:0/272106663 shutdown_connections 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.997+0000 7f542c5bc740 1 -- 172.21.15.22:0/272106663 wait complete. 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.998+0000 7f5422a00640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 0x7f54080a4580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x556d6cd38340 tx=0x7f54140f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.998+0000 7f542c5bc740 1 -- start start 2024-03-24T06:39:13.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.998+0000 7f542c5bc740 1 Processor -- start 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd41ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd48d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556d6cc43fb0 con 0x556d6cd37550 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556d6cc43c50 con 0x556d6cd37f40 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556d6c9e6f20 con 0x556d6cd3d0b0 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f542c5bc740 10 client.0 _ll_register_callbacks cb 0x556d6c9afac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f5423400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f5423400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39158/0 (socket says 172.21.15.22:39158) 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f5423400640 1 -- 172.21.15.22:0/2697925983 learned_addr learned my addr 172.21.15.22:0/2697925983 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f5422000640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd41ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:12.999+0000 7f5423400640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd48d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 -- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 msgr2=0x556d6cd48d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd48d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 -- 172.21.15.22:0/2697925983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 msgr2=0x556d6cd417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 -- 172.21.15.22:0/2697925983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c009be0 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f5422000640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd41ce0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f542419af40 tx=0x7f5410012ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19292c4a5791487f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f540cc00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410017070 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f541000f920 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.000+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556d6c99e6c0 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f541000fae0 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 10 client.25166 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d6cc9d3e0 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556d6c99e6c0 con 0x556d6cd37f40 2024-03-24T06:39:13.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f541001c420 con 0x556d6cd37f40 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 10 client.25166 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 10 client.25166 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 20 client.25166 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cd4e3f0 0x556d6cd4f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 4 client.25166 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.001+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d6c9ef270 con 0x556d6cd4e3f0 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.002+0000 7f5423400640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cd4e3f0 0x556d6cd4f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.002+0000 7f5423400640 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cd4e3f0 0x556d6cd4f860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca3efcc0b190396 server_cookie=ebda1e164a75b86b in_seq=0 out_seq=0 2024-03-24T06:39:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.002+0000 7f540cc00640 10 client.25166 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f540cc00640 1 -- 172.21.15.22:0/2697925983 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d6c9ef270 con 0x556d6cd4e3f0 2024-03-24T06:39:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f540cc00640 10 client.25166 handle_command_reply: tid=0 2024-03-24T06:39:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 client.25166 shutdown 2024-03-24T06:39:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 msgr2=0x556d6cd41ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd41ce0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f542419af40 tx=0x7f5410012ca0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 shutdown_connections 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cd4e3f0 0x556d6cd4f860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd48d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd41ce0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f540cc00640 0 client.25166 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2697925983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 >> 172.21.15.22:0/2697925983 conn(0x556d6cd32ac0 msgr2=0x556d6cd31cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 shutdown_connections 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.003+0000 7f542c5bc740 1 -- 172.21.15.22:0/2697925983 wait complete. 2024-03-24T06:39:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.004+0000 7f542c5bc740 20 client.25166 trim_cache size 0 max 16384 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.023+0000 7f542c5bc740 1 Processor -- start 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 -- start start 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd48ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd2e440 con 0x556d6cd37550 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd499a0 con 0x556d6cd37f40 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f542c5bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd4a0a0 con 0x556d6cd3d0b0 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f5423400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f5422000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd48ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f5423400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39166/0 (socket says 172.21.15.22:39166) 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f5423400640 1 -- 172.21.15.22:0/2334343763 learned_addr learned my addr 172.21.15.22:0/2334343763 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.024+0000 7f5423400640 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 -- 172.21.15.22:0/2334343763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 msgr2=0x556d6cd413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 -- 172.21.15.22:0/2334343763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 msgr2=0x556d6cd32ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 -- 172.21.15.22:0/2334343763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6cc39900 con 0x556d6cd37f40 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5423400640 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f5422000640 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd48ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x556d6cd38960 tx=0x7f541000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9444c63810948b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f540f400640 1 -- 172.21.15.22:0/2334343763 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410017040 con 0x556d6cd37f40 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f540f400640 1 -- 172.21.15.22:0/2334343763 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5410012e90 con 0x556d6cd37f40 2024-03-24T06:39:13.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.025+0000 7f540f400640 1 -- 172.21.15.22:0/2334343763 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541000fba0 con 0x556d6cd37f40 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- 172.21.15.22:0/2334343763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 msgr2=0x556d6cd48ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd48ce0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x556d6cd38960 tx=0x7f541000f790 comp rx=0 tx=0).stop 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- 172.21.15.22:0/2334343763 shutdown_connections 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd3d0b0 0x556d6cd413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd37f40 0x556d6cd48ce0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 --2- 172.21.15.22:0/2334343763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd32ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- 172.21.15.22:0/2334343763 >> 172.21.15.22:0/2334343763 conn(0x556d6cd32ac0 msgr2=0x556d6cd4d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- 172.21.15.22:0/2334343763 shutdown_connections 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- 172.21.15.22:0/2334343763 wait complete. 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.026+0000 7f542c5bc740 1 -- start start 2024-03-24T06:39:13.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 Processor -- start 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd44c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd486f0 con 0x556d6cd37550 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd48860 con 0x556d6cd3d0b0 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556d6cd48a20 con 0x556d6cd37f40 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f542c5bc740 10 client.0 _ll_register_callbacks cb 0x7f54100099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f5422000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f5423400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd44c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f5422000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55322/0 (socket says 172.21.15.22:55322) 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f5422000640 1 -- 172.21.15.22:0/1764160941 learned_addr learned my addr 172.21.15.22:0/1764160941 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.027+0000 7f5423400640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 -- 172.21.15.22:0/1764160941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 msgr2=0x556d6cd481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 -- 172.21.15.22:0/1764160941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 msgr2=0x556d6cd44c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd44c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 -- 172.21.15.22:0/1764160941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c022040 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5423400640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f5422000640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f541001c9c0 tx=0x7f541000fd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbff28a1e9b14c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f540cc00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410017030 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f54100050c0 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.028+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541c012d50 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f541000fec0 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 10 client.25207 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d6cc9d3e0 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f541c012d50 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f541002a440 con 0x556d6cd37f40 2024-03-24T06:39:13.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 10 client.25207 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 10 client.25207 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 20 client.25207 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cdb8c60 0x556d6cdb7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 4 client.25207 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.029+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "status"}) -- 0x556d6c9db610 con 0x556d6cdb8c60 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f5423400640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cdb8c60 0x556d6cdb7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f5423400640 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cdb8c60 0x556d6cdb7970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2a1f706a439ffbb server_cookie=bf33ee11d7d8d885 in_seq=0 out_seq=0 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f540cc00640 10 client.25207 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f540cc00640 1 -- 172.21.15.22:0/1764160941 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556d6c9db610 con 0x556d6cdb8c60 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f540cc00640 10 client.25207 handle_command_reply: tid=0 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f542c5bc740 1 client.25207 shutdown 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 msgr2=0x556d6cd45190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.030+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f541001c9c0 tx=0x7f541000fd50 comp rx=0 tx=0).stop 2024-03-24T06:39:13.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 shutdown_connections 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x556d6cdb8c60 0x556d6cdb7970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556d6cd3d0b0 0x556d6cd481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556d6cd37f40 0x556d6cd45190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 --2- 172.21.15.22:0/1764160941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556d6cd37550 0x556d6cd44c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f540cc00640 0 client.25207 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 >> 172.21.15.22:0/1764160941 conn(0x556d6cd32ac0 msgr2=0x556d6cd32ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 shutdown_connections 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 1 -- 172.21.15.22:0/1764160941 wait complete. 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.031+0000 7f542c5bc740 20 client.25207 trim_cache size 0 max 16384 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:13.034 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 3, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24544, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 101.63279221800001, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 160.15088578300001 2024-03-24T06:39:13.035 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 msgr2=0x7f54080a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 0x7f54080a4580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x556d6cd38340 tx=0x7f54140f6530 comp rx=0 tx=0).stop 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 msgr2=0x7f5424199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5414000c00 tx=0x7f54140040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 shutdown_connections 2024-03-24T06:39:13.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f54080a20c0 0x7f54080a4580 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54241021c0 0x7f54241a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5424101800 0x7f5424199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 --2- 172.21.15.22:0/3213844833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5424100610 0x7f5424199a30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 >> 172.21.15.22:0/3213844833 conn(0x7f54240fbba0 msgr2=0x7f54240fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 shutdown_connections 2024-03-24T06:39:13.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.032+0000 7f5429800640 1 -- 172.21.15.22:0/3213844833 wait complete. 2024-03-24T06:39:13.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:12 smithi022 ceph-mon[32003]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T06:39:13.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1123663702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.047 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:12 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/938807972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.049 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T06:39:13.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:12 smithi143 ceph-mon[36520]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T06:39:13.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1123663702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:12 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/938807972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 -- 172.21.15.22:0/1624014500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0100510 msgr2=0x7fa8e0100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 --2- 172.21.15.22:0/1624014500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0100510 0x7fa8e0100930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009900 tx=0x7fa8d002bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 -- 172.21.15.22:0/1624014500 shutdown_connections 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 --2- 172.21.15.22:0/1624014500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 0x7fa8e0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 --2- 172.21.15.22:0/1624014500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0101700 0x7fa8e0101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 --2- 172.21.15.22:0/1624014500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0100510 0x7fa8e0100930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 -- 172.21.15.22:0/1624014500 >> 172.21.15.22:0/1624014500 conn(0x7fa8e00fba80 msgr2=0x7fa8e00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 -- 172.21.15.22:0/1624014500 shutdown_connections 2024-03-24T06:39:13.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.259+0000 7fa8e5000640 1 -- 172.21.15.22:0/1624014500 wait complete. 2024-03-24T06:39:13.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.261+0000 7fa8e5000640 1 Processor -- start 2024-03-24T06:39:13.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.261+0000 7fa8e5000640 1 -- start start 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0100510 0x7fa8e0199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 0x7fa8e01a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01a1430 con 0x7fa8e0100510 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01a15a0 con 0x7fa8e01020c0 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8e5000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e01a1760 con 0x7fa8e0101700 2024-03-24T06:39:13.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8df400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0100510 0x7fa8e0199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55352/0 (socket says 172.21.15.22:55352) 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 -- 172.21.15.22:0/403353171 learned_addr learned my addr 172.21.15.22:0/403353171 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 0x7fa8e01a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 -- 172.21.15.22:0/403353171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 msgr2=0x7fa8e01a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 0x7fa8e01a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 -- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0100510 msgr2=0x7fa8e0199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0100510 0x7fa8e0199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.262+0000 7fa8dea00640 1 -- 172.21.15.22:0/403353171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d0009560 con 0x7fa8e0101700 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8dea00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00d950 tx=0x7fa8cc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8d7e00640 1 -- 172.21.15.22:0/403353171 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc009c30 con 0x7fa8e0101700 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8d7e00640 1 -- 172.21.15.22:0/403353171 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc009d90 con 0x7fa8e0101700 2024-03-24T06:39:13.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8d7e00640 1 -- 172.21.15.22:0/403353171 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00f7e0 con 0x7fa8e0101700 2024-03-24T06:39:13.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e01a1a40 con 0x7fa8e0101700 2024-03-24T06:39:13.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.263+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e01a1fa0 con 0x7fa8e0101700 2024-03-24T06:39:13.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.267+0000 7fa8d7e00640 1 -- 172.21.15.22:0/403353171 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fa8cc010ad0 con 0x7fa8e0101700 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.268+0000 7fa8e921e740 1 Processor -- start 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.268+0000 7fa8d7e00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 0x7fa8c40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.268+0000 7fa8d7e00640 1 -- 172.21.15.22:0/403353171 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa8cc0cdbd0 con 0x7fa8e0101700 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.268+0000 7fa8e921e740 1 -- start start 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 0x7fa8c40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed2142ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ed21398050 con 0x55ed2142f280 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ed21395580 con 0x55ed2142a110 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8e921e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed21390d60 con 0x55ed21429720 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8df400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8df400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55368/0 (socket says 172.21.15.22:55368) 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8df400640 1 -- 172.21.15.22:0/1546716848 learned_addr learned my addr 172.21.15.22:0/1546716848 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8dea00640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed2142ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.269+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 -- 172.21.15.22:0/1546716848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 msgr2=0x55ed21429b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 -- 172.21.15.22:0/1546716848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 msgr2=0x55ed21431710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 0x7fa8c40a4600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8005fd0 tx=0x7fa8d8011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 -- 172.21.15.22:0/1546716848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed21349620 con 0x55ed2142a110 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8df400640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8dea00640 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed2142ec10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc09a560 tx=0x7fa8cc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=325bbf73017de732 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8d4c00640 1 -- 172.21.15.22:0/1546716848 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc09e670 con 0x55ed2142a110 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8d4c00640 1 -- 172.21.15.22:0/1546716848 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc096040 con 0x55ed2142a110 2024-03-24T06:39:13.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.270+0000 7fa8d4c00640 1 -- 172.21.15.22:0/1546716848 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc083dc0 con 0x55ed2142a110 2024-03-24T06:39:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 -- 172.21.15.22:0/1546716848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 msgr2=0x55ed2142ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed2142ec10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc09a560 tx=0x7fa8cc005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 -- 172.21.15.22:0/1546716848 shutdown_connections 2024-03-24T06:39:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed2142ec10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 --2- 172.21.15.22:0/1546716848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed21429720 0x55ed21429b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 -- 172.21.15.22:0/1546716848 >> 172.21.15.22:0/1546716848 conn(0x55ed21424c90 msgr2=0x55ed21425090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 -- 172.21.15.22:0/1546716848 shutdown_connections 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.271+0000 7fa8e921e740 1 -- 172.21.15.22:0/1546716848 wait complete. 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 -- start start 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 Processor -- start 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed214338b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed2143ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ed21335fb0 con 0x55ed21429720 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ed21335c50 con 0x55ed2142a110 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed210d8f20 con 0x55ed2142f280 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.272+0000 7fa8e921e740 10 client.0 _ll_register_callbacks cb 0x55ed210a1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8df400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed214338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46160/0 (socket says 172.21.15.22:46160) 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 -- 172.21.15.22:0/4173519374 learned_addr learned my addr 172.21.15.22:0/4173519374 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed2143ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 -- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 msgr2=0x55ed2143ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed2143ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 -- 172.21.15.22:0/4173519374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 msgr2=0x55ed214338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed214338b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 -- 172.21.15.22:0/4173519374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e01a1fa0 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8dea00640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0a2790 tx=0x7fa8cc00df40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4db6c9ca801a4a59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8c9600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.273+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc083d60 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa8cc096db0 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ed210906c0 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc09e670 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 10 client.25184 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ed2138f3e0 con 0x55ed2142a110 2024-03-24T06:39:13.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ed210906c0 con 0x55ed2142a110 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa8cc099520 con 0x55ed2142a110 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 10 client.25184 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 10 client.25184 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 20 client.25184 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed2143f320 0x55ed21441440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 4 client.25184 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.274+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed210e1270 con 0x55ed2143f320 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.275+0000 7fa8df400640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed2143f320 0x55ed21441440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.275+0000 7fa8df400640 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed2143f320 0x55ed21441440 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f2272878092d008 server_cookie=67d7ef1783c5d60e in_seq=0 out_seq=0 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.275+0000 7fa8c9600640 10 client.25184 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.276+0000 7fa8c9600640 1 -- 172.21.15.22:0/4173519374 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed210e1270 con 0x55ed2143f320 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.276+0000 7fa8c9600640 10 client.25184 handle_command_reply: tid=0 2024-03-24T06:39:13.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.276+0000 7fa8e921e740 1 client.25184 shutdown 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 msgr2=0x55ed21433df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0a2790 tx=0x7fa8cc00df40 comp rx=0 tx=0).stop 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 shutdown_connections 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed2143f320 0x55ed21441440 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed2143ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed21433df0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8c9600640 0 client.25184 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 --2- 172.21.15.22:0/4173519374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed214338b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 >> 172.21.15.22:0/4173519374 conn(0x55ed21424c90 msgr2=0x55ed21423dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 shutdown_connections 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 1 -- 172.21.15.22:0/4173519374 wait complete. 2024-03-24T06:39:13.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.277+0000 7fa8e921e740 20 client.25184 trim_cache size 0 max 16384 2024-03-24T06:39:13.282 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:12 smithi155 ceph-mon[36262]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T06:39:13.282 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1123663702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.282 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:12 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/938807972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:13.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 Processor -- start 2024-03-24T06:39:13.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 -- start start 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 0x55ed2142ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ed2143bab0 con 0x55ed2142f280 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ed2141fd10 con 0x55ed21429720 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.312+0000 7fa8e921e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed2143fb30 con 0x55ed2142a110 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8df400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 0x55ed2142ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55382/0 (socket says 172.21.15.22:55382) 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 -- 172.21.15.22:0/2649223452 learned_addr learned my addr 172.21.15.22:0/2649223452 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8df400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 0x55ed2142ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46174/0 (socket says 172.21.15.22:46174) 2024-03-24T06:39:13.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 -- 172.21.15.22:0/2649223452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 msgr2=0x55ed2142ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 0x55ed2142ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 -- 172.21.15.22:0/2649223452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 msgr2=0x55ed21431670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dea00640 1 -- 172.21.15.22:0/2649223452 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed2132b900 con 0x55ed2142a110 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.313+0000 7fa8dfe00640 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.314+0000 7fa8dea00640 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc09a790 tx=0x7fa8cc098ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a4768e9235639cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.314+0000 7fa8cbe00640 1 -- 172.21.15.22:0/2649223452 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0ba180 con 0x55ed2142a110 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.314+0000 7fa8cbe00640 1 -- 172.21.15.22:0/2649223452 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc09f6b0 con 0x55ed2142a110 2024-03-24T06:39:13.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.314+0000 7fa8cbe00640 1 -- 172.21.15.22:0/2649223452 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0a2cc0 con 0x55ed2142a110 2024-03-24T06:39:13.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- 172.21.15.22:0/2649223452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 msgr2=0x55ed2143adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc09a790 tx=0x7fa8cc098ae0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- 172.21.15.22:0/2649223452 shutdown_connections 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed2142f280 0x55ed21431670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142a110 0x55ed2143adf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 --2- 172.21.15.22:0/2649223452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed21429720 0x55ed2142ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- 172.21.15.22:0/2649223452 >> 172.21.15.22:0/2649223452 conn(0x55ed21424c90 msgr2=0x55ed21423dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- 172.21.15.22:0/2649223452 shutdown_connections 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- 172.21.15.22:0/2649223452 wait complete. 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 -- start start 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.315+0000 7fa8e921e740 1 Processor -- start 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed21438de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed214a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ed214a49b0 con 0x55ed21429720 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ed214a4b20 con 0x55ed2142a110 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed214a4ce0 con 0x55ed2142f280 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8e921e740 10 client.0 _ll_register_callbacks cb 0x7fa8d8009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8dfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed214a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8dea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46190/0 (socket says 172.21.15.22:46190) 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8dea00640 1 -- 172.21.15.22:0/3229573818 learned_addr learned my addr 172.21.15.22:0/3229573818 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.316+0000 7fa8df400640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed21438de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 -- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 msgr2=0x55ed214a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed214a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 -- 172.21.15.22:0/3229573818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 msgr2=0x55ed21438de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed21438de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 -- 172.21.15.22:0/3229573818 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed2138f3e0 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8dea00640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc096fa0 tx=0x7fa8cc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8477a6b58523662 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8c9600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc083d60 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa8cc0a2cc0 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8cc096040 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa8cc0ba180 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8e921e740 10 client.25190 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.317+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ed21351560 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8cc096040 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa8cc09e070 con 0x55ed2142a110 2024-03-24T06:39:13.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 10 client.25190 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 10 client.25190 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 20 client.25190 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed214aa0f0 0x55ed214a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 4 client.25190 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "status"}) -- 0x55ed210cd610 con 0x55ed214aa0f0 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.318+0000 7fa8df400640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed214aa0f0 0x55ed214a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8df400640 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed214aa0f0 0x55ed214a8e00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a4cfeeb80b69c64 server_cookie=3ae00dc22bcf416a in_seq=0 out_seq=0 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8c9600640 10 client.25190 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8c9600640 1 -- 172.21.15.22:0/3229573818 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ed210cd610 con 0x55ed214aa0f0 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8c9600640 10 client.25190 handle_command_reply: tid=0 2024-03-24T06:39:13.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 client.25190 shutdown 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 msgr2=0x55ed214a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc096fa0 tx=0x7fa8cc005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 shutdown_connections 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ed214aa0f0 0x55ed214a8e00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed2142f280 0x55ed214a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ed2142a110 0x55ed214a1d20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8c9600640 0 client.25190 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 --2- 172.21.15.22:0/3229573818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ed21429720 0x55ed21438de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 >> 172.21.15.22:0/3229573818 conn(0x55ed21424c90 msgr2=0x55ed2142a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 shutdown_connections 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.319+0000 7fa8e921e740 1 -- 172.21.15.22:0/3229573818 wait complete. 2024-03-24T06:39:13.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e921e740 20 client.25190 trim_cache size 0 max 16384 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 2, 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24565, 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 102.935072547, 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:13.324 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 154.40134290200001 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 msgr2=0x7fa8c40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 0x7fa8c40a4600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8005fd0 tx=0x7fa8d8011810 comp rx=0 tx=0).stop 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 msgr2=0x7fa8e0199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00d950 tx=0x7fa8cc00de10 comp rx=0 tx=0).stop 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 shutdown_connections 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa8c40a2140 0x7fa8c40a4600 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8e01020c0 0x7fa8e01a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8e0101700 0x7fa8e0199e80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 --2- 172.21.15.22:0/403353171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa8e0100510 0x7fa8e0199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.320+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 >> 172.21.15.22:0/403353171 conn(0x7fa8e00fba80 msgr2=0x7fa8e00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.321+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 shutdown_connections 2024-03-24T06:39:13.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.321+0000 7fa8e5000640 1 -- 172.21.15.22:0/403353171 wait complete. 2024-03-24T06:39:13.337 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T06:39:13.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 -- 172.21.15.22:0/2946821655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 msgr2=0x7fadc4104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 --2- 172.21.15.22:0/2946821655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc4104d80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009900 tx=0x7fadb002bc50 comp rx=0 tx=0).stop 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 -- 172.21.15.22:0/2946821655 shutdown_connections 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 --2- 172.21.15.22:0/2946821655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc4105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 --2- 172.21.15.22:0/2946821655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc4104d80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 --2- 172.21.15.22:0/2946821655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 -- 172.21.15.22:0/2946821655 >> 172.21.15.22:0/2946821655 conn(0x7fadc40fbba0 msgr2=0x7fadc40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 -- 172.21.15.22:0/2946821655 shutdown_connections 2024-03-24T06:39:13.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.545+0000 7fadcc200640 1 -- 172.21.15.22:0/2946821655 wait complete. 2024-03-24T06:39:13.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.546+0000 7fadcc200640 1 Processor -- start 2024-03-24T06:39:13.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 -- start start 2024-03-24T06:39:13.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc419e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc41a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41a5900 con 0x7fadc40fffa0 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41a5a70 con 0x7fadc41052c0 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadcc200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fadc41a5c30 con 0x7fadc4102910 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadc9800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc419e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadc9800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39264/0 (socket says 172.21.15.22:39264) 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadc9800640 1 -- 172.21.15.22:0/1216260455 learned_addr learned my addr 172.21.15.22:0/1216260455 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.547+0000 7fadca200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc41a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 -- 172.21.15.22:0/1216260455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 msgr2=0x7fadc41a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc41a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 -- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 msgr2=0x7fadc419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 -- 172.21.15.22:0/1216260455 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb0009560 con 0x7fadc4102910 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadca200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc41a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc8e00640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc419e350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fadb002c190 tx=0x7fadb00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc2000640 1 -- 172.21.15.22:0/1216260455 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00044c0 con 0x7fadc4102910 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadc2000640 1 -- 172.21.15.22:0/1216260455 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00052d0 con 0x7fadc4102910 2024-03-24T06:39:13.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.548+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc41a5eb0 con 0x7fadc4102910 2024-03-24T06:39:13.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.549+0000 7fadc2000640 1 -- 172.21.15.22:0/1216260455 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0003d80 con 0x7fadc4102910 2024-03-24T06:39:13.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.549+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc41a6380 con 0x7fadc4102910 2024-03-24T06:39:13.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 Processor -- start 2024-03-24T06:39:13.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadc2000640 1 -- 172.21.15.22:0/1216260455 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fadb0005440 con 0x7fadc4102910 2024-03-24T06:39:13.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 -- start start 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac206970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac20e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac175050 con 0x55d5ac206550 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac172580 con 0x55d5ac206f40 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.552+0000 7fadcee1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac16dd60 con 0x55d5ac20c0b0 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadca200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac206970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac20e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46222/0 (socket says 172.21.15.22:46222) 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 -- 172.21.15.22:0/1542743701 learned_addr learned my addr 172.21.15.22:0/1542743701 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 -- 172.21.15.22:0/1542743701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 msgr2=0x55d5ac20e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac20e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 -- 172.21.15.22:0/1542743701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 msgr2=0x55d5ac206970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac206970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 -- 172.21.15.22:0/1542743701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5ac126620 con 0x55d5ac206f40 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadc9800640 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fadb800b890 tx=0x7fadb800bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb909449b3dcecc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.553+0000 7fadb6a00640 1 -- 172.21.15.22:0/1542743701 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb80041b0 con 0x55d5ac206f40 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadb6a00640 1 -- 172.21.15.22:0/1542743701 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb8003dd0 con 0x55d5ac206f40 2024-03-24T06:39:13.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadb6a00640 1 -- 172.21.15.22:0/1542743701 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8011a10 con 0x55d5ac206f40 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadc2000640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 0x7fadac0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadc2000640 1 -- 172.21.15.22:0/1216260455 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fadb00f0880 con 0x7fadc4102910 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadca200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 0x7fadac0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 -- 172.21.15.22:0/1542743701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 msgr2=0x55d5ac20ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fadb800b890 tx=0x7fadb800bd50 comp rx=0 tx=0).stop 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 -- 172.21.15.22:0/1542743701 shutdown_connections 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac20e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac20ba40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 --2- 172.21.15.22:0/1542743701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac206970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.554+0000 7fadcee1f740 1 -- 172.21.15.22:0/1542743701 >> 172.21.15.22:0/1542743701 conn(0x55d5ac201ac0 msgr2=0x55d5ac201ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.555+0000 7fadcee1f740 1 -- 172.21.15.22:0/1542743701 shutdown_connections 2024-03-24T06:39:13.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.555+0000 7fadcee1f740 1 -- 172.21.15.22:0/1542743701 wait complete. 2024-03-24T06:39:13.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.555+0000 7fadca200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 0x7fadac0a4600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55d5ac2077a0 tx=0x7fadbc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.555+0000 7fadcee1f740 1 -- start start 2024-03-24T06:39:13.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 Processor -- start 2024-03-24T06:39:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac2107c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac112fb0 con 0x55d5ac206550 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac112c50 con 0x55d5ac206f40 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d5abeb5f20 con 0x55d5ac20c0b0 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.556+0000 7fadcee1f740 10 client.0 _ll_register_callbacks cb 0x55d5abe7eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadca200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46230/0 (socket says 172.21.15.22:46230) 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 -- 172.21.15.22:0/2081396457 learned_addr learned my addr 172.21.15.22:0/2081396457 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 -- 172.21.15.22:0/2081396457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 msgr2=0x55d5ac217d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc9800640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac2107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 -- 172.21.15.22:0/2081396457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 msgr2=0x55d5ac2107c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac2107c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 -- 172.21.15.22:0/2081396457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb8004310 con 0x55d5ac206f40 2024-03-24T06:39:13.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc9800640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac2107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:39:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadc8e00640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadb00b9cc0 tx=0x7fadb0005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9905000dca7b968d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.557+0000 7fadab400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00bcc50 con 0x55d5ac206f40 2024-03-24T06:39:13.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fadb00bd9f0 con 0x55d5ac206f40 2024-03-24T06:39:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d5abe6d6c0 con 0x55d5ac206f40 2024-03-24T06:39:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadb00bd2f0 con 0x55d5ac206f40 2024-03-24T06:39:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadcee1f740 10 client.25202 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d5ac16c3e0 con 0x55d5ac206f40 2024-03-24T06:39:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00da030 con 0x55d5ac206f40 2024-03-24T06:39:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.558+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadb003c070 con 0x55d5ac206f40 2024-03-24T06:39:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 10 client.25202 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 10 client.25202 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T06:39:13.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 20 client.25202 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac21d170 0x55d5ac21e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 4 client.25202 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5abebe270 con 0x55d5ac21d170 2024-03-24T06:39:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.559+0000 7fadc9800640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac21d170 0x55d5ac21e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.560+0000 7fadc9800640 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac21d170 0x55d5ac21e5e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a22cf467e22b2fe server_cookie=3246dd6e4bcc4450 in_seq=0 out_seq=0 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.560+0000 7fadab400640 10 client.25202 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.560+0000 7fadab400640 1 -- 172.21.15.22:0/2081396457 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5abebe270 con 0x55d5ac21d170 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.560+0000 7fadab400640 10 client.25202 handle_command_reply: tid=0 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.560+0000 7fadcee1f740 1 client.25202 shutdown 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 msgr2=0x55d5ac210d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadb00b9cc0 tx=0x7fadb0005c00 comp rx=0 tx=0).stop 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 shutdown_connections 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac21d170 0x55d5ac21e5e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206f40 0x55d5ac210d00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 --2- 172.21.15.22:0/2081396457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac2107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadab400640 0 client.25202 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 >> 172.21.15.22:0/2081396457 conn(0x55d5ac201ac0 msgr2=0x55d5ac200cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 shutdown_connections 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 1 -- 172.21.15.22:0/2081396457 wait complete. 2024-03-24T06:39:13.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.561+0000 7fadcee1f740 20 client.25202 trim_cache size 0 max 16384 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.581+0000 7fadcee1f740 1 Processor -- start 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.581+0000 7fadcee1f740 1 -- start start 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.581+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 0x55d5ac20f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.581+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac20c0b0 0x55d5ac200430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadcee1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac1fd440 con 0x55d5ac20c0b0 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadcee1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac202510 con 0x55d5ac206550 2024-03-24T06:39:13.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadcee1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac2190c0 con 0x55d5ac206f40 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc9800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 0x55d5ac20f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55418/0 (socket says 172.21.15.22:55418) 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc9800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 0x55d5ac20f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46240/0 (socket says 172.21.15.22:46240) 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 -- 172.21.15.22:0/4189810189 learned_addr learned my addr 172.21.15.22:0/4189810189 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadca200640 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac20c0b0 0x55d5ac200430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 -- 172.21.15.22:0/4189810189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 msgr2=0x55d5ac20f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 0x55d5ac20f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 -- 172.21.15.22:0/4189810189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac20c0b0 msgr2=0x55d5ac200430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac20c0b0 0x55d5ac200430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 -- 172.21.15.22:0/4189810189 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5ac108900 con 0x55d5ac206f40 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.582+0000 7fadc8e00640 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadb002c230 tx=0x7fadb00f6220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e13e76be134f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadb6000640 1 -- 172.21.15.22:0/4189810189 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00bcc50 con 0x55d5ac206f40 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadb6000640 1 -- 172.21.15.22:0/4189810189 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00da070 con 0x55d5ac206f40 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadb6000640 1 -- 172.21.15.22:0/4189810189 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00c5a90 con 0x55d5ac206f40 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 -- 172.21.15.22:0/4189810189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 msgr2=0x55d5ac217d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadb002c230 tx=0x7fadb00f6220 comp rx=0 tx=0).stop 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 -- 172.21.15.22:0/4189810189 shutdown_connections 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac20c0b0 0x55d5ac200430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac217d00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 --2- 172.21.15.22:0/4189810189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac206550 0x55d5ac20f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 -- 172.21.15.22:0/4189810189 >> 172.21.15.22:0/4189810189 conn(0x55d5ac201ac0 msgr2=0x55d5ac2055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 -- 172.21.15.22:0/4189810189 shutdown_connections 2024-03-24T06:39:13.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.583+0000 7fadcee1f740 1 -- 172.21.15.22:0/4189810189 wait complete. 2024-03-24T06:39:13.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.584+0000 7fadcee1f740 1 -- start start 2024-03-24T06:39:13.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.584+0000 7fadcee1f740 1 Processor -- start 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac213bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac21e810 con 0x55d5ac206550 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac2189c0 con 0x55d5ac20c0b0 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d5ac112fb0 con 0x55d5ac206f40 2024-03-24T06:39:13.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadcee1f740 10 client.0 _ll_register_callbacks cb 0x7fadb0009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc9800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac213bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55430/0 (socket says 172.21.15.22:55430) 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 -- 172.21.15.22:0/990630414 learned_addr learned my addr 172.21.15.22:0/990630414 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadca200640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 -- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac20c0b0 msgr2=0x55d5ac217150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 -- 172.21.15.22:0/990630414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 msgr2=0x55d5ac213bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac213bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.585+0000 7fadc8e00640 1 -- 172.21.15.22:0/990630414 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5ac16c3e0 con 0x55d5ac206f40 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadc8e00640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006010 tx=0x7fadb00f6530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a238ca43f775a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00c5890 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb8003dd0 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fadb00bd9f0 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadb00bcc50 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 10 client.25243 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d5ac12e560 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb8003dd0 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadb003c070 con 0x55d5ac206f40 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 10 client.25243 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 10 client.25243 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.586+0000 7fadcee1f740 20 client.25243 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac287290 0x55d5ac285fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadcee1f740 4 client.25243 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "status"}) -- 0x55d5abeaa610 con 0x55d5ac287290 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadc9800640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac287290 0x55d5ac285fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadc9800640 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac287290 0x55d5ac285fa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b8eafa4bb20f516 server_cookie=6d29b051e348a5ec in_seq=0 out_seq=0 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.587+0000 7fadab400640 10 client.25243 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadab400640 1 -- 172.21.15.22:0/990630414 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d5abeaa610 con 0x55d5ac287290 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadab400640 10 client.25243 handle_command_reply: tid=0 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 client.25243 shutdown 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 msgr2=0x55d5ac214130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006010 tx=0x7fadb00f6530 comp rx=0 tx=0).stop 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 shutdown_connections 2024-03-24T06:39:13.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d5ac287290 0x55d5ac285fa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d5ac20c0b0 0x55d5ac217150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5ac206f40 0x55d5ac214130 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 --2- 172.21.15.22:0/990630414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d5ac206550 0x55d5ac213bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadab400640 0 client.25243 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 >> 172.21.15.22:0/990630414 conn(0x55d5ac201ac0 msgr2=0x55d5ac2055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 shutdown_connections 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 1 -- 172.21.15.22:0/990630414 wait complete. 2024-03-24T06:39:13.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.588+0000 7fadcee1f740 20 client.25243 trim_cache size 0 max 16384 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 1, 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24584, 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 104.21609872400001, 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:13.595 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:13.596 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 143.42623044499999 2024-03-24T06:39:13.596 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:13.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 msgr2=0x7fadac0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 0x7fadac0a4600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55d5ac2077a0 tx=0x7fadbc009500 comp rx=0 tx=0).stop 2024-03-24T06:39:13.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 msgr2=0x7fadc419e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc419e350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fadb002c190 tx=0x7fadb00040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 shutdown_connections 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fadac0a2140 0x7fadac0a4600 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadc41052c0 0x7fadc41a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc4102910 0x7fadc419e350 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 --2- 172.21.15.22:0/1216260455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fadc40fffa0 0x7fadc419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 >> 172.21.15.22:0/1216260455 conn(0x7fadc40fbba0 msgr2=0x7fadc4100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 shutdown_connections 2024-03-24T06:39:13.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.589+0000 7fadcc200640 1 -- 172.21.15.22:0/1216260455 wait complete. 2024-03-24T06:39:13.612 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T06:39:13.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.824+0000 7f3f7d400640 1 -- 172.21.15.22:0/689229029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f78104a20 msgr2=0x7f3f78104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.824+0000 7f3f7d400640 1 --2- 172.21.15.22:0/689229029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f78104a20 0x7f3f78104ea0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009be0 tx=0x7f3f6802bef0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 -- 172.21.15.22:0/689229029 shutdown_connections 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 --2- 172.21.15.22:0/689229029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f781053e0 0x7f3f7810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 --2- 172.21.15.22:0/689229029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f78104a20 0x7f3f78104ea0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 --2- 172.21.15.22:0/689229029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 0x7f3f78103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 -- 172.21.15.22:0/689229029 >> 172.21.15.22:0/689229029 conn(0x7f3f780fedc0 msgr2=0x7f3f781011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 -- 172.21.15.22:0/689229029 shutdown_connections 2024-03-24T06:39:13.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.825+0000 7f3f7d400640 1 -- 172.21.15.22:0/689229029 wait complete. 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 Processor -- start 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 -- start start 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 0x7f3f78199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 0x7f3f781a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781a1350 con 0x7f3f78103830 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781a14c0 con 0x7f3f78104a20 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7d400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3f781a1680 con 0x7f3f781053e0 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f77400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 0x7f3f78199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f77400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46282/0 (socket says 172.21.15.22:46282) 2024-03-24T06:39:13.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f77400640 1 -- 172.21.15.22:0/2078037631 learned_addr learned my addr 172.21.15.22:0/2078037631 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.826+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 0x7f3f781a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 -- 172.21.15.22:0/2078037631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 msgr2=0x7f3f781a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 0x7f3f781a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 -- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 msgr2=0x7f3f78199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 0x7f3f78199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 -- 172.21.15.22:0/2078037631 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f68009840 con 0x7f3f78104a20 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 0x7f3f781a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:13.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f77400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3f68004af0 tx=0x7f3f68033d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.827+0000 7f3f74c00640 1 -- 172.21.15.22:0/2078037631 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f68003d80 con 0x7f3f78104a20 2024-03-24T06:39:13.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.828+0000 7f3f74c00640 1 -- 172.21.15.22:0/2078037631 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f68004520 con 0x7f3f78104a20 2024-03-24T06:39:13.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.828+0000 7f3f74c00640 1 -- 172.21.15.22:0/2078037631 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680348f0 con 0x7f3f78104a20 2024-03-24T06:39:13.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.828+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f781a1900 con 0x7f3f78104a20 2024-03-24T06:39:13.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.828+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f781a1d60 con 0x7f3f78104a20 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f74c00640 1 -- 172.21.15.22:0/2078037631 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f3f68034a50 con 0x7f3f78104a20 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 Processor -- start 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 -- start start 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f0ff2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5594f0f5c050 con 0x5594f0ff3280 2024-03-24T06:39:13.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5594f0f59580 con 0x5594f0fee110 2024-03-24T06:39:13.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.831+0000 7f3f817f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594f0f54d60 con 0x5594f0fed720 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55458/0 (socket says 172.21.15.22:55458) 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 -- 172.21.15.22:0/4080398843 learned_addr learned my addr 172.21.15.22:0/4080398843 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f77400640 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f77e00640 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f0ff2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 -- 172.21.15.22:0/4080398843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 msgr2=0x5594f0ff2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f0ff2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 -- 172.21.15.22:0/4080398843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 msgr2=0x5594f0ff5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 -- 172.21.15.22:0/4080398843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594f0f0d620 con 0x5594f0fed720 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011540 tx=0x7f3f70011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a0c48e5b1137b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.832+0000 7f3f74c00640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 0x7f3f5c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f74c00640 1 -- 172.21.15.22:0/2078037631 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f3f680f0680 con 0x7f3f78104a20 2024-03-24T06:39:13.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f6d200640 1 -- 172.21.15.22:0/4080398843 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000f5c0 con 0x5594f0fed720 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f6d200640 1 -- 172.21.15.22:0/4080398843 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f7000fc00 con 0x5594f0fed720 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f6d200640 1 -- 172.21.15.22:0/4080398843 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f70019cb0 con 0x5594f0fed720 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 0x7f3f5c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f817f9740 1 -- 172.21.15.22:0/4080398843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 msgr2=0x5594f0fedb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f817f9740 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3f70011540 tx=0x7f3f70011a00 comp rx=0 tx=0).stop 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f817f9740 1 -- 172.21.15.22:0/4080398843 shutdown_connections 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.833+0000 7f3f817f9740 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f0ff2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 --2- 172.21.15.22:0/4080398843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0fedb40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 -- 172.21.15.22:0/4080398843 >> 172.21.15.22:0/4080398843 conn(0x5594f0fe8c90 msgr2=0x5594f0fe9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 -- 172.21.15.22:0/4080398843 shutdown_connections 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 0x7f3f5c0a4600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5594f0fee510 tx=0x7f3f60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 -- 172.21.15.22:0/4080398843 wait complete. 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 -- start start 2024-03-24T06:39:13.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.834+0000 7f3f817f9740 1 Processor -- start 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fee110 0x5594f0ff7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0ff3280 0x5594f0ffef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5594f0ef9fb0 con 0x5594f0fee110 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5594f0ef9c50 con 0x5594f0ff3280 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594f0c9cf20 con 0x5594f0fed720 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f817f9740 10 client.0 _ll_register_callbacks cb 0x5594f0c65ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fee110 0x5594f0ff7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55464/0 (socket says 172.21.15.22:55464) 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 -- 172.21.15.22:0/2442971519 learned_addr learned my addr 172.21.15.22:0/2442971519 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f7ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0ff3280 0x5594f0ffef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 -- 172.21.15.22:0/2442971519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0ff3280 msgr2=0x5594f0ffef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0ff3280 0x5594f0ffef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 -- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fee110 msgr2=0x5594f0ff7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fee110 0x5594f0ff7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.835+0000 7f3f77e00640 1 -- 172.21.15.22:0/2442971519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f70009d40 con 0x5594f0fed720 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3f60013010 tx=0x7f3f60004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa889971d2058413 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f65600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60022070 con 0x5594f0fed720 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f3f60009df0 con 0x5594f0fed720 2024-03-24T06:39:13.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594f0c546c0 con 0x5594f0fed720 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f60017070 con 0x5594f0fed720 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f817f9740 10 client.25255 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.836+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5594f0f533e0 con 0x5594f0fed720 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5594f0c546c0 con 0x5594f0fed720 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3f6000f580 con 0x5594f0fed720 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 10 client.25255 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 10 client.25255 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 20 client.25255 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f1004640 0x5594f1005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 4 client.25255 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594f0ca5270 con 0x5594f1004640 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.837+0000 7f3f77400640 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f1004640 0x5594f1005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.838+0000 7f3f77400640 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f1004640 0x5594f1005ab0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e5a38ab9115dd83 server_cookie=401c0666df61957c in_seq=0 out_seq=0 2024-03-24T06:39:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.838+0000 7f3f65600640 10 client.25255 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:13.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f65600640 1 -- 172.21.15.22:0/2442971519 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5594f0ca5270 con 0x5594f1004640 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f65600640 10 client.25255 handle_command_reply: tid=0 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 client.25255 shutdown 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 msgr2=0x5594f0ff79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3f60013010 tx=0x7f3f60004730 comp rx=0 tx=0).stop 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 shutdown_connections 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f1004640 0x5594f1005ab0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0ff3280 0x5594f0ffef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fee110 0x5594f0ff7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f65600640 0 client.25255 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2442971519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fed720 0x5594f0ff79a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.839+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 >> 172.21.15.22:0/2442971519 conn(0x5594f0fe8c90 msgr2=0x5594f0fe7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.840+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 shutdown_connections 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.840+0000 7f3f817f9740 1 -- 172.21.15.22:0/2442971519 wait complete. 2024-03-24T06:39:13.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.840+0000 7f3f817f9740 20 client.25255 trim_cache size 0 max 16384 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 Processor -- start 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 -- start start 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fed720 0x5594f0ff5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5594f0ff71f0 con 0x5594f0ff3280 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5594f0fffba0 con 0x5594f0fed720 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.860+0000 7f3f817f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594f0ca5270 con 0x5594f0fee110 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:55474/0 (socket says 172.21.15.22:55474) 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 -- 172.21.15.22:0/2467051185 learned_addr learned my addr 172.21.15.22:0/2467051185 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fed720 0x5594f0ff5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 -- 172.21.15.22:0/2467051185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fed720 msgr2=0x5594f0ff5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fed720 0x5594f0ff5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 -- 172.21.15.22:0/2467051185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 msgr2=0x5594f0ff7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 -- 172.21.15.22:0/2467051185 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594f0eef900 con 0x5594f0fee110 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f77400640 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3f68034f60 tx=0x7f3f68009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7efe192b3c565395 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.861+0000 7f3f67e00640 1 -- 172.21.15.22:0/2467051185 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680bcb10 con 0x5594f0fee110 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.862+0000 7f3f67e00640 1 -- 172.21.15.22:0/2467051185 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f680bc930 con 0x5594f0fee110 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.862+0000 7f3f67e00640 1 -- 172.21.15.22:0/2467051185 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680edeb0 con 0x5594f0fee110 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.862+0000 7f3f817f9740 1 -- 172.21.15.22:0/2467051185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 msgr2=0x5594f0ffeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.862+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3f68034f60 tx=0x7f3f68009280 comp rx=0 tx=0).stop 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 -- 172.21.15.22:0/2467051185 shutdown_connections 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0ff3280 0x5594f0ff7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0fee110 0x5594f0ffeeb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2467051185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fed720 0x5594f0ff5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 -- 172.21.15.22:0/2467051185 >> 172.21.15.22:0/2467051185 conn(0x5594f0fe8c90 msgr2=0x5594f0ff6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 -- 172.21.15.22:0/2467051185 shutdown_connections 2024-03-24T06:39:13.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 -- 172.21.15.22:0/2467051185 wait complete. 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.863+0000 7f3f817f9740 1 -- start start 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 Processor -- start 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fed720 0x5594f0ffce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0ff3280 0x5594f10688e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5594f1068f20 con 0x5594f0fed720 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5594f1069090 con 0x5594f0fee110 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594f1069250 con 0x5594f0ff3280 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f817f9740 10 client.0 _ll_register_callbacks cb 0x7f3f60009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f77400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fed720 0x5594f0ffce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f77400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46330/0 (socket says 172.21.15.22:46330) 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f77400640 1 -- 172.21.15.22:0/2274606171 learned_addr learned my addr 172.21.15.22:0/2274606171 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.864+0000 7f3f7ca00640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0ff3280 0x5594f10688e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 -- 172.21.15.22:0/2274606171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0ff3280 msgr2=0x5594f10688e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0ff3280 0x5594f10688e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 -- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fed720 msgr2=0x5594f0ffce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fed720 0x5594f0ffce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 -- 172.21.15.22:0/2274606171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f781a1d60 con 0x5594f0fee110 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f77400640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f680096f0 tx=0x7f3f680f3d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2d73a64151b75aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:13.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f65600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f680c1120 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f3f6809be40 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.865+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7000fc00 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3f680bd6d0 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 10 client.25226 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5594f0f15560 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f680a3080 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3f68004690 con 0x5594f0fee110 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 10 client.25226 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 10 client.25226 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 20 client.25226 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f106e590 0x5594f106d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 4 client.25226 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "status"}) -- 0x5594f0c91610 con 0x5594f106e590 2024-03-24T06:39:13.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.866+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f106e590 0x5594f106d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f77e00640 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f106e590 0x5594f106d2a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1397da045ec10e server_cookie=a1342801c98f077a in_seq=0 out_seq=0 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f65600640 10 client.25226 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f65600640 1 -- 172.21.15.22:0/2274606171 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5594f0c91610 con 0x5594f106e590 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f65600640 10 client.25226 handle_command_reply: tid=0 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f817f9740 1 client.25226 shutdown 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 msgr2=0x5594f1066390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.867+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f680096f0 tx=0x7f3f680f3d80 comp rx=0 tx=0).stop 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 shutdown_connections 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5594f0ff3280 0x5594f10688e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5594f106e590 0x5594f106d2a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5594f0fee110 0x5594f1066390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 --2- 172.21.15.22:0/2274606171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5594f0fed720 0x5594f0ffce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f65600640 0 client.25226 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 >> 172.21.15.22:0/2274606171 conn(0x5594f0fe8c90 msgr2=0x5594f0feeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 shutdown_connections 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 1 -- 172.21.15.22:0/2274606171 wait complete. 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.868+0000 7f3f817f9740 20 client.25226 trim_cache size 0 max 16384 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 0, 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24601, 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:13.870 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 136.70275820800001, 2024-03-24T06:39:13.871 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:13.871 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:13.871 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:13.871 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 146.60324839399999 2024-03-24T06:39:13.871 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 msgr2=0x7f3f5c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 0x7f3f5c0a4600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x5594f0fee510 tx=0x7f3f60009260 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 msgr2=0x7f3f78199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3f68004af0 tx=0x7f3f68033d00 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 shutdown_connections 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f3f5c0a2140 0x7f3f5c0a4600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f781053e0 0x7f3f781a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f78104a20 0x7f3f78199da0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 --2- 172.21.15.22:0/2078037631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f78103830 0x7f3f78199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 >> 172.21.15.22:0/2078037631 conn(0x7f3f780fedc0 msgr2=0x7f3f78107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 shutdown_connections 2024-03-24T06:39:13.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:13.869+0000 7f3f7d400640 1 -- 172.21.15.22:0/2078037631 wait complete. 2024-03-24T06:39:13.886 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T06:39:13.887 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T06:39:13.896 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:39:13.896 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T06:39:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:15.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:14 smithi022 ceph-mon[32003]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:39:15.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:14 smithi143 ceph-mon[36520]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:39:15.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:14 smithi155 ceph-mon[36262]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:39:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 -- 172.21.15.22:0/2233523029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8102250 msgr2=0x7fc6d8102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/2233523029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8102250 0x7fc6d8102670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800b0a0 tx=0x7fc6c800b3b0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 -- 172.21.15.22:0/2233523029 shutdown_connections 2024-03-24T06:39:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/2233523029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8103df0 0x7fc6d810a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/2233523029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8102250 0x7fc6d8102670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/2233523029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d81038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 -- 172.21.15.22:0/2233523029 >> 172.21.15.22:0/2233523029 conn(0x7fc6d80fd7b0 msgr2=0x7fc6d80ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.520+0000 7fc6ddc00700 1 -- 172.21.15.22:0/2233523029 shutdown_connections 2024-03-24T06:39:16.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.521+0000 7fc6ddc00700 1 -- 172.21.15.22:0/2233523029 wait complete. 2024-03-24T06:39:16.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.522+0000 7fc6ddc00700 1 Processor -- start 2024-03-24T06:39:16.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.522+0000 7fc6ddc00700 1 -- start start 2024-03-24T06:39:16.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:16.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8103df0 0x7fc6d81a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81a2140 con 0x7fc6d8103450 2024-03-24T06:39:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81a2280 con 0x7fc6d8102250 2024-03-24T06:39:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6ddc00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d81a2410 con 0x7fc6d8103df0 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d6a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d6000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d7400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8103df0 0x7fc6d81a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d6a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46288/0 (socket says 172.21.15.22:46288) 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d6000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d819aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59778/0 (socket says 172.21.15.22:59778) 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.523+0000 7fc6d6a00700 1 -- 172.21.15.22:0/3057887028 learned_addr learned my addr 172.21.15.22:0/3057887028 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 -- 172.21.15.22:0/3057887028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8103df0 msgr2=0x7fc6d81a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8103df0 0x7fc6d81a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 -- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 msgr2=0x7fc6d819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 -- 172.21.15.22:0/3057887028 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c8009d00 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6d6a00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800b070 tx=0x7fc6c80079b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.524+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c800e040 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.525+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c800be40 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.525+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d81a2690 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.525+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c801b6c0 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.525+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d81a2b00 con 0x7fc6d8102250 2024-03-24T06:39:16.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.526+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d8068ec0 con 0x7fc6d8102250 2024-03-24T06:39:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.529+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fc6c801d470 con 0x7fc6d8102250 2024-03-24T06:39:16.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.531+0000 7fc6cf400700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 0x7fc6bc0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:16.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.531+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fc6c80cef20 con 0x7fc6d8102250 2024-03-24T06:39:16.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.531+0000 7fc6d6000700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 0x7fc6bc0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:16.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.532+0000 7fc6d6000700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 0x7fc6bc0a4690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009c60 tx=0x7fc6c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:16.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.539+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc6c8097ba0 con 0x7fc6d8102250 2024-03-24T06:39:16.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.789+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fc6d8075b50 con 0x7fc6bc0a21d0 2024-03-24T06:39:16.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.791+0000 7fc6cf400700 1 -- 172.21.15.22:0/3057887028 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc6d8075b50 con 0x7fc6bc0a21d0 2024-03-24T06:39:16.790 INFO:teuthology.orchestra.run.smithi022.stdout:Backend: cephadm 2024-03-24T06:39:16.791 INFO:teuthology.orchestra.run.smithi022.stdout:Available: Yes 2024-03-24T06:39:16.791 INFO:teuthology.orchestra.run.smithi022.stdout:Paused: No 2024-03-24T06:39:16.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 msgr2=0x7fc6bc0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:16.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 0x7fc6bc0a4690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009c60 tx=0x7fc6c4009260 comp rx=0 tx=0).stop 2024-03-24T06:39:16.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 msgr2=0x7fc6d819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800b070 tx=0x7fc6c80079b0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 shutdown_connections 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc6bc0a21d0 0x7fc6bc0a4690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc6d8103df0 0x7fc6d81a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d8102250 0x7fc6d819a5b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 --2- 172.21.15.22:0/3057887028 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d8103450 0x7fc6d819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 >> 172.21.15.22:0/3057887028 conn(0x7fc6d80fd7b0 msgr2=0x7fc6d8074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 shutdown_connections 2024-03-24T06:39:16.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:16.795+0000 7fc6ddc00700 1 -- 172.21.15.22:0/3057887028 wait complete. 2024-03-24T06:39:17.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:16 smithi022 ceph-mon[32003]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:17.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:16 smithi143 ceph-mon[36520]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:17.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:16 smithi155 ceph-mon[36262]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:17.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T06:39:18.061 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:18.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:17 smithi022 ceph-mon[32003]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:18.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:17 smithi143 ceph-mon[36520]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:17 smithi155 ceph-mon[36262]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:19.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:18 smithi022 ceph-mon[32003]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:19.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:18 smithi143 ceph-mon[36520]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:19.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:18 smithi155 ceph-mon[36262]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:20.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 -- 172.21.15.22:0/3569303730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c105df0 msgr2=0x7fa10c106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:20.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 --2- 172.21.15.22:0/3569303730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c105df0 0x7fa10c106270 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b0a0 tx=0x7fa10400b3b0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 -- 172.21.15.22:0/3569303730 shutdown_connections 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 --2- 172.21.15.22:0/3569303730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c100ab0 0x7fa10c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 --2- 172.21.15.22:0/3569303730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c105df0 0x7fa10c106270 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 --2- 172.21.15.22:0/3569303730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 -- 172.21.15.22:0/3569303730 >> 172.21.15.22:0/3569303730 conn(0x7fa10c0fc6d0 msgr2=0x7fa10c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 -- 172.21.15.22:0/3569303730 shutdown_connections 2024-03-24T06:39:20.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.452+0000 7fa113c00700 1 -- 172.21.15.22:0/3569303730 wait complete. 2024-03-24T06:39:20.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.453+0000 7fa113c00700 1 Processor -- start 2024-03-24T06:39:20.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.453+0000 7fa113c00700 1 -- start start 2024-03-24T06:39:20.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c100ab0 0x7fa10c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:20.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:20.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c105df0 0x7fa10c1a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1a6500 con 0x7fa10c105df0 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1a6640 con 0x7fa10c100ab0 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa113c00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa10c1a67d0 con 0x7fa10c103420 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44912/0 (socket says 172.21.15.22:44912) 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 -- 172.21.15.22:0/3450845908 learned_addr learned my addr 172.21.15.22:0/3450845908 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa111200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c100ab0 0x7fa10c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa111c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c105df0 0x7fa10c1a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 -- 172.21.15.22:0/3450845908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c100ab0 msgr2=0x7fa10c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c100ab0 0x7fa10c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 -- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c105df0 msgr2=0x7fa10c1a5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c105df0 0x7fa10c1a5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.454+0000 7fa10be00700 1 -- 172.21.15.22:0/3450845908 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa104009d00 con 0x7fa10c103420 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa10be00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e9d0 tx=0x7fa0f800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8009790 con 0x7fa10c103420 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f800cbd0 con 0x7fa10c103420 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10c1a6ab0 con 0x7fa10c103420 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8018680 con 0x7fa10c103420 2024-03-24T06:39:20.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.455+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10c1a7000 con 0x7fa10c103420 2024-03-24T06:39:20.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.456+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa10c0fe250 con 0x7fa10c103420 2024-03-24T06:39:20.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.458+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fa0f800cd40 con 0x7fa10c103420 2024-03-24T06:39:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.461+0000 7fa109600700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 0x7fa0f40a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.461+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa0f8014070 con 0x7fa10c103420 2024-03-24T06:39:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.461+0000 7fa111200700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 0x7fa0f40a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:20.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.462+0000 7fa111200700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 0x7fa0f40a4690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa100009c60 tx=0x7fa100009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:20.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.464+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa0f8053280 con 0x7fa10c103420 2024-03-24T06:39:20.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.774+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fa10c02dc70 con 0x7fa0f40a21d0 2024-03-24T06:39:20.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.802+0000 7fa109600700 1 -- 172.21.15.22:0/3450845908 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fa10c02dc70 con 0x7fa0f40a21d0 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.a smithi022 running (2m) 32s ago 2m 26.7M - 19.0.0-1855-g21b3bd67 553463a90fd5 eddae7b6d838 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.b smithi143 running (2m) 81s ago 2m 17.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 4434474f729d 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.c smithi155 running (2m) 81s ago 2m 17.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 b98fdcb700ea 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.d smithi022 running (2m) 32s ago 2m 12.4M - 19.0.0-1855-g21b3bd67 553463a90fd5 506cb8f9e4c6 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.e smithi143 running (2m) 81s ago 2m 15.7M - 19.0.0-1855-g21b3bd67 553463a90fd5 0f41d1e7b8ed 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.f smithi155 running (2m) 81s ago 2m 19.3M - 19.0.0-1855-g21b3bd67 553463a90fd5 a19cae688fa8 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.g smithi022 running (2m) 32s ago 2m 12.3M - 19.0.0-1855-g21b3bd67 553463a90fd5 e181e59c64ef 2024-03-24T06:39:20.803 INFO:teuthology.orchestra.run.smithi022.stdout:mds.h smithi143 running (2m) 81s ago 2m 16.9M - 19.0.0-1855-g21b3bd67 553463a90fd5 086cde1d2cec 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mds.i smithi155 running (2m) 81s ago 2m 16.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 4039ab6cfeca 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mds.j smithi022 running (3m) 32s ago 3m 14.8M - 19.0.0-1855-g21b3bd67 553463a90fd5 492442409fbf 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mds.k smithi143 running (3m) 81s ago 3m 17.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 d95ef13680ee 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mds.l smithi155 running (3m) 81s ago 3m 15.9M - 19.0.0-1855-g21b3bd67 553463a90fd5 0c59592ec916 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.x smithi022 *:9283,8765 running (11m) 32s ago 11m 514M - 19.0.0-1855-g21b3bd67 553463a90fd5 c73fd17ae3c8 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.y smithi143 *:8443,8765 running (9m) 81s ago 9m 450M - 19.0.0-1855-g21b3bd67 553463a90fd5 f2ffe76ce0a1 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.z smithi155 *:8443,8765 running (9m) 81s ago 9m 453M - 19.0.0-1855-g21b3bd67 553463a90fd5 3fc325da53bf 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mon.a smithi022 running (11m) 32s ago 11m 48.1M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 c6e194ac910d 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mon.b smithi143 running (9m) 81s ago 9m 42.1M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 afc985d51a41 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:mon.c smithi155 running (9m) 81s ago 9m 43.9M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 9915a307e287 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (8m) 32s ago 8m 46.9M 4273M 19.0.0-1855-g21b3bd67 553463a90fd5 c0bdd3170b41 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi143 running (8m) 81s ago 8m 53.2M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 098bde2ac34e 2024-03-24T06:39:20.804 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi155 running (7m) 81s ago 7m 46.9M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 0484b9372d44 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi022 running (7m) 32s ago 7m 50.3M 4273M 19.0.0-1855-g21b3bd67 553463a90fd5 afd041a25700 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi143 running (6m) 81s ago 6m 53.9M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 9421ea1fd317 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi155 running (6m) 81s ago 6m 53.4M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 608c6ecff728 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 smithi022 running (5m) 32s ago 5m 42.0M 4273M 19.0.0-1855-g21b3bd67 553463a90fd5 b6779de5a45e 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 smithi143 running (5m) 81s ago 5m 44.7M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 22f74fb3e458 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.8 smithi155 running (4m) 81s ago 4m 52.0M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 10b2ed30cd82 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.9 smithi022 running (4m) 32s ago 4m 48.0M 4273M 19.0.0-1855-g21b3bd67 553463a90fd5 ac987208f9b0 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.10 smithi143 running (3m) 81s ago 3m 46.6M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 f042533963af 2024-03-24T06:39:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:osd.11 smithi155 running (3m) 81s ago 3m 41.0M 4270M 19.0.0-1855-g21b3bd67 553463a90fd5 143f11ab3ae3 2024-03-24T06:39:20.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.805+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 msgr2=0x7fa0f40a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:20.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 0x7fa0f40a4690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa100009c60 tx=0x7fa100009260 comp rx=0 tx=0).stop 2024-03-24T06:39:20.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 msgr2=0x7fa10c19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:20.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e9d0 tx=0x7fa0f800ece0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 shutdown_connections 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa10c105df0 0x7fa10c1a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa0f40a21d0 0x7fa0f40a4690 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10c100ab0 0x7fa10c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 --2- 172.21.15.22:0/3450845908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa10c103420 0x7fa10c19eeb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 >> 172.21.15.22:0/3450845908 conn(0x7fa10c0fc6d0 msgr2=0x7fa10c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 shutdown_connections 2024-03-24T06:39:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:20.806+0000 7fa113c00700 1 -- 172.21.15.22:0/3450845908 wait complete. 2024-03-24T06:39:21.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:20 smithi022 ceph-mon[32003]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:21.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:20 smithi143 ceph-mon[36520]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:21.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:20 smithi155 ceph-mon[36262]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:21.624 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T06:39:22.142 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:22.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:21 smithi022 ceph-mon[32003]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:22.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:21 smithi143 ceph-mon[36520]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:22.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:21 smithi155 ceph-mon[36262]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:23.144 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:22 smithi022 ceph-mon[32003]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:23.177 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:22 smithi143 ceph-mon[36520]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:23.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:22 smithi155 ceph-mon[36262]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 -- 172.21.15.22:0/336061495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103d10 msgr2=0x7f842810a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 --2- 172.21.15.22:0/336061495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103d10 0x7f842810a5c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f842000b0a0 tx=0x7f842000b3b0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 -- 172.21.15.22:0/336061495 shutdown_connections 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 --2- 172.21.15.22:0/336061495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103d10 0x7f842810a5c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 --2- 172.21.15.22:0/336061495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428102170 0x7f8428102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 --2- 172.21.15.22:0/336061495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428103370 0x7f84281037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 -- 172.21.15.22:0/336061495 >> 172.21.15.22:0/336061495 conn(0x7f84280fd710 msgr2=0x7f84280ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:24.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.404+0000 7f842d800700 1 -- 172.21.15.22:0/336061495 shutdown_connections 2024-03-24T06:39:24.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.405+0000 7f842d800700 1 -- 172.21.15.22:0/336061495 wait complete. 2024-03-24T06:39:24.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 Processor -- start 2024-03-24T06:39:24.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 -- start start 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 0x7f842819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 0x7f842819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 0x7f84281a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f84281a1f90 con 0x7f8428102170 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f84281a20d0 con 0x7f8428103d10 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.406+0000 7f842d800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84281a2260 con 0x7f8428103370 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f8427e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 0x7f842819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f8427400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 0x7f842819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:24.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f8427400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 0x7f842819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44942/0 (socket says 172.21.15.22:44942) 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f8427e00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 0x7f842819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59816/0 (socket says 172.21.15.22:59816) 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f8427400700 1 -- 172.21.15.22:0/3068309477 learned_addr learned my addr 172.21.15.22:0/3068309477 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 0x7f84281a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 -- 172.21.15.22:0/3068309477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 msgr2=0x7f842819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 0x7f842819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 -- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 msgr2=0x7f842819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 0x7f842819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 -- 172.21.15.22:0/3068309477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8418009520 con 0x7f8428103d10 2024-03-24T06:39:24.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.407+0000 7f842ce00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 0x7f84281a19c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8420007c10 tx=0x7f8420007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:24.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.408+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842000e040 con 0x7f8428103d10 2024-03-24T06:39:24.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.408+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f842001d470 con 0x7f8428103d10 2024-03-24T06:39:24.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.408+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8420009d00 con 0x7f8428103d10 2024-03-24T06:39:24.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.408+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842001b6e0 con 0x7f8428103d10 2024-03-24T06:39:24.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.408+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84281a27c0 con 0x7f8428103d10 2024-03-24T06:39:24.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.409+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8428068ec0 con 0x7f8428103d10 2024-03-24T06:39:24.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.411+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f842001d5e0 con 0x7f8428103d10 2024-03-24T06:39:24.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.412+0000 7f8424c00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 0x7f840c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:24.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.412+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f84200cee40 con 0x7f8428103d10 2024-03-24T06:39:24.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.412+0000 7f8427e00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 0x7f840c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:24.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.413+0000 7f8427e00700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 0x7f840c0a4690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84281031d0 tx=0x7f8418009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:24.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.416+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8420097ac0 con 0x7f8428103d10 2024-03-24T06:39:24.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.652+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f8428075b50 con 0x7f840c0a21d0 2024-03-24T06:39:24.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.666+0000 7f8424c00700 1 -- 172.21.15.22:0/3068309477 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f8428075b50 con 0x7f840c0a21d0 2024-03-24T06:39:24.666 INFO:teuthology.orchestra.run.smithi022.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T06:39:24.667 INFO:teuthology.orchestra.run.smithi022.stdout:mds.all 12/12 85s ago 3m smithi022=a;smithi022=d;smithi022=g;smithi022=j;smithi143=b;smithi143=e;smithi143=h;smithi143=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-24T06:39:24.667 INFO:teuthology.orchestra.run.smithi022.stdout:mgr 3/3 85s ago 9m smithi022=x;smithi143=y;smithi155=z;count:3 2024-03-24T06:39:24.667 INFO:teuthology.orchestra.run.smithi022.stdout:mon 3/3 85s ago 9m smithi022:172.21.15.22=a;smithi143:172.21.15.143=b;smithi155:172.21.15.155=c;count:3 2024-03-24T06:39:24.667 INFO:teuthology.orchestra.run.smithi022.stdout:osd 12 85s ago - 2024-03-24T06:39:24.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 msgr2=0x7f840c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:24.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 0x7f840c0a4690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84281031d0 tx=0x7f8418009260 comp rx=0 tx=0).stop 2024-03-24T06:39:24.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 msgr2=0x7f84281a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:24.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 0x7f84281a19c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8420007c10 tx=0x7f8420007930 comp rx=0 tx=0).stop 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 shutdown_connections 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8428102170 0x7f842819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f840c0a21d0 0x7f840c0a4690 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8428103d10 0x7f84281a19c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 --2- 172.21.15.22:0/3068309477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8428103370 0x7f842819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 >> 172.21.15.22:0/3068309477 conn(0x7f84280fd710 msgr2=0x7f8428074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 shutdown_connections 2024-03-24T06:39:24.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:24.669+0000 7f842d800700 1 -- 172.21.15.22:0/3068309477 wait complete. 2024-03-24T06:39:25.078 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:24 smithi022 ceph-mon[32003]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:25.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:24 smithi143 ceph-mon[36520]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:25.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:24 smithi155 ceph-mon[36262]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:25.618 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T06:39:26.083 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:26.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:25 smithi022 ceph-mon[32003]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:25 smithi143 ceph-mon[36520]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:25 smithi155 ceph-mon[36262]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:27.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:26 smithi022 ceph-mon[32003]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:27.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:26 smithi143 ceph-mon[36520]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:26 smithi155 ceph-mon[36262]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:28.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.340+0000 7f1d39a00700 1 -- 172.21.15.22:0/3495351594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103480 msgr2=0x7f1d341038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:28.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 --2- 172.21.15.22:0/3495351594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103480 0x7f1d341038e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1d200099c0 tx=0x7f1d20009cd0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 -- 172.21.15.22:0/3495351594 shutdown_connections 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 --2- 172.21.15.22:0/3495351594 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103e20 0x7f1d3410a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 --2- 172.21.15.22:0/3495351594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 --2- 172.21.15.22:0/3495351594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103480 0x7f1d341038e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 -- 172.21.15.22:0/3495351594 >> 172.21.15.22:0/3495351594 conn(0x7f1d340fd820 msgr2=0x7f1d340ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 -- 172.21.15.22:0/3495351594 shutdown_connections 2024-03-24T06:39:28.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.341+0000 7f1d39a00700 1 -- 172.21.15.22:0/3495351594 wait complete. 2024-03-24T06:39:28.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.342+0000 7f1d39a00700 1 Processor -- start 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.342+0000 7f1d39a00700 1 -- start start 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 0x7f1d341986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 0x7f1d3419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3419fd40 con 0x7f1d34103480 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3419fe80 con 0x7f1d34103e20 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d39a00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1d341a0010 con 0x7f1d34102280 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d32a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d32a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43634/0 (socket says 172.21.15.22:43634) 2024-03-24T06:39:28.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d32a00700 1 -- 172.21.15.22:0/123768987 learned_addr learned my addr 172.21.15.22:0/123768987 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d33400700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 0x7f1d3419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.343+0000 7f1d32000700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 0x7f1d341986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 -- 172.21.15.22:0/123768987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 msgr2=0x7f1d3419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 0x7f1d3419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 -- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 msgr2=0x7f1d341986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 0x7f1d341986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 -- 172.21.15.22:0/123768987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d200095f0 con 0x7f1d34102280 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32000700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 0x7f1d341986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:28.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d33400700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 0x7f1d3419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d32a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e9a0 tx=0x7f1d2400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24009790 con 0x7f1d34102280 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d34074490 con 0x7f1d34102280 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d24004500 con 0x7f1d34102280 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.344+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24003e30 con 0x7f1d34102280 2024-03-24T06:39:28.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.345+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d340749e0 con 0x7f1d34102280 2024-03-24T06:39:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.346+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d341095e0 con 0x7f1d34102280 2024-03-24T06:39:28.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.348+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f1d24004000 con 0x7f1d34102280 2024-03-24T06:39:28.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.349+0000 7f1d2b400700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 0x7f1d180a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:28.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.349+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f1d24014070 con 0x7f1d34102280 2024-03-24T06:39:28.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.350+0000 7f1d32000700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 0x7f1d180a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:28.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.350+0000 7f1d32000700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 0x7f1d180a4690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000b580 tx=0x7f1d20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:28.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.358+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1d24096ec0 con 0x7f1d34102280 2024-03-24T06:39:28.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.599+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f1d3402dc00 con 0x7f1d180a21d0 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.604+0000 7f1d2b400700 1 -- 172.21.15.22:0/123768987 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f1d3402dc00 con 0x7f1d180a21d0 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stdout:HOST ADDR LABELS STATUS 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 172.21.15.22 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stdout:smithi143 172.21.15.143 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stdout:smithi155 172.21.15.155 2024-03-24T06:39:28.604 INFO:teuthology.orchestra.run.smithi022.stdout:3 hosts in cluster 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.607+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 msgr2=0x7f1d180a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.607+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 0x7f1d180a4690 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000b580 tx=0x7f1d20005fb0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.607+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 msgr2=0x7f1d341981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.607+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e9a0 tx=0x7f1d2400ecb0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 shutdown_connections 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f1d180a21d0 0x7f1d180a4690 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d34103e20 0x7f1d3419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d34102280 0x7f1d341981b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 --2- 172.21.15.22:0/123768987 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1d34103480 0x7f1d341986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:28.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 >> 172.21.15.22:0/123768987 conn(0x7f1d340fd820 msgr2=0x7f1d34106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:28.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 shutdown_connections 2024-03-24T06:39:28.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:28.608+0000 7f1d39a00700 1 -- 172.21.15.22:0/123768987 wait complete. 2024-03-24T06:39:29.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:28 smithi022 ceph-mon[32003]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:29.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:28 smithi143 ceph-mon[36520]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:29.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:28 smithi155 ceph-mon[36262]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:29.534 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T06:39:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:30.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:29 smithi022 ceph-mon[32003]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:29 smithi143 ceph-mon[36520]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:29 smithi155 ceph-mon[36262]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:31.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:30 smithi022 ceph-mon[32003]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:31.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:30 smithi143 ceph-mon[36520]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:31.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:30 smithi155 ceph-mon[36262]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:32.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 -- 172.21.15.22:0/2990644774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4103dd0 msgr2=0x7f50e410a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:32.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 --2- 172.21.15.22:0/2990644774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4103dd0 0x7f50e410a680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009f90 tx=0x7f50dc00b330 comp rx=0 tx=0).stop 2024-03-24T06:39:32.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 -- 172.21.15.22:0/2990644774 shutdown_connections 2024-03-24T06:39:32.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 --2- 172.21.15.22:0/2990644774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4103dd0 0x7f50e410a680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 --2- 172.21.15.22:0/2990644774 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4102230 0x7f50e4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 --2- 172.21.15.22:0/2990644774 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 0x7f50e4103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 -- 172.21.15.22:0/2990644774 >> 172.21.15.22:0/2990644774 conn(0x7f50e40fd810 msgr2=0x7f50e40ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:32.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.394+0000 7f50ea600700 1 -- 172.21.15.22:0/2990644774 shutdown_connections 2024-03-24T06:39:32.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.395+0000 7f50ea600700 1 -- 172.21.15.22:0/2990644774 wait complete. 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 Processor -- start 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 -- start start 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 0x7f50e40745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4103dd0 0x7f50e4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f50e40752b0 con 0x7f50e4103430 2024-03-24T06:39:32.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f50e4075420 con 0x7f50e4103dd0 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50ea600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f50e41a2380 con 0x7f50e4102230 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50e3400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50e3400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43648/0 (socket says 172.21.15.22:43648) 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50e3400700 1 -- 172.21.15.22:0/4261294230 learned_addr learned my addr 172.21.15.22:0/4261294230 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.396+0000 7f50e2a00700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 0x7f50e40745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3e00700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4103dd0 0x7f50e4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 -- 172.21.15.22:0/4261294230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4103dd0 msgr2=0x7f50e4074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4103dd0 0x7f50e4074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 -- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 msgr2=0x7f50e40745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:32.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 0x7f50e40745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 -- 172.21.15.22:0/4261294230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc009c40 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.397+0000 7f50e3400700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50d400e9f0 tx=0x7f50d400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.398+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4004220 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.398+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d4004380 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.398+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e41a2660 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.398+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4003c40 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.398+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e41a2bb0 con 0x7f50e4102230 2024-03-24T06:39:32.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.399+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e41095f0 con 0x7f50e4102230 2024-03-24T06:39:32.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.401+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f50d400c7c0 con 0x7f50e4102230 2024-03-24T06:39:32.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.402+0000 7f50dbe00700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 0x7f50c80a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:32.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.402+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f50d4014070 con 0x7f50e4102230 2024-03-24T06:39:32.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.403+0000 7f50e2a00700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 0x7f50c80a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:32.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.403+0000 7f50e2a00700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 0x7f50c80a4690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009c60 tx=0x7f50d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:32.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.410+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f50d4096cc0 con 0x7f50e4102230 2024-03-24T06:39:32.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.649+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 --> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f50e402dc00 con 0x7f50c80a21d0 2024-03-24T06:39:32.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.652+0000 7f50dbe00700 1 -- 172.21.15.22:0/4261294230 <== mgr.14152 v2:172.21.15.22:6800/503368228 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f50e402dc00 con 0x7f50c80a21d0 2024-03-24T06:39:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T06:39:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402EC480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-24T06:39:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:smithi143 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330073400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T06:39:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T06:39:32.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 msgr2=0x7f50c80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:32.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 0x7f50c80a4690 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009c60 tx=0x7f50d0009260 comp rx=0 tx=0).stop 2024-03-24T06:39:32.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 msgr2=0x7f50e4075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f50d400e9f0 tx=0x7f50d400ed00 comp rx=0 tx=0).stop 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 shutdown_connections 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f50c80a21d0 0x7f50c80a4690 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e4103dd0 0x7f50e4074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50e4102230 0x7f50e4075fa0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 --2- 172.21.15.22:0/4261294230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f50e4103430 0x7f50e40745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 >> 172.21.15.22:0/4261294230 conn(0x7f50e40fd810 msgr2=0x7f50e4106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 shutdown_connections 2024-03-24T06:39:32.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:32.656+0000 7f50ea600700 1 -- 172.21.15.22:0/4261294230 wait complete. 2024-03-24T06:39:33.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:32 smithi143 ceph-mon[36520]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:33.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:32 smithi155 ceph-mon[36262]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:33.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:32 smithi022 ceph-mon[32003]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:33.647 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T06:39:33.657 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:39:33.658 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T06:39:34.157 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:33 smithi143 ceph-mon[36520]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:33 smithi155 ceph-mon[36262]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:34.310 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:33 smithi022 ceph-mon[32003]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:39:35.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:34 smithi143 ceph-mon[36520]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:35.184 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:34 smithi022 ceph-mon[32003]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:35.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:34 smithi155 ceph-mon[36262]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:36.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 -- 172.21.15.22:0/610194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 msgr2=0x7f5ad40750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:36.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 --2- 172.21.15.22:0/610194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad40750e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009990 tx=0x7f5abc009ca0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 -- 172.21.15.22:0/610194103 shutdown_connections 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 --2- 172.21.15.22:0/610194103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad410a110 0x7f5ad410c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 --2- 172.21.15.22:0/610194103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad4076230 0x7f5ad4074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 --2- 172.21.15.22:0/610194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad40750e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 -- 172.21.15.22:0/610194103 >> 172.21.15.22:0/610194103 conn(0x7f5ad40fd7a0 msgr2=0x7f5ad40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 -- 172.21.15.22:0/610194103 shutdown_connections 2024-03-24T06:39:36.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.377+0000 7f5ad9800700 1 -- 172.21.15.22:0/610194103 wait complete. 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.379+0000 7f5ad9800700 1 Processor -- start 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.379+0000 7f5ad9800700 1 -- start start 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.379+0000 7f5ad9800700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.379+0000 7f5ad9800700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad4076230 0x7f5ad419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad9800700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 0x7f5ad41a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:36.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad9800700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41a1fb0 con 0x7f5ad4076230 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad9800700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41a20f0 con 0x7f5ad410a110 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad9800700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41a2280 con 0x7f5ad4074c60 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40690/0 (socket says 172.21.15.22:40690) 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 -- 172.21.15.22:0/235600264 learned_addr learned my addr 172.21.15.22:0/235600264 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5acbe00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad4076230 0x7f5ad419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad3400700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 0x7f5ad41a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 -- 172.21.15.22:0/235600264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 msgr2=0x7f5ad41a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 0x7f5ad41a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 -- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad4076230 msgr2=0x7f5ad419a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad4076230 0x7f5ad419a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad2a00700 1 -- 172.21.15.22:0/235600264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc0095f0 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.380+0000 7f5ad3400700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 0x7f5ad41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5ad2a00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400e9a0 tx=0x7f5ac400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4009790 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac4004d10 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad41a2560 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40056f0 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.381+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a2ab0 con 0x7f5ad4074c60 2024-03-24T06:39:36.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.382+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad4108ed0 con 0x7f5ad4074c60 2024-03-24T06:39:36.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.384+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5ac4004750 con 0x7f5ad4074c60 2024-03-24T06:39:36.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.385+0000 7f5acb400700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 0x7f5ab80a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:36.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.386+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f5ac4014070 con 0x7f5ad4074c60 2024-03-24T06:39:36.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.386+0000 7f5acbe00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 0x7f5ab80a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:36.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.386+0000 7f5acbe00700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 0x7f5ab80a4690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00b650 tx=0x7f5abc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:36.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.388+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5ac40972b0 con 0x7f5ad4074c60 2024-03-24T06:39:36.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.654+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f5ad410fda0 con 0x7f5ad4074c60 2024-03-24T06:39:36.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.655+0000 7f5acb400700 1 -- 172.21.15.22:0/235600264 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3096 (secure 0 0 0) 0x7f5ac4004e80 con 0x7f5ad4074c60 2024-03-24T06:39:36.657 INFO:teuthology.orchestra.run.smithi022.stdout:e28 2024-03-24T06:39:36.657 INFO:teuthology.orchestra.run.smithi022.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:legacy client fscid: 1 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:Filesystem 'cephfs' (1) 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:fs_name cephfs 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 28 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T06:39:36.658 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-24T06:36:56.216020+0000 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:modified 2024-03-24T06:37:33.427816+0000 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:tableserver 0 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:root 0 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:session_timeout 300 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:session_autoclose 300 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:max_file_size 1099511627776 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:max_xattr_size 65536 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:required_client_features {} 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure 0 2024-03-24T06:39:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure_osd_epoch 0 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:max_mds 5 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:in 0,1,2,3,4 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:up {0=24601,1=24584,2=24565,3=24544,4=24532} 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:failed 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:damaged 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:stopped 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:data_pools [3,4] 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:metadata_pool 2 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:inline_data disabled 2024-03-24T06:39:36.660 INFO:teuthology.orchestra.run.smithi022.stdout:balancer 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:bal_rank_mask -1 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:standby_count_wanted 0 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:qdb_cluster leader: 24601 members: 24584,24601,24532,24544,24565 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.a{0:24601} state up:active seq 4 addr [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.h{0:24530} state up:standby-replay seq 1 addr [v2:172.21.15.143:6836/988781970,v1:172.21.15.143:6837/988781970] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.b{1:24584} state up:active seq 11 addr [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.e{1:24506} state up:standby-replay seq 1 addr [v2:172.21.15.143:6834/1050031463,v1:172.21.15.143:6835/1050031463] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.i{2:24565} state up:active seq 14 addr [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.j{2:24496} state up:standby-replay seq 1 addr [v2:172.21.15.22:6834/298241489,v1:172.21.15.22:6835/298241489] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.c{3:24544} state up:active seq 16 addr [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.661 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.l{3:24469} state up:standby-replay seq 1 addr [v2:172.21.15.155:6832/715907814,v1:172.21.15.155:6833/715907814] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.f{4:24532} state up:active seq 17 addr [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.k{4:24458} state up:standby-replay seq 1 addr [v2:172.21.15.143:6832/3570497194,v1:172.21.15.143:6833/3570497194] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout:Standby daemons: 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.d{-1:14394} state up:standby seq 1 addr [v2:172.21.15.22:6836/4113284449,v1:172.21.15.22:6837/4113284449] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.662 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.g{-1:14445} state up:standby seq 1 addr [v2:172.21.15.22:6838/2318828115,v1:172.21.15.22:6839/2318828115] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T06:39:36.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 msgr2=0x7f5ab80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:36.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 0x7f5ab80a4690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00b650 tx=0x7f5abc005fd0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 msgr2=0x7f5ad419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:36.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400e9a0 tx=0x7f5ac400ed60 comp rx=0 tx=0).stop 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 shutdown_connections 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ab80a21d0 0x7f5ab80a4690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ad410a110 0x7f5ad41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ad4074c60 0x7f5ad419a420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 --2- 172.21.15.22:0/235600264 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ad4076230 0x7f5ad419a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 >> 172.21.15.22:0/235600264 conn(0x7f5ad40fd7a0 msgr2=0x7f5ad4106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 shutdown_connections 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:36.658+0000 7f5ad9800700 1 -- 172.21.15.22:0/235600264 wait complete. 2024-03-24T06:39:36.664 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:36 smithi143 ceph-mon[36520]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:37.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/235600264' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T06:39:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:36 smithi155 ceph-mon[36262]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:37.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/235600264' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T06:39:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:36 smithi022 ceph-mon[32003]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:37.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/235600264' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T06:39:37.603 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T06:39:38.094 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:39.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:38 smithi143 ceph-mon[36520]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:39.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:38 smithi155 ceph-mon[36262]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:39.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:38 smithi022 ceph-mon[32003]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:40.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 -- 172.21.15.22:0/3552048908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe11410e820 msgr2=0x7fe114110c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:40.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 --2- 172.21.15.22:0/3552048908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe11410e820 0x7fe114110c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00b0a0 tx=0x7fe10c00b3b0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 -- 172.21.15.22:0/3552048908 shutdown_connections 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 --2- 172.21.15.22:0/3552048908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe11410e820 0x7fe114110c10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 --2- 172.21.15.22:0/3552048908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe114076230 0x7fe114074690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.319+0000 7fe11a000700 1 --2- 172.21.15.22:0/3552048908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114074c60 0x7fe1140750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.320+0000 7fe11a000700 1 -- 172.21.15.22:0/3552048908 >> 172.21.15.22:0/3552048908 conn(0x7fe1140fd7a0 msgr2=0x7fe1140ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:40.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.320+0000 7fe11a000700 1 -- 172.21.15.22:0/3552048908 shutdown_connections 2024-03-24T06:39:40.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.320+0000 7fe11a000700 1 -- 172.21.15.22:0/3552048908 wait complete. 2024-03-24T06:39:40.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.321+0000 7fe11a000700 1 Processor -- start 2024-03-24T06:39:40.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.321+0000 7fe11a000700 1 -- start start 2024-03-24T06:39:40.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114076230 0x7fe11419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 0x7fe1141a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe1141a64c0 con 0x7fe114076230 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe1141a6600 con 0x7fe11410e820 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe11a000700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe1141a6790 con 0x7fe114074c60 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe113400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe113400700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40700/0 (socket says 172.21.15.22:40700) 2024-03-24T06:39:40.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe113400700 1 -- 172.21.15.22:0/2445343424 learned_addr learned my addr 172.21.15.22:0/2445343424 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe112a00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114076230 0x7fe11419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.322+0000 7fe113e00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 0x7fe1141a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 -- 172.21.15.22:0/2445343424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 msgr2=0x7fe11419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 -- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114076230 msgr2=0x7fe11419ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114076230 0x7fe11419ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 -- 172.21.15.22:0/2445343424 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c009d00 con 0x7fe11410e820 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113400700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe113e00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 0x7fe1141a5ef0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe10c007ab0 tx=0x7fe10c007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00e040 con 0x7fe11410e820 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10c00be40 con 0x7fe11410e820 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1141a6a10 con 0x7fe11410e820 2024-03-24T06:39:40.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c01b6e0 con 0x7fe11410e820 2024-03-24T06:39:40.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.323+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1141a6e20 con 0x7fe11410e820 2024-03-24T06:39:40.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.325+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe114104a80 con 0x7fe11410e820 2024-03-24T06:39:40.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.326+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe10c01d470 con 0x7fe11410e820 2024-03-24T06:39:40.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.328+0000 7fe10be00700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 0x7fe0f80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:40.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.328+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fe10c0ceee0 con 0x7fe11410e820 2024-03-24T06:39:40.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.328+0000 7fe113400700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 0x7fe0f80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:40.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.329+0000 7fe113400700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 0x7fe0f80a4610 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe104005950 tx=0x7fe104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:40.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.332+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe10c097b60 con 0x7fe11410e820 2024-03-24T06:39:40.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.579+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fe1141141f0 con 0x7fe11410e820 2024-03-24T06:39:40.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.580+0000 7fe10be00700 1 -- 172.21.15.22:0/2445343424 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5629 (secure 0 0 0) 0x7fe10c09b220 con 0x7fe11410e820 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 93 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-24T06:27:47.343581+0000 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:modified 2024-03-24T06:38:57.766765+0000 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:crush_version 27 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:full_ratio 0.95 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:backfillfull_ratio 0.9 2024-03-24T06:39:40.582 INFO:teuthology.orchestra.run.smithi022.stdout:nearfull_ratio 0.85 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:require_min_compat_client luminous 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:min_compat_client luminous 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:require_osd_release squid 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:stretch_mode_enabled false 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-24T06:39:40.583 INFO:teuthology.orchestra.run.smithi022.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:max_osd 12 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6802/671685493,v1:172.21.15.22:6803/671685493] [v2:172.21.15.22:6804/671685493,v1:172.21.15.22:6805/671685493] exists,up b50e1d7a-25e6-45e4-8f2e-51198cd3c017 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6800/2665804124,v1:172.21.15.143:6801/2665804124] [v2:172.21.15.143:6802/2665804124,v1:172.21.15.143:6803/2665804124] exists,up 572a8c88-6fa5-4e41-9c09-562f3cdd36f3 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6800/2561722475,v1:172.21.15.155:6801/2561722475] [v2:172.21.15.155:6802/2561722475,v1:172.21.15.155:6803/2561722475] exists,up 43917ed1-92cb-42b5-98e4-a92deee31e5a 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6810/3163117945,v1:172.21.15.22:6811/3163117945] [v2:172.21.15.22:6812/3163117945,v1:172.21.15.22:6813/3163117945] exists,up 85fc0b5c-cb79-4dc9-84c8-95968df70ee2 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6808/4206631935,v1:172.21.15.143:6809/4206631935] [v2:172.21.15.143:6810/4206631935,v1:172.21.15.143:6811/4206631935] exists,up c774c865-b9c3-4054-a328-e0c3a3c84d56 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/455132245,v1:172.21.15.155:6809/455132245] [v2:172.21.15.155:6810/455132245,v1:172.21.15.155:6811/455132245] exists,up c149b822-fc96-49aa-bb16-e1318e3572e4 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6818/3272271064,v1:172.21.15.22:6819/3272271064] [v2:172.21.15.22:6820/3272271064,v1:172.21.15.22:6821/3272271064] exists,up 7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5 2024-03-24T06:39:40.584 INFO:teuthology.orchestra.run.smithi022.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6816/1253244511,v1:172.21.15.143:6817/1253244511] [v2:172.21.15.143:6818/1253244511,v1:172.21.15.143:6819/1253244511] exists,up 1f669e98-5416-456f-8f7c-e3d8fd6608c6 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6816/3178166768,v1:172.21.15.155:6817/3178166768] [v2:172.21.15.155:6818/3178166768,v1:172.21.15.155:6819/3178166768] exists,up 75386ae9-abf1-4061-b5a6-ae879ab75235 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6826/2177893113,v1:172.21.15.22:6827/2177893113] [v2:172.21.15.22:6828/2177893113,v1:172.21.15.22:6829/2177893113] exists,up 0937ff6c-3ac5-4da0-b56c-f8286d053eff 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6824/2026852023,v1:172.21.15.143:6825/2026852023] [v2:172.21.15.143:6826/2026852023,v1:172.21.15.143:6827/2026852023] exists,up 095f0ee4-d45c-48ea-961d-f60c8c0869bb 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6824/2681113949,v1:172.21.15.155:6825/2681113949] [v2:172.21.15.155:6826/2681113949,v1:172.21.15.155:6827/2681113949] exists,up 4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 2.4 [2,11] 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 2.f [8,11] 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 2.1b [2,4] 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 3.1 [4,9] 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 3.e [5,9] 2024-03-24T06:39:40.585 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 3.21 [4,8] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 3.2c [4,9] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 4.e [9,11] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 4.10 [9,8] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 4.12 [5,10] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:pg_upmap_items 4.13 [9,8] 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/896325545 expires 2024-03-25T06:28:54.941425+0000 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/2695260292 expires 2024-03-25T06:28:54.941425+0000 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:6801/2009113619 expires 2024-03-25T06:28:54.941425+0000 2024-03-24T06:39:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:6800/2009113619 expires 2024-03-25T06:28:54.941425+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/2447624484 expires 2024-03-25T06:28:20.396453+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:6800/37818756 expires 2024-03-25T06:28:20.396453+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:6801/37818756 expires 2024-03-25T06:28:20.396453+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/1811358368 expires 2024-03-25T06:28:20.396453+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/3441957019 expires 2024-03-25T06:28:20.396453+0000 2024-03-24T06:39:40.587 INFO:teuthology.orchestra.run.smithi022.stdout:blocklist 172.21.15.22:0/1555230471 expires 2024-03-25T06:28:54.941425+0000 2024-03-24T06:39:40.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.583+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 msgr2=0x7fe0f80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 0x7fe0f80a4610 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe104005950 tx=0x7fe104009260 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 msgr2=0x7fe1141a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 0x7fe1141a5ef0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe10c007ab0 tx=0x7fe10c007930 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 shutdown_connections 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fe0f80a2150 0x7fe0f80a4610 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe11410e820 0x7fe1141a5ef0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe114074c60 0x7fe11419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 --2- 172.21.15.22:0/2445343424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe114076230 0x7fe11419ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:40.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 >> 172.21.15.22:0/2445343424 conn(0x7fe1140fd7a0 msgr2=0x7fe114101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:40.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 shutdown_connections 2024-03-24T06:39:40.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:40.584+0000 7fe11a000700 1 -- 172.21.15.22:0/2445343424 wait complete. 2024-03-24T06:39:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:40 smithi143 ceph-mon[36520]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:41.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:40 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2445343424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T06:39:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:40 smithi155 ceph-mon[36262]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:41.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:40 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2445343424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T06:39:41.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:40 smithi022 ceph-mon[32003]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T06:39:41.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:40 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2445343424' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T06:39:41.504 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T06:39:41.517 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.744+0000 7ff1b4000640 1 -- 172.21.15.22:0/2982353197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac103950 msgr2=0x7ff1ac103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.744+0000 7ff1b4000640 1 --2- 172.21.15.22:0/2982353197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac103950 0x7ff1ac103d70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009900 tx=0x7ff19c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 -- 172.21.15.22:0/2982353197 shutdown_connections 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 --2- 172.21.15.22:0/2982353197 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac105500 0x7ff1ac10bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 --2- 172.21.15.22:0/2982353197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac104b40 0x7ff1ac104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 --2- 172.21.15.22:0/2982353197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac103950 0x7ff1ac103d70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 -- 172.21.15.22:0/2982353197 >> 172.21.15.22:0/2982353197 conn(0x7ff1ac0fef00 msgr2=0x7ff1ac101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 -- 172.21.15.22:0/2982353197 shutdown_connections 2024-03-24T06:39:41.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.745+0000 7ff1b4000640 1 -- 172.21.15.22:0/2982353197 wait complete. 2024-03-24T06:39:41.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.746+0000 7ff1b4000640 1 Processor -- start 2024-03-24T06:39:41.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.746+0000 7ff1b4000640 1 -- start start 2024-03-24T06:39:41.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 0x7ff1ac199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 0x7ff1ac199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 0x7ff1ac1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1a1510 con 0x7ff1ac103950 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1a1680 con 0x7ff1ac104b40 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b4000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1a1840 con 0x7ff1ac105500 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b1600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 0x7ff1ac199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 0x7ff1ac1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b1600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 0x7ff1ac199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50508/0 (socket says 172.21.15.22:50508) 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b1600640 1 -- 172.21.15.22:0/111146102 learned_addr learned my addr 172.21.15.22:0/111146102 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.747+0000 7ff1b0c00640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 0x7ff1ac199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 -- 172.21.15.22:0/111146102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 msgr2=0x7ff1ac199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:41.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 0x7ff1ac199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 -- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 msgr2=0x7ff1ac199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 0x7ff1ac199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 -- 172.21.15.22:0/111146102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c009560 con 0x7ff1ac105500 2024-03-24T06:39:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b0c00640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 0x7ff1ac199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:41.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b2000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 0x7ff1ac1a0fd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400da10 tx=0x7ff1a400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400bd70 con 0x7ff1ac105500 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac1a1b20 con 0x7ff1ac105500 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a4003e50 con 0x7ff1ac105500 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40106c0 con 0x7ff1ac105500 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.748+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac1a1fe0 con 0x7ff1ac105500 2024-03-24T06:39:41.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.750+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac10ac70 con 0x7ff1ac105500 2024-03-24T06:39:41.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.751+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ff1a4004000 con 0x7ff1ac105500 2024-03-24T06:39:41.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.753+0000 7ff1aa000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 0x7ff1940a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:41.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.753+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ff1a40ce3f0 con 0x7ff1ac105500 2024-03-24T06:39:41.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.753+0000 7ff1b1600640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 0x7ff1940a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:41.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.754+0000 7ff1b1600640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 0x7ff1940a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006010 tx=0x7ff19c02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.757+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1a4096fe0 con 0x7ff1ac105500 2024-03-24T06:39:41.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.964+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1ac10f370 con 0x7ff1ac105500 2024-03-24T06:39:41.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.966+0000 7ff1aa000640 1 -- 172.21.15.22:0/111146102 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7ff1a409a6a0 con 0x7ff1ac105500 2024-03-24T06:39:41.966 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:41.966 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 msgr2=0x7ff1940a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 0x7ff1940a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff19c006010 tx=0x7ff19c02c7f0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 msgr2=0x7ff1ac1a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 0x7ff1ac1a0fd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400da10 tx=0x7ff1a400ded0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 shutdown_connections 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff1940a20c0 0x7ff1940a4580 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1ac105500 0x7ff1ac1a0fd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1ac104b40 0x7ff1ac199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 --2- 172.21.15.22:0/111146102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1ac103950 0x7ff1ac199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 >> 172.21.15.22:0/111146102 conn(0x7ff1ac0fef00 msgr2=0x7ff1ac107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 shutdown_connections 2024-03-24T06:39:41.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:41.969+0000 7ff1b4000640 1 -- 172.21.15.22:0/111146102 wait complete. 2024-03-24T06:39:41.990 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:39:41.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 -- 172.21.15.22:0/2222078351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 msgr2=0x7fb96c110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 --2- 172.21.15.22:0/2222078351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c110050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb964009900 tx=0x7fb96402bc50 comp rx=0 tx=0).stop 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 -- 172.21.15.22:0/2222078351 shutdown_connections 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 --2- 172.21.15.22:0/2222078351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c110050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 --2- 172.21.15.22:0/2222078351 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c108e00 0x7fb96c109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 --2- 172.21.15.22:0/2222078351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c107c10 0x7fb96c108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 -- 172.21.15.22:0/2222078351 >> 172.21.15.22:0/2222078351 conn(0x7fb96c076bb0 msgr2=0x7fb96c078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:42.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 -- 172.21.15.22:0/2222078351 shutdown_connections 2024-03-24T06:39:42.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.214+0000 7fb971200640 1 -- 172.21.15.22:0/2222078351 wait complete. 2024-03-24T06:39:42.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.215+0000 7fb971200640 1 Processor -- start 2024-03-24T06:39:42.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 -- start start 2024-03-24T06:39:42.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 0x7fb96c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c1a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c1a57e0 con 0x7fb96c107c10 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c1a5950 con 0x7fb96c1097c0 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb971200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c1a5b10 con 0x7fb96c108e00 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb96b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 0x7fb96c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb962a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb96b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 0x7fb96c19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50524/0 (socket says 172.21.15.22:50524) 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb962a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40734/0 (socket says 172.21.15.22:40734) 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb96b400640 1 -- 172.21.15.22:0/3816844629 learned_addr learned my addr 172.21.15.22:0/3816844629 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.216+0000 7fb96be00640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c1a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 -- 172.21.15.22:0/3816844629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 msgr2=0x7fb96c1a5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c1a5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 -- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 msgr2=0x7fb96c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 0x7fb96c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 -- 172.21.15.22:0/3816844629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb964009560 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb962a00640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb954005fd0 tx=0x7fb95400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954009c30 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb954004510 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.217+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c1a5df0 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.218+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954003e70 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.218+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c1a6270 con 0x7fb96c108e00 2024-03-24T06:39:42.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.219+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb96c10ee60 con 0x7fb96c108e00 2024-03-24T06:39:42.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.222+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fb954004000 con 0x7fb96c108e00 2024-03-24T06:39:42.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.224+0000 7fb968c00640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 0x7fb9500a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.224+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fb9540cdb10 con 0x7fb96c108e00 2024-03-24T06:39:42.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.224+0000 7fb96b400640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 0x7fb9500a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.225+0000 7fb96b400640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 0x7fb9500a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb96c19f1d0 tx=0x7fb95c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:42.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.232+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb954096720 con 0x7fb96c108e00 2024-03-24T06:39:42.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.432+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb96c02d1d0 con 0x7fb96c108e00 2024-03-24T06:39:42.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.433+0000 7fb968c00640 1 -- 172.21.15.22:0/3816844629 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fb954098650 con 0x7fb96c108e00 2024-03-24T06:39:42.433 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:39:42.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 msgr2=0x7fb9500a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 0x7fb9500a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb96c19f1d0 tx=0x7fb95c009500 comp rx=0 tx=0).stop 2024-03-24T06:39:42.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 msgr2=0x7fb96c19e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb954005fd0 tx=0x7fb95400dde0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 shutdown_connections 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fb9500a20c0 0x7fb9500a4580 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb96c1097c0 0x7fb96c1a5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb96c108e00 0x7fb96c19e240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 --2- 172.21.15.22:0/3816844629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb96c107c10 0x7fb96c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 >> 172.21.15.22:0/3816844629 conn(0x7fb96c076bb0 msgr2=0x7fb96c10c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.435+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 shutdown_connections 2024-03-24T06:39:42.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.436+0000 7fb971200640 1 -- 172.21.15.22:0/3816844629 wait complete. 2024-03-24T06:39:42.455 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:39:42.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.666+0000 7efd19c00640 1 -- 172.21.15.22:0/3394930854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1410dbe0 msgr2=0x7efd1410ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.666+0000 7efd19c00640 1 --2- 172.21.15.22:0/3394930854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1410dbe0 0x7efd1410ffd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009c10 tx=0x7efd0c02bef0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 -- 172.21.15.22:0/3394930854 shutdown_connections 2024-03-24T06:39:42.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 --2- 172.21.15.22:0/3394930854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd1410dbe0 0x7efd1410ffd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 --2- 172.21.15.22:0/3394930854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd14074170 0x7efd140745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 --2- 172.21.15.22:0/3394930854 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14075750 0x7efd14073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 -- 172.21.15.22:0/3394930854 >> 172.21.15.22:0/3394930854 conn(0x7efd140fcb80 msgr2=0x7efd140fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:42.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 -- 172.21.15.22:0/3394930854 shutdown_connections 2024-03-24T06:39:42.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.667+0000 7efd19c00640 1 -- 172.21.15.22:0/3394930854 wait complete. 2024-03-24T06:39:42.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 Processor -- start 2024-03-24T06:39:42.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 -- start start 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14074170 0x7efd1419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 0x7efd141a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efd141a57a0 con 0x7efd14074170 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efd141a5910 con 0x7efd14075750 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd19c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd141a5ad0 con 0x7efd1410dbe0 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.668+0000 7efd13e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14074170 0x7efd1419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd19200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 0x7efd141a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43846/0 (socket says 172.21.15.22:43846) 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 -- 172.21.15.22:0/1501661108 learned_addr learned my addr 172.21.15.22:0/1501661108 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 -- 172.21.15.22:0/1501661108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 msgr2=0x7efd141a5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 0x7efd141a5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 -- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14074170 msgr2=0x7efd1419dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14074170 0x7efd1419dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 -- 172.21.15.22:0/1501661108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c009840 con 0x7efd14075750 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd19200640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 0x7efd141a5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd0be00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e9f0 tx=0x7efcfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:42.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.669+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00ccc0 con 0x7efd14075750 2024-03-24T06:39:42.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.670+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc00ce20 con 0x7efd14075750 2024-03-24T06:39:42.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.670+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc010690 con 0x7efd14075750 2024-03-24T06:39:42.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.670+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd141a5db0 con 0x7efd14075750 2024-03-24T06:39:42.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.670+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd141a6260 con 0x7efd14075750 2024-03-24T06:39:42.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.671+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd14103db0 con 0x7efd14075750 2024-03-24T06:39:42.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.672+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7efcfc0046c0 con 0x7efd14075750 2024-03-24T06:39:42.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.675+0000 7efd11600640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 0x7efcf80a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:42.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.675+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7efcfc014070 con 0x7efd14075750 2024-03-24T06:39:42.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.675+0000 7efd13e00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 0x7efcf80a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:42.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.676+0000 7efd13e00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 0x7efcf80a4600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd1419eb60 tx=0x7efd04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:42.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.678+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efcfc096ea0 con 0x7efd14075750 2024-03-24T06:39:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.894+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd1402d130 con 0x7efd14075750 2024-03-24T06:39:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.895+0000 7efd11600640 1 -- 172.21.15.22:0/1501661108 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7efcfc09a560 con 0x7efd14075750 2024-03-24T06:39:42.895 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 msgr2=0x7efcf80a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 0x7efcf80a4600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efd1419eb60 tx=0x7efd04005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 msgr2=0x7efd1419e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e9f0 tx=0x7efcfc00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 shutdown_connections 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7efcf80a2140 0x7efcf80a4600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd1410dbe0 0x7efd141a5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14075750 0x7efd1419e1d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 --2- 172.21.15.22:0/1501661108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efd14074170 0x7efd1419dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 >> 172.21.15.22:0/1501661108 conn(0x7efd140fcb80 msgr2=0x7efd14100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 shutdown_connections 2024-03-24T06:39:42.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:42.898+0000 7efd19c00640 1 -- 172.21.15.22:0/1501661108 wait complete. 2024-03-24T06:39:42.917 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 -- 172.21.15.22:0/550171179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8570100840 msgr2=0x7f8570100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 --2- 172.21.15.22:0/550171179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8570100840 0x7f8570100ca0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f855c009900 tx=0x7f855c02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 -- 172.21.15.22:0/550171179 shutdown_connections 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 --2- 172.21.15.22:0/550171179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 0x7f8570101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 --2- 172.21.15.22:0/550171179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8570100840 0x7f8570100ca0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 --2- 172.21.15.22:0/550171179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f85700ffee0 0x7f8570100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 -- 172.21.15.22:0/550171179 >> 172.21.15.22:0/550171179 conn(0x7f85700fba80 msgr2=0x7f85700fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 -- 172.21.15.22:0/550171179 shutdown_connections 2024-03-24T06:39:43.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.159+0000 7f8576000640 1 -- 172.21.15.22:0/550171179 wait complete. 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 Processor -- start 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 -- start start 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8570100840 0x7f8570199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 0x7f85701a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f85701a1360 con 0x7f8570100840 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f85701a14d0 con 0x7f85701011e0 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.160+0000 7f8576000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f85701a1690 con 0x7f85700ffee0 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40772/0 (socket says 172.21.15.22:40772) 2024-03-24T06:39:43.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 -- 172.21.15.22:0/2464982909 learned_addr learned my addr 172.21.15.22:0/2464982909 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f856fe00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8570100840 0x7f8570199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8575600640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 0x7f85701a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 -- 172.21.15.22:0/2464982909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 msgr2=0x7f85701a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 0x7f85701a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 -- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8570100840 msgr2=0x7f8570199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8570100840 0x7f8570199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.161+0000 7f8574c00640 1 -- 172.21.15.22:0/2464982909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c009560 con 0x7f85700ffee0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f8574c00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f856400e9f0 tx=0x7f856400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564009700 con 0x7f85700ffee0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f856400cd80 con 0x7f85700ffee0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85701a1970 con 0x7f85700ffee0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564018920 con 0x7f85700ffee0 2024-03-24T06:39:43.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.162+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85701a1ea0 con 0x7f85700ffee0 2024-03-24T06:39:43.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.164+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8570068420 con 0x7f85700ffee0 2024-03-24T06:39:43.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.165+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f8564018a80 con 0x7f85700ffee0 2024-03-24T06:39:43.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.167+0000 7f856d600640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 0x7f85580a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.167+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f8564014070 con 0x7f85700ffee0 2024-03-24T06:39:43.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.167+0000 7f856fe00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 0x7f85580a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.168+0000 7f856fe00640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 0x7f85580a4600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c02c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:43.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.170+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8564053280 con 0x7f85700ffee0 2024-03-24T06:39:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:42 smithi143 ceph-mon[36520]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:42 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/111146102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:42 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3816844629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:42 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1501661108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:42 smithi155 ceph-mon[36262]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:43.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:42 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/111146102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:43.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:42 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3816844629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:42 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1501661108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.371+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f857010ac30 con 0x7f85700ffee0 2024-03-24T06:39:43.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.372+0000 7f856d600640 1 -- 172.21.15.22:0/2464982909 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f8564096ff0 con 0x7f85700ffee0 2024-03-24T06:39:43.371 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:39:43.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 msgr2=0x7f85580a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 0x7f85580a4600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f855c000c00 tx=0x7f855c02c880 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 msgr2=0x7f8570199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f856400e9f0 tx=0x7f856400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 shutdown_connections 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f85580a2140 0x7f85580a4600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85701011e0 0x7f85701a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8570100840 0x7f8570199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 --2- 172.21.15.22:0/2464982909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85700ffee0 0x7f8570199870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 >> 172.21.15.22:0/2464982909 conn(0x7f85700fba80 msgr2=0x7f85700fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.374+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 shutdown_connections 2024-03-24T06:39:43.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.375+0000 7f8576000640 1 -- 172.21.15.22:0/2464982909 wait complete. 2024-03-24T06:39:43.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:42 smithi022 ceph-mon[32003]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:43.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/111146102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:43.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3816844629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.390 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:42 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1501661108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:43.393 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:39:43.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 -- 172.21.15.22:0/4264528170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47901031c0 msgr2=0x7f4790109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 --2- 172.21.15.22:0/4264528170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47901031c0 0x7f4790109a50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4788009ca0 tx=0x7f478802c100 comp rx=0 tx=0).stop 2024-03-24T06:39:43.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 -- 172.21.15.22:0/4264528170 shutdown_connections 2024-03-24T06:39:43.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 --2- 172.21.15.22:0/4264528170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47901031c0 0x7f4790109a50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 --2- 172.21.15.22:0/4264528170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 --2- 172.21.15.22:0/4264528170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4790101610 0x7f4790101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 -- 172.21.15.22:0/4264528170 >> 172.21.15.22:0/4264528170 conn(0x7f47900fcb80 msgr2=0x7f47900fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:43.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 -- 172.21.15.22:0/4264528170 shutdown_connections 2024-03-24T06:39:43.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.633+0000 7f4797000640 1 -- 172.21.15.22:0/4264528170 wait complete. 2024-03-24T06:39:43.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 Processor -- start 2024-03-24T06:39:43.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 -- start start 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 0x7f47901a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f47901a12e0 con 0x7f47901031c0 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f47901a1450 con 0x7f4790102800 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.635+0000 7f4797000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47901a1610 con 0x7f4790101610 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40792/0 (socket says 172.21.15.22:40792) 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 -- 172.21.15.22:0/3726175426 learned_addr learned my addr 172.21.15.22:0/3726175426 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4796600640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 0x7f47901a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 -- 172.21.15.22:0/3726175426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 msgr2=0x7f4790199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 -- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 msgr2=0x7f47901a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 0x7f47901a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 -- 172.21.15.22:0/3726175426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4788009900 con 0x7f4790101610 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4796600640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 0x7f47901a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795200640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.636+0000 7f4795c00640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f478400d770 tx=0x7f478400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.637+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784009c30 con 0x7f4790101610 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.637+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478400de80 con 0x7f4790101610 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.637+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784010460 con 0x7f4790101610 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.637+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47901a18f0 con 0x7f4790101610 2024-03-24T06:39:43.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.637+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47901a1e50 con 0x7f4790101610 2024-03-24T06:39:43.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.638+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479010d020 con 0x7f4790101610 2024-03-24T06:39:43.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.640+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f4784003f60 con 0x7f4790101610 2024-03-24T06:39:43.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.641+0000 7f478e000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 0x7f47780a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:43.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.641+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f47840cdd10 con 0x7f4790101610 2024-03-24T06:39:43.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.641+0000 7f4795200640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 0x7f47780a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:43.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.642+0000 7f4795200640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 0x7f47780a4600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f479019ad00 tx=0x7f4780005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.650+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f47840d3020 con 0x7f4790101610 2024-03-24T06:39:43.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.909+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f479010d1b0 con 0x7f4790101610 2024-03-24T06:39:43.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.910+0000 7f478e000640 1 -- 172.21.15.22:0/3726175426 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4784096900 con 0x7f4790101610 2024-03-24T06:39:43.910 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 msgr2=0x7f47780a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 0x7f47780a4600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f479019ad00 tx=0x7f4780005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 msgr2=0x7f47901997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f478400d770 tx=0x7f478400dc30 comp rx=0 tx=0).stop 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 shutdown_connections 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f47780a2140 0x7f47780a4600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47901031c0 0x7f47901a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4790102800 0x7f4790199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 --2- 172.21.15.22:0/3726175426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4790101610 0x7f47901997f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:43.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 >> 172.21.15.22:0/3726175426 conn(0x7f47900fcb80 msgr2=0x7f4790073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:43.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 shutdown_connections 2024-03-24T06:39:43.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:43.913+0000 7f4797000640 1 -- 172.21.15.22:0/3726175426 wait complete. 2024-03-24T06:39:43.937 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.150+0000 7ffac4400640 1 -- 172.21.15.22:0/1025789043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc1021d0 msgr2=0x7ffabc1045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.150+0000 7ffac4400640 1 --2- 172.21.15.22:0/1025789043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc1021d0 0x7ffabc1045c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009900 tx=0x7ffab002bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 -- 172.21.15.22:0/1025789043 shutdown_connections 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 --2- 172.21.15.22:0/1025789043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc107430 0x7ffabc109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 --2- 172.21.15.22:0/1025789043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc104b00 0x7ffabc106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 --2- 172.21.15.22:0/1025789043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc1021d0 0x7ffabc1045c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 -- 172.21.15.22:0/1025789043 >> 172.21.15.22:0/1025789043 conn(0x7ffabc0fbbc0 msgr2=0x7ffabc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 -- 172.21.15.22:0/1025789043 shutdown_connections 2024-03-24T06:39:44.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.151+0000 7ffac4400640 1 -- 172.21.15.22:0/1025789043 wait complete. 2024-03-24T06:39:44.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.152+0000 7ffac4400640 1 Processor -- start 2024-03-24T06:39:44.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 -- start start 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc104b00 0x7ffabc199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 0x7ffabc1a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1a1530 con 0x7ffabc104b00 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1a16a0 con 0x7ffabc107430 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.153+0000 7ffac4400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc1a1860 con 0x7ffabc1021d0 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc104b00 0x7ffabc199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40810/0 (socket says 172.21.15.22:40810) 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 -- 172.21.15.22:0/1780133282 learned_addr learned my addr 172.21.15.22:0/1780133282 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac2400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 0x7ffabc1a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 -- 172.21.15.22:0/1780133282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 msgr2=0x7ffabc1a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 0x7ffabc1a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 -- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc104b00 msgr2=0x7ffabc199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc104b00 0x7ffabc199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac1a00640 1 -- 172.21.15.22:0/1780133282 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab0009560 con 0x7ffabc1021d0 2024-03-24T06:39:44.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.154+0000 7ffac2400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 0x7ffabc1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffac1a00640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffab002c6a0 tx=0x7ffab00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab00044c0 con 0x7ffabc1021d0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab00052d0 con 0x7ffabc1021d0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0003e80 con 0x7ffabc1021d0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffabc1a1ae0 con 0x7ffabc1021d0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.155+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffabc1a2040 con 0x7ffabc1021d0 2024-03-24T06:39:44.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.156+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffabc108f80 con 0x7ffabc1021d0 2024-03-24T06:39:44.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.160+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ffab0005440 con 0x7ffabc1021d0 2024-03-24T06:39:44.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.161+0000 7ffaba000640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 0x7ffaa40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.162+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ffab00f09b0 con 0x7ffabc1021d0 2024-03-24T06:39:44.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.162+0000 7ffac1000640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 0x7ffaa40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.162+0000 7ffac1000640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 0x7ffaa40a4600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffabc19af50 tx=0x7ffaa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:44.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.165+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffab00b96c0 con 0x7ffabc1021d0 2024-03-24T06:39:44.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:43 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2464982909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:43 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3726175426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2464982909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:43 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3726175426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.409+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffabc10d1b0 con 0x7ffabc1021d0 2024-03-24T06:39:44.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.410+0000 7ffaba000640 1 -- 172.21.15.22:0/1780133282 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7ffab0043020 con 0x7ffabc1021d0 2024-03-24T06:39:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:44.410 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:39:44.411 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.412+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 msgr2=0x7ffaa40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.412+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 0x7ffaa40a4600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffabc19af50 tx=0x7ffaa8005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.412+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 msgr2=0x7ffabc199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.412+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffab002c6a0 tx=0x7ffab00040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 shutdown_connections 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ffaa40a2140 0x7ffaa40a4600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffabc107430 0x7ffabc1a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffabc104b00 0x7ffabc199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 --2- 172.21.15.22:0/1780133282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffabc1021d0 0x7ffabc199a40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 >> 172.21.15.22:0/1780133282 conn(0x7ffabc0fbbc0 msgr2=0x7ffabc1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:44.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 shutdown_connections 2024-03-24T06:39:44.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.413+0000 7ffac4400640 1 -- 172.21.15.22:0/1780133282 wait complete. 2024-03-24T06:39:44.427 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:43 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2464982909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:43 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3726175426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:44.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.656+0000 7fcf43000640 1 -- 172.21.15.22:0/1476353983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c103950 msgr2=0x7fcf3c103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.656+0000 7fcf43000640 1 --2- 172.21.15.22:0/1476353983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c103950 0x7fcf3c103d70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009900 tx=0x7fcf2c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 -- 172.21.15.22:0/1476353983 shutdown_connections 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 --2- 172.21.15.22:0/1476353983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c105500 0x7fcf3c10bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 --2- 172.21.15.22:0/1476353983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c104b40 0x7fcf3c104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 --2- 172.21.15.22:0/1476353983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c103950 0x7fcf3c103d70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 -- 172.21.15.22:0/1476353983 >> 172.21.15.22:0/1476353983 conn(0x7fcf3c0fef00 msgr2=0x7fcf3c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:44.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 -- 172.21.15.22:0/1476353983 shutdown_connections 2024-03-24T06:39:44.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.657+0000 7fcf43000640 1 -- 172.21.15.22:0/1476353983 wait complete. 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.658+0000 7fcf43000640 1 Processor -- start 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.658+0000 7fcf43000640 1 -- start start 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.658+0000 7fcf43000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.658+0000 7fcf43000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 0x7fcf3c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.658+0000 7fcf43000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c105500 0x7fcf3c1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf43000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c1a1500 con 0x7fcf3c105500 2024-03-24T06:39:44.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf43000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c1a1670 con 0x7fcf3c104b40 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf43000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c1a1830 con 0x7fcf3c103950 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf41000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c105500 0x7fcf3c1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 0x7fcf3c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40826/0 (socket says 172.21.15.22:40826) 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 -- 172.21.15.22:0/3097922090 learned_addr learned my addr 172.21.15.22:0/3097922090 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 -- 172.21.15.22:0/3097922090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 msgr2=0x7fcf3c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 0x7fcf3c199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 -- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c105500 msgr2=0x7fcf3c1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c105500 0x7fcf3c1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3be00640 1 -- 172.21.15.22:0/3097922090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2c009560 con 0x7fcf3c103950 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.659+0000 7fcf3b400640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 0x7fcf3c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:44.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf3be00640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c000c00 tx=0x7fcf2c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:44.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0043d0 con 0x7fcf3c103950 2024-03-24T06:39:44.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf2c0052d0 con 0x7fcf3c103950 2024-03-24T06:39:44.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3c1a1ab0 con 0x7fcf3c103950 2024-03-24T06:39:44.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c003da0 con 0x7fcf3c103950 2024-03-24T06:39:44.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.660+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf3c1a1fc0 con 0x7fcf3c103950 2024-03-24T06:39:44.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.661+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf3c068950 con 0x7fcf3c103950 2024-03-24T06:39:44.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.663+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fcf2c005440 con 0x7fcf3c103950 2024-03-24T06:39:44.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.664+0000 7fcf38c00640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 0x7fcf200a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:44.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.664+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fcf2c0f0810 con 0x7fcf3c103950 2024-03-24T06:39:44.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.665+0000 7fcf3b400640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 0x7fcf200a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:44.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.666+0000 7fcf3b400640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 0x7fcf200a4580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c19af20 tx=0x7fcf28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:44.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.668+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf2c0b9520 con 0x7fcf3c103950 2024-03-24T06:39:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.878+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf3c02d2f0 con 0x7fcf3c103950 2024-03-24T06:39:44.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.880+0000 7fcf38c00640 1 -- 172.21.15.22:0/3097922090 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7fcf2c043020 con 0x7fcf3c103950 2024-03-24T06:39:44.881 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:44.881 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 msgr2=0x7fcf200a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 0x7fcf200a4580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c19af20 tx=0x7fcf28005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 msgr2=0x7fcf3c199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c000c00 tx=0x7fcf2c005e80 comp rx=0 tx=0).stop 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 shutdown_connections 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fcf200a20c0 0x7fcf200a4580 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcf3c105500 0x7fcf3c1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf3c104b40 0x7fcf3c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 --2- 172.21.15.22:0/3097922090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf3c103950 0x7fcf3c199a10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 >> 172.21.15.22:0/3097922090 conn(0x7fcf3c0fef00 msgr2=0x7fcf3c107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 shutdown_connections 2024-03-24T06:39:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:44.883+0000 7fcf43000640 1 -- 172.21.15.22:0/3097922090 wait complete. 2024-03-24T06:39:44.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:39:44.904 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:39:45.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 -- 172.21.15.22:0/3543121429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754101610 msgr2=0x7f6754101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 --2- 172.21.15.22:0/3543121429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754101610 0x7f6754101a30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6744009900 tx=0x7f674402bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 -- 172.21.15.22:0/3543121429 shutdown_connections 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 --2- 172.21.15.22:0/3543121429 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f6754109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 --2- 172.21.15.22:0/3543121429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754102800 0x7f6754102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 --2- 172.21.15.22:0/3543121429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754101610 0x7f6754101a30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 -- 172.21.15.22:0/3543121429 >> 172.21.15.22:0/3543121429 conn(0x7f67540fcb80 msgr2=0x7f67540fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 -- 172.21.15.22:0/3543121429 shutdown_connections 2024-03-24T06:39:45.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.123+0000 7f6759400640 1 -- 172.21.15.22:0/3543121429 wait complete. 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 Processor -- start 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 -- start start 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754102800 0x7f6754199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f67541a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f67541a13e0 con 0x7f67541031c0 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67541a1550 con 0x7f6754101610 2024-03-24T06:39:45.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.125+0000 7f6759400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f67541a1710 con 0x7f6754102800 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35306/0 (socket says 172.21.15.22:35306) 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 -- 172.21.15.22:0/3583840079 learned_addr learned my addr 172.21.15.22:0/3583840079 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754102800 0x7f6754199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6758a00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f67541a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 -- 172.21.15.22:0/3583840079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754102800 msgr2=0x7f6754199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754102800 0x7f6754199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 -- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 msgr2=0x7f67541a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f67541a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 -- 172.21.15.22:0/3583840079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6744009560 con 0x7f6754101610 2024-03-24T06:39:45.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6758a00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f67541a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.126+0000 7f6753e00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6744000c00 tx=0x7f67440040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.127+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67440044c0 con 0x7f6754101610 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.127+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67440052d0 con 0x7f6754101610 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.127+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67541a1990 con 0x7f6754101610 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.127+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674403e450 con 0x7f6754101610 2024-03-24T06:39:45.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.127+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67541a1ef0 con 0x7f6754101610 2024-03-24T06:39:45.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.128+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675410d040 con 0x7f6754101610 2024-03-24T06:39:45.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.130+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f6744005440 con 0x7f6754101610 2024-03-24T06:39:45.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.131+0000 7f6750c00640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 0x7f67380a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.131+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f67440f08e0 con 0x7f6754101610 2024-03-24T06:39:45.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.131+0000 7f6753400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 0x7f67380a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.132+0000 7f6753400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 0x7f67380a4600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f675419ae00 tx=0x7f6740005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:45.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.135+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f67440b95f0 con 0x7f6754101610 2024-03-24T06:39:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:44 smithi155 ceph-mon[36262]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:44 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1780133282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:44 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3097922090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.337+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67541a2190 con 0x7f6754101610 2024-03-24T06:39:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.338+0000 7f6750c00640 1 -- 172.21.15.22:0/3583840079 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f6744043070 con 0x7f6754101610 2024-03-24T06:39:45.338 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:39:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 msgr2=0x7f67380a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 0x7f67380a4600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f675419ae00 tx=0x7f6740005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 msgr2=0x7f67541998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6744000c00 tx=0x7f67440040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 shutdown_connections 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f67380a2140 0x7f67380a4600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f67541031c0 0x7f67541a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6754102800 0x7f6754199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 --2- 172.21.15.22:0/3583840079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6754101610 0x7f67541998f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 >> 172.21.15.22:0/3583840079 conn(0x7f67540fcb80 msgr2=0x7f6754073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 shutdown_connections 2024-03-24T06:39:45.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.341+0000 7f6759400640 1 -- 172.21.15.22:0/3583840079 wait complete. 2024-03-24T06:39:45.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:44 smithi022 ceph-mon[32003]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:45.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:44 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1780133282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.358 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:44 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3097922090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.362 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:39:45.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:44 smithi143 ceph-mon[36520]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:39:45.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:44 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1780133282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:44 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3097922090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.573+0000 7fc814c00640 1 -- 172.21.15.22:0/3827925810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810101720 msgr2=0x7fc810101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.573+0000 7fc814c00640 1 --2- 172.21.15.22:0/3827925810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810101720 0x7fc810101b40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc800009900 tx=0x7fc80002bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 -- 172.21.15.22:0/3827925810 shutdown_connections 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 --2- 172.21.15.22:0/3827925810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc810109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 --2- 172.21.15.22:0/3827925810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810102910 0x7fc810102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 --2- 172.21.15.22:0/3827925810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810101720 0x7fc810101b40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 -- 172.21.15.22:0/3827925810 >> 172.21.15.22:0/3827925810 conn(0x7fc8100fcc90 msgr2=0x7fc8100ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:45.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 -- 172.21.15.22:0/3827925810 shutdown_connections 2024-03-24T06:39:45.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.574+0000 7fc814c00640 1 -- 172.21.15.22:0/3827925810 wait complete. 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.575+0000 7fc814c00640 1 Processor -- start 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.575+0000 7fc814c00640 1 -- start start 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.575+0000 7fc814c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.575+0000 7fc814c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 0x7fc810197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc814c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc81019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc814c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc81019f310 con 0x7fc810101720 2024-03-24T06:39:45.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc814c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc81019f480 con 0x7fc810102910 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc814c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc81019f640 con 0x7fc8101032d0 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 0x7fc810197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59406/0 (socket says 172.21.15.22:59406) 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80de00640 1 -- 172.21.15.22:0/3585554057 learned_addr learned my addr 172.21.15.22:0/3585554057 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80e800640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc81019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 -- 172.21.15.22:0/3585554057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 msgr2=0x7fc81019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc81019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 -- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 msgr2=0x7fc810197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80d400640 1 -- 172.21.15.22:0/3585554057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc800009560 con 0x7fc810102910 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80e800640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc81019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.576+0000 7fc80de00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc80d400640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 0x7fc810197d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc005fd0 tx=0x7fc7fc00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc009c30 con 0x7fc810102910 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc004510 con 0x7fc810102910 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81019f920 con 0x7fc810102910 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc003e60 con 0x7fc810102910 2024-03-24T06:39:45.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.577+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81019fe80 con 0x7fc810102910 2024-03-24T06:39:45.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.578+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc810108ae0 con 0x7fc810102910 2024-03-24T06:39:45.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.580+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fc7fc004000 con 0x7fc810102910 2024-03-24T06:39:45.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.582+0000 7fc806a00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 0x7fc7f40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:45.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.582+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fc7fc0cdae0 con 0x7fc810102910 2024-03-24T06:39:45.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.582+0000 7fc80de00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 0x7fc7f40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:45.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.583+0000 7fc80de00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 0x7fc7f40a4600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc800006010 tx=0x7fc80002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:45.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.585+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7fc096780 con 0x7fc810102910 2024-03-24T06:39:45.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.831+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc81002d130 con 0x7fc810102910 2024-03-24T06:39:45.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.832+0000 7fc806a00640 1 -- 172.21.15.22:0/3585554057 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fc7fc099e40 con 0x7fc810102910 2024-03-24T06:39:45.832 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:45.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.834+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 msgr2=0x7fc7f40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.834+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 0x7fc7f40a4600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc800006010 tx=0x7fc80002c7e0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.834+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 msgr2=0x7fc810197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:45.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.834+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 0x7fc810197d60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc005fd0 tx=0x7fc7fc00dde0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 shutdown_connections 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fc7f40a2140 0x7fc7f40a4600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8101032d0 0x7fc81019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc810102910 0x7fc810197d60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 --2- 172.21.15.22:0/3585554057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc810101720 0x7fc810197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 >> 172.21.15.22:0/3585554057 conn(0x7fc8100fcc90 msgr2=0x7fc810105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 shutdown_connections 2024-03-24T06:39:45.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:45.835+0000 7fc814c00640 1 -- 172.21.15.22:0/3585554057 wait complete. 2024-03-24T06:39:45.853 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:39:46.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 -- 172.21.15.22:0/4273077614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 msgr2=0x7fbcfc109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 --2- 172.21.15.22:0/4273077614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc109850 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009ca0 tx=0x7fbcf402c100 comp rx=0 tx=0).stop 2024-03-24T06:39:46.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 -- 172.21.15.22:0/4273077614 shutdown_connections 2024-03-24T06:39:46.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 --2- 172.21.15.22:0/4273077614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc109850 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 --2- 172.21.15.22:0/4273077614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 --2- 172.21.15.22:0/4273077614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 -- 172.21.15.22:0/4273077614 >> 172.21.15.22:0/4273077614 conn(0x7fbcfc0fbbb0 msgr2=0x7fbcfc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:46.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 -- 172.21.15.22:0/4273077614 shutdown_connections 2024-03-24T06:39:46.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.073+0000 7fbd04600640 1 -- 172.21.15.22:0/4273077614 wait complete. 2024-03-24T06:39:46.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.074+0000 7fbd04600640 1 Processor -- start 2024-03-24T06:39:46.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.074+0000 7fbd04600640 1 -- start start 2024-03-24T06:39:46.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc19b650 con 0x7fbcfc1021a0 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc100bd0 con 0x7fbcfc104ad0 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd04600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc100d60 con 0x7fbcfc107400 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd01200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd01c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd01200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35352/0 (socket says 172.21.15.22:35352) 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:32902/0 (socket says 172.21.15.22:32902) 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd01200640 1 -- 172.21.15.22:0/2054656509 learned_addr learned my addr 172.21.15.22:0/2054656509 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 -- 172.21.15.22:0/2054656509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 msgr2=0x7fbcfc100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc100150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 -- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 msgr2=0x7fbcfc101ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc101ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd02600640 1 -- 172.21.15.22:0/2054656509 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf4009900 con 0x7fbcfc107400 2024-03-24T06:39:46.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.075+0000 7fbd01c00640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:46.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbd02600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4037040 tx=0x7fbcf4032fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4034640 con 0x7fbcfc107400 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf4003e50 con 0x7fbcfc107400 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4040450 con 0x7fbcfc107400 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc1a6010 con 0x7fbcfc107400 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.076+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc1a6380 con 0x7fbcfc107400 2024-03-24T06:39:46.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.077+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcfc068950 con 0x7fbcfc107400 2024-03-24T06:39:46.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.079+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fbcf400a040 con 0x7fbcfc107400 2024-03-24T06:39:46.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.080+0000 7fbcfa000640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 0x7fbce40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.081+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fbcf40f09b0 con 0x7fbcfc107400 2024-03-24T06:39:46.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.081+0000 7fbd01c00640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 0x7fbce40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.081+0000 7fbd01c00640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 0x7fbce40a4600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009fc0 tx=0x7fbcf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:46.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.084+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbcf40b95a0 con 0x7fbcfc107400 2024-03-24T06:39:46.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:45 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3583840079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:45 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3585554057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.290+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbcfc10d110 con 0x7fbcfc107400 2024-03-24T06:39:46.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.290+0000 7fbcfa000640 1 -- 172.21.15.22:0/2054656509 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fbcf40bcc60 con 0x7fbcfc107400 2024-03-24T06:39:46.291 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:39:46.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 msgr2=0x7fbce40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 0x7fbce40a4600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009fc0 tx=0x7fbcf0005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:46.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 msgr2=0x7fbcfc100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4037040 tx=0x7fbcf4032fc0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 shutdown_connections 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fbce40a2140 0x7fbce40a4600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcfc107400 0x7fbcfc100690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbcfc104ad0 0x7fbcfc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 --2- 172.21.15.22:0/2054656509 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcfc1021a0 0x7fbcfc101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 >> 172.21.15.22:0/2054656509 conn(0x7fbcfc0fbbb0 msgr2=0x7fbcfc102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 shutdown_connections 2024-03-24T06:39:46.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.294+0000 7fbd04600640 1 -- 172.21.15.22:0/2054656509 wait complete. 2024-03-24T06:39:46.311 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:45 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3583840079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.312 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:45 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3585554057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.314 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:39:46.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:45 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3583840079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:45 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3585554057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 -- 172.21.15.22:0/4011640733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 msgr2=0x7f96b0109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 --2- 172.21.15.22:0/4011640733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b0109850 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009ca0 tx=0x7f96a802c100 comp rx=0 tx=0).stop 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 -- 172.21.15.22:0/4011640733 shutdown_connections 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 --2- 172.21.15.22:0/4011640733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b0109850 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 --2- 172.21.15.22:0/4011640733 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 0x7f96b0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 --2- 172.21.15.22:0/4011640733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b0104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.529+0000 7f96b6200640 1 -- 172.21.15.22:0/4011640733 >> 172.21.15.22:0/4011640733 conn(0x7f96b00fbbb0 msgr2=0x7f96b00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.530+0000 7f96b6200640 1 -- 172.21.15.22:0/4011640733 shutdown_connections 2024-03-24T06:39:46.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.530+0000 7f96b6200640 1 -- 172.21.15.22:0/4011640733 wait complete. 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.530+0000 7f96b6200640 1 Processor -- start 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 -- start start 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 0x7f96b0199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b01a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01a14d0 con 0x7f96b0104ad0 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01a1640 con 0x7f96b01021a0 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96b6200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01a1800 con 0x7f96b0107400 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96af400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96aea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 0x7f96b0199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96afe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b01a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96af400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35370/0 (socket says 172.21.15.22:35370) 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.531+0000 7f96af400640 1 -- 172.21.15.22:0/2266781377 learned_addr learned my addr 172.21.15.22:0/2266781377 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 -- 172.21.15.22:0/2266781377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 msgr2=0x7f96b01999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 -- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 msgr2=0x7f96b0199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 0x7f96b0199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 -- 172.21.15.22:0/2266781377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a0009490 con 0x7f96b0107400 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96af400640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96afe00640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b01a0f90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96a8037040 tx=0x7f96a8032fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8034640 con 0x7f96b0107400 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96a8003e50 con 0x7f96b0107400 2024-03-24T06:39:46.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.532+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a8009900 con 0x7f96b0107400 2024-03-24T06:39:46.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.533+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8040450 con 0x7f96b0107400 2024-03-24T06:39:46.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.533+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01a1dc0 con 0x7f96b0107400 2024-03-24T06:39:46.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.534+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b0108ef0 con 0x7f96b0107400 2024-03-24T06:39:46.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.536+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f96a800a040 con 0x7f96b0107400 2024-03-24T06:39:46.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.539+0000 7f96a7e00640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 0x7f96940a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:46.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.539+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f96a80f08f0 con 0x7f96b0107400 2024-03-24T06:39:46.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.539+0000 7f96af400640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 0x7f96940a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:46.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.540+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f96a80b95c0 con 0x7f96b0107400 2024-03-24T06:39:46.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.540+0000 7f96af400640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 0x7f96940a4600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011690 tx=0x7f96a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:46.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.741+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f96b010d110 con 0x7f96b0107400 2024-03-24T06:39:46.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.742+0000 7f96a7e00640 1 -- 172.21.15.22:0/2266781377 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f96a80440a0 con 0x7f96b0107400 2024-03-24T06:39:46.742 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:39:46.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 msgr2=0x7f96940a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 0x7f96940a4600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011690 tx=0x7f96a0005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 msgr2=0x7f96b01a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b01a0f90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f96a8037040 tx=0x7f96a8032fc0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 shutdown_connections 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f96940a2140 0x7f96940a4600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96b0107400 0x7f96b01a0f90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f96b0104ad0 0x7f96b0199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 --2- 172.21.15.22:0/2266781377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96b01021a0 0x7f96b01999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 >> 172.21.15.22:0/2266781377 conn(0x7f96b00fbbb0 msgr2=0x7f96b01018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.744+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 shutdown_connections 2024-03-24T06:39:46.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.745+0000 7f96b6200640 1 -- 172.21.15.22:0/2266781377 wait complete. 2024-03-24T06:39:46.767 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:39:46.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 -- 172.21.15.22:0/300636348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 msgr2=0x7f2360103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:46.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 --2- 172.21.15.22:0/300636348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360103d70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2350009900 tx=0x7f235002bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:46.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 -- 172.21.15.22:0/300636348 shutdown_connections 2024-03-24T06:39:46.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 --2- 172.21.15.22:0/300636348 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 0x7f236010bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:46.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 --2- 172.21.15.22:0/300636348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 0x7f2360104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 --2- 172.21.15.22:0/300636348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360103d70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 -- 172.21.15.22:0/300636348 >> 172.21.15.22:0/300636348 conn(0x7f23600feec0 msgr2=0x7f2360101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 -- 172.21.15.22:0/300636348 shutdown_connections 2024-03-24T06:39:47.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:46.999+0000 7f2365200640 1 -- 172.21.15.22:0/300636348 wait complete. 2024-03-24T06:39:47.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.000+0000 7f2365200640 1 Processor -- start 2024-03-24T06:39:47.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 -- start start 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 0x7f2360199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 0x7f23601a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f23601a1550 con 0x7f2360105500 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f23601a16c0 con 0x7f2360104b40 2024-03-24T06:39:47.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2365200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f23601a1880 con 0x7f2360103950 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f235de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f235de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:32924/0 (socket says 172.21.15.22:32924) 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f235de00640 1 -- 172.21.15.22:0/1752457313 learned_addr learned my addr 172.21.15.22:0/1752457313 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.001+0000 7f2357e00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 0x7f2360199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235e800640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 0x7f23601a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 -- 172.21.15.22:0/1752457313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 msgr2=0x7f2360199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 0x7f2360199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 -- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 msgr2=0x7f23601a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 0x7f23601a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 -- 172.21.15.22:0/1752457313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2350009560 con 0x7f2360103950 2024-03-24T06:39:47.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f235de00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2350000c00 tx=0x7f23500040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23500044c0 con 0x7f2360103950 2024-03-24T06:39:47.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23500052d0 con 0x7f2360103950 2024-03-24T06:39:47.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.002+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350003e30 con 0x7f2360103950 2024-03-24T06:39:47.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.003+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601a1b00 con 0x7f2360103950 2024-03-24T06:39:47.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.003+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601a1fc0 con 0x7f2360103950 2024-03-24T06:39:47.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.004+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236010ac70 con 0x7f2360103950 2024-03-24T06:39:47.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.005+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f2350005440 con 0x7f2360103950 2024-03-24T06:39:47.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.006+0000 7f2356a00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 0x7f23440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.007+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f23500f0900 con 0x7f2360103950 2024-03-24T06:39:47.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.007+0000 7f2357e00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 0x7f23440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.008+0000 7f2357e00640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 0x7f23440a4580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f236019af70 tx=0x7f2348005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.010+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f23500b9610 con 0x7f2360103950 2024-03-24T06:39:47.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:46 smithi022 ceph-mon[32003]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:47.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:46 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2054656509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:46 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2266781377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.253+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f236010a6e0 con 0x7f2360103950 2024-03-24T06:39:47.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.254+0000 7f2356a00640 1 -- 172.21.15.22:0/1752457313 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11939 (secure 0 0 0) 0x7f2350043020 con 0x7f2360103950 2024-03-24T06:39:47.255 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:47.255 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:37:33.427816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:39:47.255 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 28 2024-03-24T06:39:47.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.257+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 msgr2=0x7f23440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.257+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 0x7f23440a4580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f236019af70 tx=0x7f2348005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:47.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.257+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 msgr2=0x7f2360199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.257+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2350000c00 tx=0x7f23500040c0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 shutdown_connections 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f23440a20c0 0x7f23440a4580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2360105500 0x7f23601a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2360104b40 0x7f2360199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 --2- 172.21.15.22:0/1752457313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2360103950 0x7f2360199a60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 >> 172.21.15.22:0/1752457313 conn(0x7f23600feec0 msgr2=0x7f2360107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 shutdown_connections 2024-03-24T06:39:47.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.258+0000 7f2365200640 1 -- 172.21.15.22:0/1752457313 wait complete. 2024-03-24T06:39:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:46 smithi155 ceph-mon[36262]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:46 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2054656509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:46 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2266781377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:46 smithi143 ceph-mon[36520]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:39:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:46 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2054656509' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:46 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2266781377' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:39:47.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 -- 172.21.15.22:0/282081539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98104910 msgr2=0x7f0d98106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/282081539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98104910 0x7f0d98106d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009900 tx=0x7f0d8002bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 -- 172.21.15.22:0/282081539 shutdown_connections 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/282081539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98107240 0x7f0d98109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/282081539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98104910 0x7f0d98106d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/282081539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98101fe0 0x7f0d981043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 -- 172.21.15.22:0/282081539 >> 172.21.15.22:0/282081539 conn(0x7f0d980fb9f0 msgr2=0x7f0d980fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.503+0000 7f0d9dc00640 1 -- 172.21.15.22:0/282081539 shutdown_connections 2024-03-24T06:39:47.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.504+0000 7f0d9dc00640 1 -- 172.21.15.22:0/282081539 wait complete. 2024-03-24T06:39:47.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.505+0000 7f0d9dc00640 1 Processor -- start 2024-03-24T06:39:47.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.505+0000 7f0d9dc00640 1 -- start start 2024-03-24T06:39:47.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98107240 0x7f0d9819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9819f140 con 0x7f0d98107240 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9819f2b0 con 0x7f0d98104910 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9dc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9819f470 con 0x7f0d98101fe0 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d9d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98107240 0x7f0d9819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d97e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d8fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d97e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:32950/0 (socket says 172.21.15.22:32950) 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d8fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35408/0 (socket says 172.21.15.22:35408) 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.506+0000 7f0d97e00640 1 -- 172.21.15.22:0/2922549006 learned_addr learned my addr 172.21.15.22:0/2922549006 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:47.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 -- 172.21.15.22:0/2922549006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 msgr2=0x7f0d98197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 -- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98107240 msgr2=0x7f0d9819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98107240 0x7f0d9819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 -- 172.21.15.22:0/2922549006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d90009840 con 0x7f0d98104910 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d97e00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:47.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d8fe00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d80036040 tx=0x7f0d80005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d800043d0 con 0x7f0d98104910 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d800052d0 con 0x7f0d98104910 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d80009560 con 0x7f0d98104910 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.507+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d80003b60 con 0x7f0d98104910 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.508+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9819fa40 con 0x7f0d98104910 2024-03-24T06:39:47.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.509+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d980fd4e0 con 0x7f0d98104910 2024-03-24T06:39:47.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.512+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0d80003cc0 con 0x7f0d98104910 2024-03-24T06:39:47.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.514+0000 7f0d95600640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 0x7f0d7c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.514+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f0d800f07e0 con 0x7f0d98104910 2024-03-24T06:39:47.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.514+0000 7f0d97e00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 0x7f0d7c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.515+0000 7f0d97e00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 0x7f0d7c0a4580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d88011550 tx=0x7f0d88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.518+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d800b93d0 con 0x7f0d98104910 2024-03-24T06:39:47.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.721+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d9802d130 con 0x7f0d98104910 2024-03-24T06:39:47.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.723+0000 7f0d95600640 1 -- 172.21.15.22:0/2922549006 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7f0d800bca90 con 0x7f0d98104910 2024-03-24T06:39:47.724 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:47.724 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:47.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 msgr2=0x7f0d7c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 0x7f0d7c0a4580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d88011550 tx=0x7f0d88005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 msgr2=0x7f0d98197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0d80036040 tx=0x7f0d80005e80 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 shutdown_connections 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f0d7c0a20c0 0x7f0d7c0a4580 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0d98107240 0x7f0d9819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d98104910 0x7f0d98197b90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 --2- 172.21.15.22:0/2922549006 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0d98101fe0 0x7f0d98197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 >> 172.21.15.22:0/2922549006 conn(0x7f0d980fb9f0 msgr2=0x7f0d98102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 shutdown_connections 2024-03-24T06:39:47.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.726+0000 7f0d9dc00640 1 -- 172.21.15.22:0/2922549006 wait complete. 2024-03-24T06:39:47.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T06:36:56.216020+0000', 'modified': '2024-03-24T06:37:33.427816+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24601, 'mds_1': 24584, 'mds_2': 24565, 'mds_3': 24544, 'mds_4': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6833/3570497194', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 3570497194}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 3570497194}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6833/715907814', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6832', 'nonce': 715907814}, {'type': 'v1', 'addr': '172.21.15.155:6833', 'nonce': 715907814}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.22:6835/298241489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6834', 'nonce': 298241489}, {'type': 'v1', 'addr': '172.21.15.22:6835', 'nonce': 298241489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6835/1050031463', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1050031463}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1050031463}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/988781970', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 988781970}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 988781970}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.155:6835/1202752642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 1202752642}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 1202752642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.155:6837/1157072254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1157072254}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1157072254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24565': {'gid': 24565, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.155:6839/2828760693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 2828760693}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 2828760693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24584': {'gid': 24584, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/2325215622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 2325215622}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 2325215622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24601': {'gid': 24601, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.22:6841/857116957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6840', 'nonce': 857116957}, {'type': 'v1', 'addr': '172.21.15.22:6841', 'nonce': 857116957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24601, 'qdb_cluster': [24584, 24601, 24532, 24544, 24565]} 2024-03-24T06:39:47.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T06:39:47.754 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T06:39:47.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 -- 172.21.15.22:0/1331379977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 msgr2=0x7f125c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 --2- 172.21.15.22:0/1331379977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c105600 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1254009ca0 tx=0x7f125402c0f0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 -- 172.21.15.22:0/1331379977 shutdown_connections 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 --2- 172.21.15.22:0/1331379977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c105600 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 --2- 172.21.15.22:0/1331379977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 --2- 172.21.15.22:0/1331379977 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 -- 172.21.15.22:0/1331379977 >> 172.21.15.22:0/1331379977 conn(0x7f125c0fba80 msgr2=0x7f125c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 -- 172.21.15.22:0/1331379977 shutdown_connections 2024-03-24T06:39:47.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.966+0000 7f1262400640 1 -- 172.21.15.22:0/1331379977 wait complete. 2024-03-24T06:39:47.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.968+0000 7f1262400640 1 Processor -- start 2024-03-24T06:39:47.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.968+0000 7f1262400640 1 -- start start 2024-03-24T06:39:47.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f125c1a56c0 con 0x7f125c0ffe60 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f125c1a5830 con 0x7f125c1027d0 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1262400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f125c1a59f0 con 0x7f125c105180 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f125be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c19dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59484/0 (socket says 172.21.15.22:59484) 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f125be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35430/0 (socket says 172.21.15.22:35430) 2024-03-24T06:39:47.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:32974/0 (socket says 172.21.15.22:32974) 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261000640 1 -- 172.21.15.22:0/1518430537 learned_addr learned my addr 172.21.15.22:0/1518430537 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 -- 172.21.15.22:0/1518430537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 msgr2=0x7f125c19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 -- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 msgr2=0x7f125c19dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c19dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f1261a00640 1 -- 172.21.15.22:0/1518430537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248009490 con 0x7f125c105180 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.969+0000 7f125be00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:47.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1261a00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1254037040 tx=0x7f1254032f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1259600640 1 -- 172.21.15.22:0/1518430537 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254034e20 con 0x7f125c105180 2024-03-24T06:39:47.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1259600640 1 -- 172.21.15.22:0/1518430537 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1254003e50 con 0x7f125c105180 2024-03-24T06:39:47.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1254009900 con 0x7f125c105180 2024-03-24T06:39:47.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1259600640 1 -- 172.21.15.22:0/1518430537 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254004800 con 0x7f125c105180 2024-03-24T06:39:47.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.970+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c1a5f60 con 0x7f125c105180 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.973+0000 7f1259600640 1 -- 172.21.15.22:0/1518430537 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f125400a040 con 0x7f125c105180 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 Processor -- start 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 -- start start 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67e9bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67ea3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x557b67e0a050 con 0x557b67ea1280 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b67e07580 con 0x557b67e9b720 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1266612740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b67e02d60 con 0x557b67e9c110 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1261000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1261000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:32982/0 (socket says 172.21.15.22:32982) 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.974+0000 7f1261000640 1 -- 172.21.15.22:0/3063762925 learned_addr learned my addr 172.21.15.22:0/3063762925 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f125be00640 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67e9bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f125be00640 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67ea3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 -- 172.21.15.22:0/3063762925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 msgr2=0x557b67e9bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67e9bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 -- 172.21.15.22:0/3063762925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 msgr2=0x557b67ea3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67ea3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 -- 172.21.15.22:0/3063762925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b67dbb620 con 0x557b67e9c110 2024-03-24T06:39:47.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f1261000640 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1250009dc0 tx=0x7f1250011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5080b84b23bce74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.975+0000 7f124e000640 1 -- 172.21.15.22:0/3063762925 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250005220 con 0x557b67e9c110 2024-03-24T06:39:47.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f124e000640 1 -- 172.21.15.22:0/3063762925 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1250004c80 con 0x557b67e9c110 2024-03-24T06:39:47.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f124e000640 1 -- 172.21.15.22:0/3063762925 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250019890 con 0x557b67e9c110 2024-03-24T06:39:47.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1259600640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 0x7f12440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 -- 172.21.15.22:0/3063762925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 msgr2=0x557b67ea0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1250009dc0 tx=0x7f1250011810 comp rx=0 tx=0).stop 2024-03-24T06:39:47.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 -- 172.21.15.22:0/3063762925 shutdown_connections 2024-03-24T06:39:47.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67ea3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea0c10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 --2- 172.21.15.22:0/3063762925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67e9bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1259600640 1 -- 172.21.15.22:0/1518430537 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f12540f0a80 con 0x7f125c105180 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f1266612740 1 -- 172.21.15.22:0/3063762925 >> 172.21.15.22:0/3063762925 conn(0x557b67e96c90 msgr2=0x557b67e97090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.976+0000 7f125be00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 0x7f12440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.977+0000 7f1266612740 1 -- 172.21.15.22:0/3063762925 shutdown_connections 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.977+0000 7f1266612740 1 -- 172.21.15.22:0/3063762925 wait complete. 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.977+0000 7f1266612740 1 -- start start 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.977+0000 7f125be00640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 0x7f12440a4580 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x557b67e9c970 tx=0x7f1248005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.977+0000 7f1266612740 1 Processor -- start 2024-03-24T06:39:47.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 0x557b67ea5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67eace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x557b67da7fb0 con 0x557b67e9c110 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b67da7c50 con 0x557b67e9b720 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b67b4af20 con 0x557b67ea1280 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1266612740 10 client.0 _ll_register_callbacks cb 0x557b67b13ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f125be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 0x557b67ea5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1261000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1261a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67eace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1261000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35448/0 (socket says 172.21.15.22:35448) 2024-03-24T06:39:47.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f1261000640 1 -- 172.21.15.22:0/2814072516 learned_addr learned my addr 172.21.15.22:0/2814072516 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.978+0000 7f125be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 0x557b67ea5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59516/0 (socket says 172.21.15.22:59516) 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 -- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 msgr2=0x557b67eace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67eace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 -- 172.21.15.22:0/2814072516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 msgr2=0x557b67ea5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 0x557b67ea5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 -- 172.21.15.22:0/2814072516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12500099f0 con 0x557b67e9b720 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261a00640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67eace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1261000640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f125001b040 tx=0x7f12500038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852014b0e5dea09d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1242a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1250003ea0 con 0x557b67e9b720 2024-03-24T06:39:47.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f1250005c50 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.979+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b67b026c0 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f12500055a0 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 10 client.25355 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557b67e013e0 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557b67b026c0 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1250021040 con 0x557b67e9b720 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 10 client.25355 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 10 client.25355 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 20 client.25355 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67eb24c0 0x557b67eb3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:47.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 4 client.25355 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.980+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b67b53270 con 0x557b67eb24c0 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.981+0000 7f125be00640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67eb24c0 0x557b67eb3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.981+0000 7f125be00640 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67eb24c0 0x557b67eb3930 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=282effcb2a7b7e32 server_cookie=7aec3294e62aa467 in_seq=0 out_seq=0 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.981+0000 7f1242a00640 10 client.25355 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1242a00640 1 -- 172.21.15.22:0/2814072516 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b67b53270 con 0x557b67eb24c0 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1242a00640 10 client.25355 handle_command_reply: tid=0 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 client.25355 shutdown 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 msgr2=0x557b67ea58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f125001b040 tx=0x7f12500038a0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 shutdown_connections 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67eb24c0 0x557b67eb3930 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67eace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9c110 0x557b67ea5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 --2- 172.21.15.22:0/2814072516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea58d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 >> 172.21.15.22:0/2814072516 conn(0x557b67e96c90 msgr2=0x557b67e95dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1242a00640 0 client.25355 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 shutdown_connections 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 1 -- 172.21.15.22:0/2814072516 wait complete. 2024-03-24T06:39:47.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:47.982+0000 7f1266612740 20 client.25355 trim_cache size 0 max 16384 2024-03-24T06:39:48.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.002+0000 7f1266612740 1 Processor -- start 2024-03-24T06:39:48.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.002+0000 7f1266612740 1 -- start start 2024-03-24T06:39:48.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9b720 0x557b67e97070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9c110 0x557b67eacd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x557b67ea0dd0 con 0x557b67e9b720 2024-03-24T06:39:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b67b53270 con 0x557b67e9c110 2024-03-24T06:39:48.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1266612740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b67eb28a0 con 0x557b67ea1280 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9b720 0x557b67e97070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33014/0 (socket says 172.21.15.22:33014) 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f125be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9c110 0x557b67eacd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 -- 172.21.15.22:0/3215774541 learned_addr learned my addr 172.21.15.22:0/3215774541 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 -- 172.21.15.22:0/3215774541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9c110 msgr2=0x557b67eacd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9c110 0x557b67eacd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 -- 172.21.15.22:0/3215774541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9b720 msgr2=0x557b67e97070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9b720 0x557b67e97070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.003+0000 7f1261a00640 1 -- 172.21.15.22:0/3215774541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b67d9d900 con 0x557b67ea1280 2024-03-24T06:39:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.004+0000 7f1261a00640 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12540b9da0 tx=0x7f12540ee070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cd71dd84a9964b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.004+0000 7f124d600640 1 -- 172.21.15.22:0/3215774541 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12540bd3d0 con 0x557b67ea1280 2024-03-24T06:39:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.004+0000 7f124d600640 1 -- 172.21.15.22:0/3215774541 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12540c1520 con 0x557b67ea1280 2024-03-24T06:39:48.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.004+0000 7f124d600640 1 -- 172.21.15.22:0/3215774541 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254032ad0 con 0x557b67ea1280 2024-03-24T06:39:48.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- 172.21.15.22:0/3215774541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 msgr2=0x557b67e95540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12540b9da0 tx=0x7f12540ee070 comp rx=0 tx=0).stop 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- 172.21.15.22:0/3215774541 shutdown_connections 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67ea1280 0x557b67e95540 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9c110 0x557b67eacd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 --2- 172.21.15.22:0/3215774541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67e9b720 0x557b67e97070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- 172.21.15.22:0/3215774541 >> 172.21.15.22:0/3215774541 conn(0x557b67e96c90 msgr2=0x557b67eb1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- 172.21.15.22:0/3215774541 shutdown_connections 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- 172.21.15.22:0/3215774541 wait complete. 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.005+0000 7f1266612740 1 -- start start 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 Processor -- start 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67eabfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x557b67eac520 con 0x557b67ea1280 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b67eac690 con 0x557b67e9b720 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b67eac850 con 0x557b67e9c110 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.006+0000 7f1266612740 10 client.0 _ll_register_callbacks cb 0x7f1250009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33022/0 (socket says 172.21.15.22:33022) 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 -- 172.21.15.22:0/148616464 learned_addr learned my addr 172.21.15.22:0/148616464 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f1261a00640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67eabfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f1261000640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 -- 172.21.15.22:0/148616464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 msgr2=0x557b67ea8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 -- 172.21.15.22:0/148616464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 msgr2=0x557b67eabfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67eabfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 -- 172.21.15.22:0/148616464 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c1a5f60 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f125be00640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1248004540 tx=0x7f1248009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daba1700f3939a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f1242a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.007+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248026040 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1250004c80 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f124802cd90 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f124802c410 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 10 client.25399 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557b67dc3560 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1250004c80 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f12480205d0 con 0x557b67e9c110 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 10 client.25399 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 10 client.25399 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T06:39:48.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 20 client.25399 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67f1cf40 0x557b67f1bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 4 client.25399 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:48.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.008+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "status"}) -- 0x557b67b3f610 con 0x557b67f1cf40 2024-03-24T06:39:48.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1261000640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67f1cf40 0x557b67f1bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1261000640 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67f1cf40 0x557b67f1bc50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb035828651795f0 server_cookie=7b62b6082db0bec4 in_seq=0 out_seq=0 2024-03-24T06:39:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1242a00640 10 client.25399 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1242a00640 1 -- 172.21.15.22:0/148616464 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557b67b3f610 con 0x557b67f1cf40 2024-03-24T06:39:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1242a00640 10 client.25399 handle_command_reply: tid=0 2024-03-24T06:39:48.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.009+0000 7f1266612740 1 client.25399 shutdown 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 msgr2=0x557b67ea8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1248004540 tx=0x7f1248009770 comp rx=0 tx=0).stop 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 shutdown_connections 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x557b67f1cf40 0x557b67f1bc50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x557b67ea1280 0x557b67eabfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b67e9c110 0x557b67ea8fc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 --2- 172.21.15.22:0/148616464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b67e9b720 0x557b67ea8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 >> 172.21.15.22:0/148616464 conn(0x557b67e96c90 msgr2=0x557b67e97070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1242a00640 0 client.25399 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 shutdown_connections 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 1 -- 172.21.15.22:0/148616464 wait complete. 2024-03-24T06:39:48.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1266612740 20 client.25399 trim_cache size 0 max 16384 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 4, 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24532, 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 135.60329561200001, 2024-03-24T06:39:48.015 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:48.016 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:48.016 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:48.016 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 198.21692979599999 2024-03-24T06:39:48.016 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:48.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 msgr2=0x7f12440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.010+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 0x7f12440a4580 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x557b67e9c970 tx=0x7f1248005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 msgr2=0x7f125c1a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1254037040 tx=0x7f1254032f80 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 shutdown_connections 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f12440a20c0 0x7f12440a4580 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f125c105180 0x7f125c1a5180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f125c1027d0 0x7f125c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 --2- 172.21.15.22:0/1518430537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f125c0ffe60 0x7f125c19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 >> 172.21.15.22:0/1518430537 conn(0x7f125c0fba80 msgr2=0x7f125c1007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 shutdown_connections 2024-03-24T06:39:48.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.011+0000 7f1262400640 1 -- 172.21.15.22:0/1518430537 wait complete. 2024-03-24T06:39:48.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1752457313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.031 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:47 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2922549006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.035 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 -- 172.21.15.22:0/2937653701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0103950 msgr2=0x7f31c0103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 --2- 172.21.15.22:0/2937653701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0103950 0x7f31c0103d70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009900 tx=0x7f31b002bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 -- 172.21.15.22:0/2937653701 shutdown_connections 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 --2- 172.21.15.22:0/2937653701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c010bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 --2- 172.21.15.22:0/2937653701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0104b40 0x7f31c0104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 --2- 172.21.15.22:0/2937653701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0103950 0x7f31c0103d70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 -- 172.21.15.22:0/2937653701 >> 172.21.15.22:0/2937653701 conn(0x7f31c00feec0 msgr2=0x7f31c0101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 -- 172.21.15.22:0/2937653701 shutdown_connections 2024-03-24T06:39:48.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.259+0000 7f31c4a00640 1 -- 172.21.15.22:0/2937653701 wait complete. 2024-03-24T06:39:48.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 Processor -- start 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 -- start start 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 0x7f31c01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c01a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01a14b0 con 0x7f31c0105500 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01a1620 con 0x7f31c0103950 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.260+0000 7f31c4a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01a17e0 con 0x7f31c0104b40 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33044/0 (socket says 172.21.15.22:33044) 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bde00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 0x7f31c01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 -- 172.21.15.22:0/3715847138 learned_addr learned my addr 172.21.15.22:0/3715847138 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31be800640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c01a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 -- 172.21.15.22:0/3715847138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 msgr2=0x7f31c01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 0x7f31c01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 -- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 msgr2=0x7f31c01a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c01a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bd400640 1 -- 172.21.15.22:0/3715847138 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b0009560 con 0x7f31c0104b40 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31bde00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 0x7f31c01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.261+0000 7f31be800640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c01a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31bd400640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f31ac00d950 tx=0x7f31ac00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31b6a00640 1 -- 172.21.15.22:0/3715847138 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac009a30 con 0x7f31c0104b40 2024-03-24T06:39:48.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31b6a00640 1 -- 172.21.15.22:0/3715847138 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ac009b90 con 0x7f31c0104b40 2024-03-24T06:39:48.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31b6a00640 1 -- 172.21.15.22:0/3715847138 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac004a00 con 0x7f31c0104b40 2024-03-24T06:39:48.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c01a1ac0 con 0x7f31c0104b40 2024-03-24T06:39:48.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.262+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c01a1f80 con 0x7f31c0104b40 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.265+0000 7f31c778c740 1 Processor -- start 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.265+0000 7f31b6a00640 1 -- 172.21.15.22:0/3715847138 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f31ac009d00 con 0x7f31c0104b40 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.265+0000 7f31c778c740 1 -- start start 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.265+0000 7f31c778c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.265+0000 7f31c778c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31c778c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31c778c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e76e245050 con 0x55e76e2d6f40 2024-03-24T06:39:48.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31c778c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e76e242580 con 0x55e76e2dc0b0 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31c778c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e76e23dd60 con 0x55e76e2d6550 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33048/0 (socket says 172.21.15.22:33048) 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 -- 172.21.15.22:0/3977574889 learned_addr learned my addr 172.21.15.22:0/3977574889 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31bd400640 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31bde00640 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 -- 172.21.15.22:0/3977574889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 msgr2=0x55e76e2de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 -- 172.21.15.22:0/3977574889 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 msgr2=0x55e76e2dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2dba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.266+0000 7f31be800640 1 -- 172.21.15.22:0/3977574889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e76e1f6620 con 0x55e76e2d6550 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31be800640 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f31b8007e60 tx=0x7f31b800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c745b74446f2228c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31b6a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 0x7f31a40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31ab000640 1 -- 172.21.15.22:0/3977574889 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8004d20 con 0x55e76e2d6550 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31ab000640 1 -- 172.21.15.22:0/3977574889 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b8004e80 con 0x55e76e2d6550 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31ab000640 1 -- 172.21.15.22:0/3977574889 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8010640 con 0x55e76e2d6550 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31b6a00640 1 -- 172.21.15.22:0/3715847138 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f31ac0cda80 con 0x7f31c0104b40 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.267+0000 7f31bde00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 0x7f31a40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- 172.21.15.22:0/3977574889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 msgr2=0x55e76e2d6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f31b8007e60 tx=0x7f31b800e720 comp rx=0 tx=0).stop 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- 172.21.15.22:0/3977574889 shutdown_connections 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 --2- 172.21.15.22:0/3977574889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d6970 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- 172.21.15.22:0/3977574889 >> 172.21.15.22:0/3977574889 conn(0x55e76e2d1ac0 msgr2=0x55e76e2d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- 172.21.15.22:0/3977574889 shutdown_connections 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31bde00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 0x7f31a40a4600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f31b002c250 tx=0x7f31b002c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- 172.21.15.22:0/3977574889 wait complete. 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.268+0000 7f31c778c740 1 -- start start 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 Processor -- start 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2e06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e76e1e2fb0 con 0x55e76e2dc0b0 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e76e1e2c50 con 0x55e76e2d6f40 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e76df85f20 con 0x55e76e2d6550 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31c778c740 10 client.0 _ll_register_callbacks cb 0x55e76df4eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31bd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.269+0000 7f31bde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2e06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35492/0 (socket says 172.21.15.22:35492) 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 -- 172.21.15.22:0/576559138 learned_addr learned my addr 172.21.15.22:0/576559138 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31be800640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 -- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 msgr2=0x55e76e2e06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2e06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 -- 172.21.15.22:0/576559138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2dc0b0 msgr2=0x55e76e2e7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 -- 172.21.15.22:0/576559138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b8007ac0 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31bd400640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31ac096da0 tx=0x7f31ac096670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0fe4dde0b5e8309 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f319f400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac099d30 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f31ac09aad0 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.270+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e76df8e270 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31ac09a3d0 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 10 client.25373 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e76e23c3e0 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e76df8e270 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31ac09e120 con 0x55e76e2d6f40 2024-03-24T06:39:48.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 10 client.25373 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 10 client.25373 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 20 client.25373 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e2ed0f0 0x55e76e2ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 4 client.25373 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.271+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e76df7a610 con 0x55e76e2ed0f0 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.272+0000 7f31bde00640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e2ed0f0 0x55e76e2ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.272+0000 7f31bde00640 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e2ed0f0 0x55e76e2ee560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1f6cb886b7dacde9 server_cookie=392ee9a689207b95 in_seq=0 out_seq=0 2024-03-24T06:39:48.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.272+0000 7f319f400640 10 client.25373 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.273+0000 7f319f400640 1 -- 172.21.15.22:0/576559138 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e76df7a610 con 0x55e76e2ed0f0 2024-03-24T06:39:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.273+0000 7f319f400640 10 client.25373 handle_command_reply: tid=0 2024-03-24T06:39:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.273+0000 7f31c778c740 1 client.25373 shutdown 2024-03-24T06:39:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.273+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 msgr2=0x55e76e2e0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.273+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f31ac096da0 tx=0x7f31ac096670 comp rx=0 tx=0).stop 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 shutdown_connections 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e2ed0f0 0x55e76e2ee560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e0c10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 --2- 172.21.15.22:0/576559138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2e06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f319f400640 0 client.25373 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 >> 172.21.15.22:0/576559138 conn(0x55e76e2d1ac0 msgr2=0x55e76e2d0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 shutdown_connections 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 1 -- 172.21.15.22:0/576559138 wait complete. 2024-03-24T06:39:48.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.274+0000 7f31c778c740 20 client.25373 trim_cache size 0 max 16384 2024-03-24T06:39:48.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1752457313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:47 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2922549006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.296+0000 7f31c778c740 1 Processor -- start 2024-03-24T06:39:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.296+0000 7f31c778c740 1 -- start start 2024-03-24T06:39:48.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2dfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2d2a60 con 0x55e76e2d6f40 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2d2510 con 0x55e76e2dc0b0 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31c778c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2ecc50 con 0x55e76e2d6550 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33054/0 (socket says 172.21.15.22:33054) 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 -- 172.21.15.22:0/597163057 learned_addr learned my addr 172.21.15.22:0/597163057 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31be800640 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2dfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bd400640 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 -- 172.21.15.22:0/597163057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 msgr2=0x55e76e2dfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2dfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 -- 172.21.15.22:0/597163057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 msgr2=0x55e76e2e7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2e7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.297+0000 7f31bde00640 1 -- 172.21.15.22:0/597163057 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e76e1d8900 con 0x55e76e2d6550 2024-03-24T06:39:48.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.298+0000 7f31bde00640 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f31b002cde0 tx=0x7f31b0042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f742f6a2bd9483e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.298+0000 7f31aa600640 1 -- 172.21.15.22:0/597163057 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0051040 con 0x55e76e2d6550 2024-03-24T06:39:48.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.298+0000 7f31aa600640 1 -- 172.21.15.22:0/597163057 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31b0044070 con 0x55e76e2d6550 2024-03-24T06:39:48.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.298+0000 7f31aa600640 1 -- 172.21.15.22:0/597163057 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0003db0 con 0x55e76e2d6550 2024-03-24T06:39:48.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- 172.21.15.22:0/597163057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 msgr2=0x55e76e2d5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f31b002cde0 tx=0x7f31b0042000 comp rx=0 tx=0).stop 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- 172.21.15.22:0/597163057 shutdown_connections 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2dc0b0 0x55e76e2dfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6f40 0x55e76e2e7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 --2- 172.21.15.22:0/597163057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2d6550 0x55e76e2d5430 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- 172.21.15.22:0/597163057 >> 172.21.15.22:0/597163057 conn(0x55e76e2d1ac0 msgr2=0x55e76e2d4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- 172.21.15.22:0/597163057 shutdown_connections 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- 172.21.15.22:0/597163057 wait complete. 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 -- start start 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.299+0000 7f31c778c740 1 Processor -- start 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6550 0x55e76e2e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2e88d0 con 0x55e76e2d6550 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2e78b0 con 0x55e76e2d6f40 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e76e2e7a70 con 0x55e76e2dc0b0 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31c778c740 10 client.0 _ll_register_callbacks cb 0x7f31ac0d3220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31bd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31be800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31bd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35520/0 (socket says 172.21.15.22:35520) 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31bd400640 1 -- 172.21.15.22:0/3240513062 learned_addr learned my addr 172.21.15.22:0/3240513062 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.300+0000 7f31bde00640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6550 0x55e76e2e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 -- 172.21.15.22:0/3240513062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 msgr2=0x55e76e2e4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 -- 172.21.15.22:0/3240513062 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6550 msgr2=0x55e76e2e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6550 0x55e76e2e3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 -- 172.21.15.22:0/3240513062 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c010c690 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31bd400640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31be800640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000f80 tx=0x7f31b800ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb8dce27477fa216 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f319f400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b8003e40 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e76e2df570 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f31b8010850 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f31b8010a10 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.301+0000 7f31c778c740 10 client.25417 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e76e1fe560 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e76e2df570 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f31b8023070 con 0x55e76e2dc0b0 2024-03-24T06:39:48.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 10 client.25417 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 10 client.25417 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 20 client.25417 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e357f10 0x55e76e356c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 4 client.25417 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "status"}) -- 0x55e76df3d6c0 con 0x55e76e357f10 2024-03-24T06:39:48.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.302+0000 7f31bde00640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e357f10 0x55e76e356c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31bde00640 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e357f10 0x55e76e356c20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=de73145630af992c server_cookie=3fb25681abd24c37 in_seq=0 out_seq=0 2024-03-24T06:39:48.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f319f400640 10 client.25417 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:48.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f319f400640 1 -- 172.21.15.22:0/3240513062 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55e76df3d6c0 con 0x55e76e357f10 2024-03-24T06:39:48.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f319f400640 10 client.25417 handle_command_reply: tid=0 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 client.25417 shutdown 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 msgr2=0x55e76e2e7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f31b8000f80 tx=0x7f31b800ef30 comp rx=0 tx=0).stop 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 shutdown_connections 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e76e357f10 0x55e76e356c20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.303+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e76e2dc0b0 0x55e76e2e7370 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f319f400640 0 client.25417 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e76e2d6f40 0x55e76e2e4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 1 --2- 172.21.15.22:0/3240513062 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e76e2d6550 0x55e76e2e3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 >> 172.21.15.22:0/3240513062 conn(0x55e76e2d1ac0 msgr2=0x55e76e2d55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 shutdown_connections 2024-03-24T06:39:48.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 1 -- 172.21.15.22:0/3240513062 wait complete. 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.304+0000 7f31c778c740 20 client.25417 trim_cache size 0 max 16384 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 3, 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24544, 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 136.90670534, 2024-03-24T06:39:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 195.424801164 2024-03-24T06:39:48.310 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:48.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 msgr2=0x7f31a40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 0x7f31a40a4600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f31b002c250 tx=0x7f31b002c7f0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 msgr2=0x7f31c0199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f31ac00d950 tx=0x7f31ac00de10 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 shutdown_connections 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f31a40a2140 0x7f31a40a4600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31c0105500 0x7f31c01a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31c0104b40 0x7f31c0199f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 --2- 172.21.15.22:0/3715847138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31c0103950 0x7f31c01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 >> 172.21.15.22:0/3715847138 conn(0x7f31c00feec0 msgr2=0x7f31c0107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 shutdown_connections 2024-03-24T06:39:48.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.305+0000 7f31c4a00640 1 -- 172.21.15.22:0/3715847138 wait complete. 2024-03-24T06:39:48.323 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T06:39:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1752457313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:47 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2922549006' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:39:48.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 -- 172.21.15.22:0/3839046506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a8104a00 msgr2=0x7f72a8104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 --2- 172.21.15.22:0/3839046506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a8104a00 0x7f72a8104e80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7294009900 tx=0x7f729402bc40 comp rx=0 tx=0).stop 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 -- 172.21.15.22:0/3839046506 shutdown_connections 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 --2- 172.21.15.22:0/3839046506 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a81053c0 0x7f72a810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 --2- 172.21.15.22:0/3839046506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a8104a00 0x7f72a8104e80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 --2- 172.21.15.22:0/3839046506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8103810 0x7f72a8103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.516+0000 7f72ae800640 1 -- 172.21.15.22:0/3839046506 >> 172.21.15.22:0/3839046506 conn(0x7f72a80fedc0 msgr2=0x7f72a81011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.517+0000 7f72ae800640 1 -- 172.21.15.22:0/3839046506 shutdown_connections 2024-03-24T06:39:48.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.517+0000 7f72ae800640 1 -- 172.21.15.22:0/3839046506 wait complete. 2024-03-24T06:39:48.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 Processor -- start 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 -- start start 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 0x7f72a8199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 0x7f72a8199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a81053c0 0x7f72a81a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81a1380 con 0x7f72a8103810 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81a14f0 con 0x7f72a8104a00 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ae800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81a16b0 con 0x7f72a81053c0 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 0x7f72a8199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 0x7f72a8199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59586/0 (socket says 172.21.15.22:59586) 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72ad400640 1 -- 172.21.15.22:0/3289501158 learned_addr learned my addr 172.21.15.22:0/3289501158 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.518+0000 7f72aca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 0x7f72a8199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72ade00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a81053c0 0x7f72a81a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 -- 172.21.15.22:0/3289501158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a81053c0 msgr2=0x7f72a81a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a81053c0 0x7f72a81a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 -- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 msgr2=0x7f72a8199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 0x7f72a8199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 -- 172.21.15.22:0/3289501158 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c009490 con 0x7f72a8104a00 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.519+0000 7f72aca00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 0x7f72a8199dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f729402c180 tx=0x7f7294005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.520+0000 7f72a6000640 1 -- 172.21.15.22:0/3289501158 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72940043d0 con 0x7f72a8104a00 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.520+0000 7f72a6000640 1 -- 172.21.15.22:0/3289501158 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72940052d0 con 0x7f72a8104a00 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.520+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7294009560 con 0x7f72a8104a00 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.520+0000 7f72a6000640 1 -- 172.21.15.22:0/3289501158 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294003d40 con 0x7f72a8104a00 2024-03-24T06:39:48.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.520+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81a1cd0 con 0x7f72a8104a00 2024-03-24T06:39:48.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.523+0000 7f72a6000640 1 -- 172.21.15.22:0/3289501158 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f7294005440 con 0x7f72a8104a00 2024-03-24T06:39:48.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.523+0000 7f72b2bd6740 1 Processor -- start 2024-03-24T06:39:48.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.523+0000 7f72b2bd6740 1 -- start start 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.523+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842b90b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.523+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b98710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564842aff050 con 0x564842b96280 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564842afc580 con 0x564842b90720 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564842af7d60 con 0x564842b91110 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33084/0 (socket says 172.21.15.22:33084) 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 -- 172.21.15.22:0/1233070609 learned_addr learned my addr 172.21.15.22:0/1233070609 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ade00640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842b90b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72aca00640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b98710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 -- 172.21.15.22:0/1233070609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 msgr2=0x564842b90b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842b90b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 -- 172.21.15.22:0/1233070609 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 msgr2=0x564842b98710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b98710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ad400640 1 -- 172.21.15.22:0/1233070609 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564842ab0620 con 0x564842b91110 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.524+0000 7f72ade00640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842b90b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f72ad400640 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f729c009f10 tx=0x7f729c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644c99a930c794fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f729a600640 1 -- 172.21.15.22:0/1233070609 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c005220 con 0x564842b91110 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f729a600640 1 -- 172.21.15.22:0/1233070609 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004c80 con 0x564842b91110 2024-03-24T06:39:48.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f729a600640 1 -- 172.21.15.22:0/1233070609 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c00f9a0 con 0x564842b91110 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f72a6000640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 0x7f72900a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f72a6000640 1 -- 172.21.15.22:0/3289501158 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f72940f0730 con 0x7f72a8104a00 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.525+0000 7f72ade00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 0x7f72900a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1233070609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 msgr2=0x564842b95c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f729c009f10 tx=0x7f729c011810 comp rx=0 tx=0).stop 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1233070609 shutdown_connections 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b98710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b95c10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1233070609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842b90b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1233070609 >> 172.21.15.22:0/1233070609 conn(0x564842b8bc90 msgr2=0x564842b8c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1233070609 shutdown_connections 2024-03-24T06:39:48.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1233070609 wait complete. 2024-03-24T06:39:48.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.526+0000 7f72ade00640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 0x7f72900a4600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x564842b91970 tx=0x7f72a0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.527+0000 7f72b2bd6740 1 -- start start 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.527+0000 7f72b2bd6740 1 Processor -- start 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564842a9cfb0 con 0x564842b90720 2024-03-24T06:39:48.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564842a9cc50 con 0x564842b96280 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56484283ff20 con 0x564842b91110 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72b2bd6740 10 client.0 _ll_register_callbacks cb 0x564842808ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ade00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59600/0 (socket says 172.21.15.22:59600) 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33090/0 (socket says 172.21.15.22:33090) 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ade00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35564/0 (socket says 172.21.15.22:35564) 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ad400640 1 -- 172.21.15.22:0/1150739583 learned_addr learned my addr 172.21.15.22:0/1150739583 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 -- 172.21.15.22:0/1150739583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 msgr2=0x564842ba1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 -- 172.21.15.22:0/1150739583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 msgr2=0x564842b9a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72aca00640 1 -- 172.21.15.22:0/1150739583 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c009b40 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.528+0000 7f72ade00640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f72aca00640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f72940bcd80 tx=0x7f72940b93e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e948f573c4b4cf50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72940bd780 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f72940bd080 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648427f76c0 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72940c11d0 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f72b2bd6740 10 client.25429 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564842af63e0 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5648427f76c0 con 0x564842b91110 2024-03-24T06:39:48.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.529+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f729403c070 con 0x564842b91110 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 10 client.25429 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 10 client.25429 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 20 client.25429 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842ba7230 0x564842ba86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 4 client.25429 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564842848270 con 0x564842ba7230 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72ad400640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842ba7230 0x564842ba86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f72ad400640 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842ba7230 0x564842ba86a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88af399bad239132 server_cookie=c4680ad883a34609 in_seq=0 out_seq=0 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.530+0000 7f728ea00640 10 client.25429 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f728ea00640 1 -- 172.21.15.22:0/1150739583 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564842848270 con 0x564842ba7230 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f728ea00640 10 client.25429 handle_command_reply: tid=0 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 client.25429 shutdown 2024-03-24T06:39:48.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 msgr2=0x564842b9aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f72940bcd80 tx=0x7f72940b93e0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 shutdown_connections 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842ba7230 0x564842ba86a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9aee0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1150739583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f728ea00640 0 client.25429 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 >> 172.21.15.22:0/1150739583 conn(0x564842b8bc90 msgr2=0x564842b8add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 shutdown_connections 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1150739583 wait complete. 2024-03-24T06:39:48.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.532+0000 7f72b2bd6740 20 client.25429 trim_cache size 0 max 16384 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.552+0000 7f72b2bd6740 1 Processor -- start 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 -- start start 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842ba2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b91b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564842ba7610 con 0x564842b96280 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564842848270 con 0x564842b90720 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564842ba9150 con 0x564842b91110 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72ad400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842ba2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33102/0 (socket says 172.21.15.22:33102) 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72aca00640 1 -- 172.21.15.22:0/3381954495 learned_addr learned my addr 172.21.15.22:0/3381954495 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.553+0000 7f72ade00640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b91b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 -- 172.21.15.22:0/3381954495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 msgr2=0x564842ba2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842ba2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 -- 172.21.15.22:0/3381954495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 msgr2=0x564842b91b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b91b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 -- 172.21.15.22:0/3381954495 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564842a92900 con 0x564842b91110 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72ad400640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842ba2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72ade00640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b91b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f72aca00640 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72940b9a00 tx=0x7f7294005790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e089e85c79c7f70c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f7299c00640 1 -- 172.21.15.22:0/3381954495 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72940d3070 con 0x564842b91110 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f7299c00640 1 -- 172.21.15.22:0/3381954495 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72940c5600 con 0x564842b91110 2024-03-24T06:39:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.554+0000 7f7299c00640 1 -- 172.21.15.22:0/3381954495 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72940eded0 con 0x564842b91110 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- 172.21.15.22:0/3381954495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 msgr2=0x564842ba1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72940b9a00 tx=0x7f7294005790 comp rx=0 tx=0).stop 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- 172.21.15.22:0/3381954495 shutdown_connections 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b96280 0x564842b91b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842ba1ee0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/3381954495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b90720 0x564842ba2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- 172.21.15.22:0/3381954495 >> 172.21.15.22:0/3381954495 conn(0x564842b8bc90 msgr2=0x564842b8add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- 172.21.15.22:0/3381954495 shutdown_connections 2024-03-24T06:39:48.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- 172.21.15.22:0/3381954495 wait complete. 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.555+0000 7f72b2bd6740 1 -- start start 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 Processor -- start 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564842ba0f40 con 0x564842b90720 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564842ba10b0 con 0x564842b96280 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564842ba1270 con 0x564842b91110 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72b2bd6740 10 client.0 _ll_register_callbacks cb 0x7f7294009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72ad400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72aca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33112/0 (socket says 172.21.15.22:33112) 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72aca00640 1 -- 172.21.15.22:0/1550705542 learned_addr learned my addr 172.21.15.22:0/1550705542 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.556+0000 7f72ade00640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 -- 172.21.15.22:0/1550705542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 msgr2=0x564842ba0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 -- 172.21.15.22:0/1550705542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 msgr2=0x564842b9d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 -- 172.21.15.22:0/1550705542 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564842ab0620 con 0x564842b91110 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72aca00640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f72940045d0 tx=0x7f7294003ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa5ee9d9d133741 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f728ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72940d3020 con 0x564842b91110 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f72940bd080 con 0x564842b91110 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.557+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f729c004c80 con 0x564842b91110 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72940c5890 con 0x564842b91110 2024-03-24T06:39:48.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 10 client.25435 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564842af63e0 con 0x564842b91110 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004c80 con 0x564842b91110 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f729403c070 con 0x564842b91110 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 10 client.25435 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 10 client.25435 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 20 client.25435 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842c10e40 0x564842c0fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 4 client.25435 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "status"}) -- 0x564842834610 con 0x564842c10e40 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.558+0000 7f72ad400640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842c10e40 0x564842c0fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72ad400640 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842c10e40 0x564842c0fb50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c955d3811e0d854c server_cookie=6fa2ea26a2eb1d62 in_seq=0 out_seq=0 2024-03-24T06:39:48.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f728ea00640 10 client.25435 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f728ea00640 1 -- 172.21.15.22:0/1550705542 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x564842834610 con 0x564842c10e40 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f728ea00640 10 client.25435 handle_command_reply: tid=0 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 client.25435 shutdown 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 msgr2=0x564842b9d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f72940045d0 tx=0x7f7294003ef0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 shutdown_connections 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564842c10e40 0x564842c0fb50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564842b96280 0x564842ba0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564842b91110 0x564842b9d9e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f728ea00640 0 client.25435 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 --2- 172.21.15.22:0/1550705542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564842b90720 0x564842b9d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 >> 172.21.15.22:0/1550705542 conn(0x564842b8bc90 msgr2=0x564842b8add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 shutdown_connections 2024-03-24T06:39:48.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.559+0000 7f72b2bd6740 1 -- 172.21.15.22:0/1550705542 wait complete. 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72b2bd6740 20 client.25435 trim_cache size 0 max 16384 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 2, 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24565, 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 138.17599722, 2024-03-24T06:39:48.565 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:48.566 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:48.566 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:48.566 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 189.64226376100001 2024-03-24T06:39:48.566 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 msgr2=0x7f72900a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 0x7f72900a4600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x564842b91970 tx=0x7f72a0009500 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 msgr2=0x7f72a8199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 0x7f72a8199dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f729402c180 tx=0x7f7294005e80 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 shutdown_connections 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f72900a2140 0x7f72900a4600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72a81053c0 0x7f72a81a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72a8104a00 0x7f72a8199dd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 --2- 172.21.15.22:0/3289501158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72a8103810 0x7f72a8199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.560+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 >> 172.21.15.22:0/3289501158 conn(0x7f72a80fedc0 msgr2=0x7f72a8107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.561+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 shutdown_connections 2024-03-24T06:39:48.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.561+0000 7f72ae800640 1 -- 172.21.15.22:0/3289501158 wait complete. 2024-03-24T06:39:48.579 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T06:39:48.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 -- 172.21.15.22:0/1704533739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8102800 msgr2=0x7f94e8102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 --2- 172.21.15.22:0/1704533739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8102800 0x7f94e8102c80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009900 tx=0x7f94d002bc30 comp rx=0 tx=0).stop 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 -- 172.21.15.22:0/1704533739 shutdown_connections 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 --2- 172.21.15.22:0/1704533739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e8109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 --2- 172.21.15.22:0/1704533739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8102800 0x7f94e8102c80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 --2- 172.21.15.22:0/1704533739 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8101610 0x7f94e8101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 -- 172.21.15.22:0/1704533739 >> 172.21.15.22:0/1704533739 conn(0x7f94e80fcb80 msgr2=0x7f94e80fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 -- 172.21.15.22:0/1704533739 shutdown_connections 2024-03-24T06:39:48.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.797+0000 7f94e7e00640 1 -- 172.21.15.22:0/1704533739 wait complete. 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.799+0000 7f94e7e00640 1 Processor -- start 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.799+0000 7f94e7e00640 1 -- start start 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.799+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.799+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 0x7f94e8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.799+0000 7f94e7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e7e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f94e819f160 con 0x7f94e8102800 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e7e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f94e819f2d0 con 0x7f94e81031c0 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e7e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f94e819f490 con 0x7f94e8101610 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33118/0 (socket says 172.21.15.22:33118) 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 -- 172.21.15.22:0/3745478131 learned_addr learned my addr 172.21.15.22:0/3745478131 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94de000640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 0x7f94e8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e7400640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 -- 172.21.15.22:0/3745478131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 msgr2=0x7f94e819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 -- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 msgr2=0x7f94e8197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 0x7f94e8197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 -- 172.21.15.22:0/3745478131 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e0009490 con 0x7f94e8101610 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94de000640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 0x7f94e8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e7400640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.800+0000 7f94e6a00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f94d800ebc0 tx=0x7f94d800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.801+0000 7f94dfe00640 1 -- 172.21.15.22:0/3745478131 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8003b70 con 0x7f94e8101610 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.801+0000 7f94dfe00640 1 -- 172.21.15.22:0/3745478131 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d800eec0 con 0x7f94e8101610 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.801+0000 7f94dfe00640 1 -- 172.21.15.22:0/3745478131 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8019800 con 0x7f94e8101610 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.801+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d0009560 con 0x7f94e8101610 2024-03-24T06:39:48.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.801+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e819fb40 con 0x7f94e8101610 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.805+0000 7f94f03b7740 1 Processor -- start 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.805+0000 7f94f03b7740 1 -- start start 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94dfe00640 1 -- 172.21.15.22:0/3745478131 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f94d8019960 con 0x7f94e8101610 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbee020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbf0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ccb53050 con 0x5605ccbee690 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ccb50580 con 0x5605ccbe8b30 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94f03b7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605ccb4bd60 con 0x5605ccbe9520 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35590/0 (socket says 172.21.15.22:35590) 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94e7400640 1 -- 172.21.15.22:0/1661692532 learned_addr learned my addr 172.21.15.22:0/1661692532 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.806+0000 7f94de000640 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbf0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 -- 172.21.15.22:0/1661692532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 msgr2=0x5605ccbee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 -- 172.21.15.22:0/1661692532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 msgr2=0x5605ccbf0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbf0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 -- 172.21.15.22:0/1661692532 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ccb04620 con 0x5605ccbe8b30 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94de000640 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbf0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94e7400640 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009d10 tx=0x7f94e0010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5760d785d0755a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94d7400640 1 -- 172.21.15.22:0/1661692532 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0005020 con 0x5605ccbe8b30 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94d7400640 1 -- 172.21.15.22:0/1661692532 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e0005660 con 0x5605ccbe8b30 2024-03-24T06:39:48.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.807+0000 7f94d7400640 1 -- 172.21.15.22:0/1661692532 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0012740 con 0x5605ccbe8b30 2024-03-24T06:39:48.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 -- 172.21.15.22:0/1661692532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 msgr2=0x5605ccbe8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009d10 tx=0x7f94e0010740 comp rx=0 tx=0).stop 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 -- 172.21.15.22:0/1661692532 shutdown_connections 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbf0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1661692532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbe8f50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 -- 172.21.15.22:0/1661692532 >> 172.21.15.22:0/1661692532 conn(0x5605ccbdfc90 msgr2=0x5605ccbe0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 -- 172.21.15.22:0/1661692532 shutdown_connections 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94f03b7740 1 -- 172.21.15.22:0/1661692532 wait complete. 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.808+0000 7f94dfe00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 0x7f94cc0a4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.809+0000 7f94dfe00640 1 -- 172.21.15.22:0/3745478131 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f94d8014070 con 0x7f94e8101610 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.809+0000 7f94de000640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 0x7f94cc0a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.809+0000 7f94f03b7740 1 -- start start 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.809+0000 7f94f03b7740 1 Processor -- start 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94de000640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 0x7f94cc0a4510 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f94d0000c00 tx=0x7f94d002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbfa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ccaf0fb0 con 0x5605ccbee690 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ccaf0c50 con 0x5605ccbe8b30 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605cc893f20 con 0x5605ccbe9520 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94f03b7740 10 client.0 _ll_register_callbacks cb 0x5605cc85cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94de000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94de000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33136/0 (socket says 172.21.15.22:33136) 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35600/0 (socket says 172.21.15.22:35600) 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94de000640 1 -- 172.21.15.22:0/1116111503 learned_addr learned my addr 172.21.15.22:0/1116111503 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e7400640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbfa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 -- 172.21.15.22:0/1116111503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 msgr2=0x5605ccbf3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 -- 172.21.15.22:0/1116111503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 msgr2=0x5605ccbfa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbfa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94e6a00640 1 -- 172.21.15.22:0/1116111503 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e00099c0 con 0x5605ccbe8b30 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.810+0000 7f94de000640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94e6a00640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f94d80cc0e0 tx=0x7f94d809b200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43a533e61750fe78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d809b370 con 0x5605ccbe8b30 2024-03-24T06:39:48.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f94d809f410 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5605cc84b6c0 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f94d80a3a60 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94f03b7740 10 client.25406 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605ccb4a3e0 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.811+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5605cc84b6c0 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f94d809b5d0 con 0x5605ccbe8b30 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 10 client.25406 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 10 client.25406 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 20 client.25406 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccbff8e0 0x5605ccc00d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 4 client.25406 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605cc89c270 con 0x5605ccbff8e0 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.812+0000 7f94de000640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccbff8e0 0x5605ccc00d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.813+0000 7f94de000640 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccbff8e0 0x5605ccc00d50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b075fb4fe1b75752 server_cookie=d9b2a6dcd90ac2ad in_seq=0 out_seq=0 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.813+0000 7f94cbe00640 10 client.25406 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94cbe00640 1 -- 172.21.15.22:0/1116111503 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605cc89c270 con 0x5605ccbff8e0 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94cbe00640 10 client.25406 handle_command_reply: tid=0 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 client.25406 shutdown 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 msgr2=0x5605ccbf2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f94d80cc0e0 tx=0x7f94d809b200 comp rx=0 tx=0).stop 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 shutdown_connections 2024-03-24T06:39:48.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbee690 0x5605ccbfa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbf3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccbff8e0 0x5605ccc00d50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 --2- 172.21.15.22:0/1116111503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe8b30 0x5605ccbf2d20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94cbe00640 0 client.25406 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 >> 172.21.15.22:0/1116111503 conn(0x5605ccbdfc90 msgr2=0x5605ccbdedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 shutdown_connections 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 1 -- 172.21.15.22:0/1116111503 wait complete. 2024-03-24T06:39:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.814+0000 7f94f03b7740 20 client.25406 trim_cache size 0 max 16384 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.834+0000 7f94f03b7740 1 Processor -- start 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.834+0000 7f94f03b7740 1 -- start start 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.834+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbe0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.834+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbfa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94f03b7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbff440 con 0x5605ccbe8b30 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94f03b7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbdad10 con 0x5605ccbee690 2024-03-24T06:39:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94f03b7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbfaf20 con 0x5605ccbe9520 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94de000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbfa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbe0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35604/0 (socket says 172.21.15.22:35604) 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 -- 172.21.15.22:0/636226094 learned_addr learned my addr 172.21.15.22:0/636226094 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 -- 172.21.15.22:0/636226094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 msgr2=0x5605ccbfa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbfa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 -- 172.21.15.22:0/636226094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 msgr2=0x5605ccbe0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbe0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.835+0000 7f94e7400640 1 -- 172.21.15.22:0/636226094 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ccae6900 con 0x5605ccbee690 2024-03-24T06:39:48.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94e7400640 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x5605ccbe9d80 tx=0x7f94e00098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58e427b17244af96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94d6a00640 1 -- 172.21.15.22:0/636226094 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0009ec0 con 0x5605ccbee690 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94d6a00640 1 -- 172.21.15.22:0/636226094 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e001f040 con 0x5605ccbee690 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94d6a00640 1 -- 172.21.15.22:0/636226094 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e00128b0 con 0x5605ccbee690 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94f03b7740 1 -- 172.21.15.22:0/636226094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 msgr2=0x5605ccbf2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.836+0000 7f94f03b7740 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x5605ccbe9d80 tx=0x7f94e00098c0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 -- 172.21.15.22:0/636226094 shutdown_connections 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbee690 0x5605ccbf2930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbe9520 0x5605ccbfa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 --2- 172.21.15.22:0/636226094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbe0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 -- 172.21.15.22:0/636226094 >> 172.21.15.22:0/636226094 conn(0x5605ccbdfc90 msgr2=0x5605ccbe9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 -- 172.21.15.22:0/636226094 shutdown_connections 2024-03-24T06:39:48.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 -- 172.21.15.22:0/636226094 wait complete. 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 -- start start 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.837+0000 7f94f03b7740 1 Processor -- start 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 0x5605ccbf69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 0x5605ccbf9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbf9b50 con 0x5605ccbe8b30 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbf9cc0 con 0x5605ccbe9520 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605ccbf9e80 con 0x5605ccbee690 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94f03b7740 10 client.0 _ll_register_callbacks cb 0x7f94d8097610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94e7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 0x5605ccbf9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94de000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 0x5605ccbf69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94e6a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59676/0 (socket says 172.21.15.22:59676) 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.838+0000 7f94e6a00640 1 -- 172.21.15.22:0/503685543 learned_addr learned my addr 172.21.15.22:0/503685543 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 -- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 msgr2=0x5605ccbf69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 0x5605ccbf69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 -- 172.21.15.22:0/503685543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 msgr2=0x5605ccbf6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 -- 172.21.15.22:0/503685543 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ccb04620 con 0x5605ccbee690 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e6a00640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94de000640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 0x5605ccbf69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94e7400640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 0x5605ccbf9610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f94e0026fd0 tx=0x7f94e0012c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb98a859d8812b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0035070 con 0x5605ccbee690 2024-03-24T06:39:48.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94e0005660 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.839+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f94e0016070 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f94e0012e40 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 10 client.25453 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5605ccb0c560 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94e0005660 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f94e0038030 con 0x5605ccbee690 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 10 client.25453 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 10 client.25453 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 20 client.25453 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccc69f30 0x5605ccc68c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 4 client.25453 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.840+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "status"}) -- 0x5605cc888610 con 0x5605ccc69f30 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94e6a00640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccc69f30 0x5605ccc68c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94e6a00640 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccc69f30 0x5605ccc68c40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=38c9c0a6a7e7e2d3 server_cookie=c09bae1a571af792 in_seq=0 out_seq=0 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94cbe00640 10 client.25453 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94cbe00640 1 -- 172.21.15.22:0/503685543 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5605cc888610 con 0x5605ccc69f30 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94cbe00640 10 client.25453 handle_command_reply: tid=0 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 client.25453 shutdown 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 msgr2=0x5605ccbf9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 0x5605ccbf9610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f94e0026fd0 tx=0x7f94e0012c50 comp rx=0 tx=0).stop 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 shutdown_connections 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605ccbee690 0x5605ccbf9610 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.841+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5605ccc69f30 0x5605ccc68c40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ccbe9520 0x5605ccbf69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 1 --2- 172.21.15.22:0/503685543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605ccbe8b30 0x5605ccbf6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94cbe00640 0 client.25453 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 >> 172.21.15.22:0/503685543 conn(0x5605ccbdfc90 msgr2=0x5605ccbdedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 shutdown_connections 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 1 -- 172.21.15.22:0/503685543 wait complete. 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.842+0000 7f94f03b7740 20 client.25453 trim_cache size 0 max 16384 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 1, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24584, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 139.47071157900001, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 178.68084390499999 2024-03-24T06:39:48.848 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 msgr2=0x7f94cc0a4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 0x7f94cc0a4510 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f94d0000c00 tx=0x7f94d002e040 comp rx=0 tx=0).stop 2024-03-24T06:39:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 msgr2=0x7f94e8197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f94d800ebc0 tx=0x7f94d800c6a0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 shutdown_connections 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f94cc0a2050 0x7f94cc0a4510 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94e81031c0 0x7f94e819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e8102800 0x7f94e8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 --2- 172.21.15.22:0/3745478131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94e8101610 0x7f94e8197670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 >> 172.21.15.22:0/3745478131 conn(0x7f94e80fcb80 msgr2=0x7f94e8105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 shutdown_connections 2024-03-24T06:39:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:48.843+0000 7f94e7e00640 1 -- 172.21.15.22:0/3745478131 wait complete. 2024-03-24T06:39:48.861 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 -- 172.21.15.22:0/3033514056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741020a0 msgr2=0x7f2574108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 --2- 172.21.15.22:0/3033514056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741020a0 0x7f2574108930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f256c009ca0 tx=0x7f256c02c0f0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 -- 172.21.15.22:0/3033514056 shutdown_connections 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 --2- 172.21.15.22:0/3033514056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741020a0 0x7f2574108930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 --2- 172.21.15.22:0/3033514056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741016e0 0x7f2574101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 --2- 172.21.15.22:0/3033514056 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 -- 172.21.15.22:0/3033514056 >> 172.21.15.22:0/3033514056 conn(0x7f25740fba80 msgr2=0x7f25740fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 -- 172.21.15.22:0/3033514056 shutdown_connections 2024-03-24T06:39:49.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.056+0000 7f2579c00640 1 -- 172.21.15.22:0/3033514056 wait complete. 2024-03-24T06:39:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.057+0000 7f2579c00640 1 Processor -- start 2024-03-24T06:39:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.057+0000 7f2579c00640 1 -- start start 2024-03-24T06:39:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741020a0 0x7f257419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f257419f100 con 0x7f25741004f0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f257419f270 con 0x7f25741020a0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f257419f430 con 0x7f25741016e0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33180/0 (socket says 172.21.15.22:33180) 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59694/0 (socket says 172.21.15.22:59694) 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 -- 172.21.15.22:0/969975302 learned_addr learned my addr 172.21.15.22:0/969975302 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2579200640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741020a0 0x7f257419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 -- 172.21.15.22:0/969975302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741020a0 msgr2=0x7f257419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741020a0 0x7f257419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 -- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 msgr2=0x7f2574197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.058+0000 7f2573400640 1 -- 172.21.15.22:0/969975302 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c009900 con 0x7f25741016e0 2024-03-24T06:39:49.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2573400640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9f0 tx=0x7f256000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2570c00640 1 -- 172.21.15.22:0/969975302 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560009720 con 0x7f25741016e0 2024-03-24T06:39:49.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2570c00640 1 -- 172.21.15.22:0/969975302 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256000cd80 con 0x7f25741016e0 2024-03-24T06:39:49.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2570c00640 1 -- 172.21.15.22:0/969975302 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560010690 con 0x7f25741016e0 2024-03-24T06:39:49.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257419f710 con 0x7f25741016e0 2024-03-24T06:39:49.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.059+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257419fc20 con 0x7f25741016e0 2024-03-24T06:39:49.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.062+0000 7f257df7d740 1 Processor -- start 2024-03-24T06:39:49.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 -- start start 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e0d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e0fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2570c00640 1 -- 172.21.15.22:0/969975302 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f25600046c0 con 0x7f25741016e0 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564338d72050 con 0x564338e0d690 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564338d6f580 con 0x564338e07b30 2024-03-24T06:39:49.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f257df7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564338d6ad60 con 0x564338e08520 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2579200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2579200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35636/0 (socket says 172.21.15.22:35636) 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2573e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e0d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2579200640 1 -- 172.21.15.22:0/3623923393 learned_addr learned my addr 172.21.15.22:0/3623923393 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.063+0000 7f2573400640 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e0fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 -- 172.21.15.22:0/3623923393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 msgr2=0x564338e0d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e0d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 -- 172.21.15.22:0/3623923393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 msgr2=0x564338e0fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e0fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 -- 172.21.15.22:0/3623923393 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564338d23620 con 0x564338e07b30 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2579200640 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f256c02c700 tx=0x7f256c0093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffbc27aab4b3d0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2569600640 1 -- 172.21.15.22:0/3623923393 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c0323c0 con 0x564338e07b30 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2569600640 1 -- 172.21.15.22:0/3623923393 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256c0327b0 con 0x564338e07b30 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.064+0000 7f2569600640 1 -- 172.21.15.22:0/3623923393 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c003c60 con 0x564338e07b30 2024-03-24T06:39:49.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f2570c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 0x7f25580a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f2570c00640 1 -- 172.21.15.22:0/969975302 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f2560014070 con 0x7f25741016e0 2024-03-24T06:39:49.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 -- 172.21.15.22:0/3623923393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 msgr2=0x564338e07f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f256c02c700 tx=0x7f256c0093f0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f2573e00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 0x7f25580a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 -- 172.21.15.22:0/3623923393 shutdown_connections 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e0fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e0d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 --2- 172.21.15.22:0/3623923393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e07f50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 -- 172.21.15.22:0/3623923393 >> 172.21.15.22:0/3623923393 conn(0x564338dfec90 msgr2=0x564338dff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.065+0000 7f257df7d740 1 -- 172.21.15.22:0/3623923393 shutdown_connections 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.066+0000 7f257df7d740 1 -- 172.21.15.22:0/3623923393 wait complete. 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.066+0000 7f257df7d740 1 -- start start 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.066+0000 7f2573e00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 0x7f25580a4600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x564338dffc00 tx=0x7f2564005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.066+0000 7f257df7d740 1 Processor -- start 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 0x564338e11cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e19270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564338d0ffb0 con 0x564338e0d690 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564338d0fc50 con 0x564338e08520 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564338ab2f20 con 0x564338e07b30 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f257df7d740 10 client.0 _ll_register_callbacks cb 0x564338a7bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 0x564338e11cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35646/0 (socket says 172.21.15.22:35646) 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 -- 172.21.15.22:0/3005206904 learned_addr learned my addr 172.21.15.22:0/3005206904 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2579200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e19270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 -- 172.21.15.22:0/3005206904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 msgr2=0x564338e11cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 0x564338e11cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 -- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 msgr2=0x564338e19270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e19270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573400640 1 -- 172.21.15.22:0/3005206904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2574109230 con 0x564338e08520 2024-03-24T06:39:49.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.067+0000 7f2573e00640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 0x564338e11cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f2573400640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2560097130 tx=0x7f256009ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3246d9b1c9e1ba53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f255e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600a31e0 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f256009acd0 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564338a6a6c0 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25600ce170 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f257df7d740 10 client.25424 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564338d693e0 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.068+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564338a6a6c0 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25600aed20 con 0x564338e08520 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 10 client.25424 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 10 client.25424 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 20 client.25424 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e1e8d0 0x564338e1fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 4 client.25424 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:39:49.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564338abb270 con 0x564338e1e8d0 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.069+0000 7f2573e00640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e1e8d0 0x564338e1fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.070+0000 7f2573e00640 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e1e8d0 0x564338e1fd40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90ed6dbde3cc5320 server_cookie=22e5c372a96ebdb1 in_seq=0 out_seq=0 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.070+0000 7f255e000640 10 client.25424 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.070+0000 7f255e000640 1 -- 172.21.15.22:0/3005206904 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564338abb270 con 0x564338e1e8d0 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.070+0000 7f255e000640 10 client.25424 handle_command_reply: tid=0 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 client.25424 shutdown 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 msgr2=0x564338e12230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2560097130 tx=0x7f256009ab60 comp rx=0 tx=0).stop 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 shutdown_connections 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e19270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e1e8d0 0x564338e1fd40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e08520 0x564338e12230 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 --2- 172.21.15.22:0/3005206904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e07b30 0x564338e11cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f255e000640 0 client.25424 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:49.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 >> 172.21.15.22:0/3005206904 conn(0x564338dfec90 msgr2=0x564338dfddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 shutdown_connections 2024-03-24T06:39:49.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 1 -- 172.21.15.22:0/3005206904 wait complete. 2024-03-24T06:39:49.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.071+0000 7f257df7d740 20 client.25424 trim_cache size 0 max 16384 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 Processor -- start 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 -- start start 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 0x564338e10800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e0d690 0x564338e08fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564338e1f8a0 con 0x564338e07b30 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564338df9d10 con 0x564338e0d690 2024-03-24T06:39:49.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f257df7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564338e19ef0 con 0x564338e08520 2024-03-24T06:39:49.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f2573e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 0x564338e10800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33218/0 (socket says 172.21.15.22:33218) 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.108+0000 7f2573400640 1 -- 172.21.15.22:0/818585944 learned_addr learned my addr 172.21.15.22:0/818585944 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2579200640 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e0d690 0x564338e08fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 -- 172.21.15.22:0/818585944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e0d690 msgr2=0x564338e08fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e0d690 0x564338e08fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 -- 172.21.15.22:0/818585944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 msgr2=0x564338e10800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 0x564338e10800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 -- 172.21.15.22:0/818585944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564338d05900 con 0x564338e08520 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573e00640 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 0x564338e10800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2573400640 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2560097650 tx=0x7f256009a8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e133db85a2567e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2568c00640 1 -- 172.21.15.22:0/818585944 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600ce170 con 0x564338e08520 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2568c00640 1 -- 172.21.15.22:0/818585944 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256009a630 con 0x564338e08520 2024-03-24T06:39:49.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.109+0000 7f2568c00640 1 -- 172.21.15.22:0/818585944 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600d00f0 con 0x564338e08520 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 -- 172.21.15.22:0/818585944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 msgr2=0x564338e19230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2560097650 tx=0x7f256009a8d0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 -- 172.21.15.22:0/818585944 shutdown_connections 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e0d690 0x564338e08fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e19230 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 --2- 172.21.15.22:0/818585944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e07b30 0x564338e10800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 -- 172.21.15.22:0/818585944 >> 172.21.15.22:0/818585944 conn(0x564338dfec90 msgr2=0x564338dfddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 -- 172.21.15.22:0/818585944 shutdown_connections 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.110+0000 7f257df7d740 1 -- 172.21.15.22:0/818585944 wait complete. 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 -- start start 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 Processor -- start 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e17220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e82b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564338e830b0 con 0x564338e0d690 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564338e831f0 con 0x564338e07b30 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564338e833b0 con 0x564338e08520 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.111+0000 7f257df7d740 10 client.0 _ll_register_callbacks cb 0x7f2560004a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e17220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33234/0 (socket says 172.21.15.22:33234) 2024-03-24T06:39:49.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2579200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e82b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 -- 172.21.15.22:0/1096443603 learned_addr learned my addr 172.21.15.22:0/1096443603 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 -- 172.21.15.22:0/1096443603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 msgr2=0x564338e17220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e17220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 -- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 msgr2=0x564338e82b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e82b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 -- 172.21.15.22:0/1096443603 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2564009d60 con 0x564338e08520 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573e00640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e17220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f2573400640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9c0 tx=0x7f256009ab60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640949f62b89f1f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.112+0000 7f255e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600ce170 con 0x564338e08520 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f256009acd0 con 0x564338e08520 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256c0327b0 con 0x564338e08520 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25600a31e0 con 0x564338e08520 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 10 client.25471 fetch_fsmap learned FSMap version 28 2024-03-24T06:39:49.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564338d693e0 con 0x564338e08520 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256c0327b0 con 0x564338e08520 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25600aed20 con 0x564338e08520 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 10 client.25471 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 10 client.25471 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 20 client.25471 populate_metadata read hostname 'smithi022' 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e885a0 0x564338e872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 4 client.25471 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.113+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "status"}) -- 0x564338aa7610 con 0x564338e885a0 2024-03-24T06:39:49.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.114+0000 7f2573e00640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e885a0 0x564338e872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.114+0000 7f2573e00640 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e885a0 0x564338e872b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75d98879e1b9a265 server_cookie=16032d1afd003309 in_seq=0 out_seq=0 2024-03-24T06:39:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.114+0000 7f255e000640 10 client.25471 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f255e000640 1 -- 172.21.15.22:0/1096443603 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564338aa7610 con 0x564338e885a0 2024-03-24T06:39:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f255e000640 10 client.25471 handle_command_reply: tid=0 2024-03-24T06:39:49.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 client.25471 shutdown 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 msgr2=0x564338e80620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9c0 tx=0x7f256009ab60 comp rx=0 tx=0).stop 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 shutdown_connections 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564338e0d690 0x564338e82b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564338e08520 0x564338e80620 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x564338e885a0 0x564338e872b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 --2- 172.21.15.22:0/1096443603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564338e07b30 0x564338e17220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 >> 172.21.15.22:0/1096443603 conn(0x564338dfec90 msgr2=0x564338dfddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f255e000640 0 client.25471 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 shutdown_connections 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 1 -- 172.21.15.22:0/1096443603 wait complete. 2024-03-24T06:39:49.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.115+0000 7f257df7d740 20 client.25471 trim_cache size 0 max 16384 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "7bca48fe-e9a7-11ee-95cd-87774f69a715", 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 0, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 24601, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 171.95107798399999, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 28, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 93, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 181.85156839499999 2024-03-24T06:39:49.119 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 msgr2=0x7f25580a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 0x7f25580a4600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x564338dffc00 tx=0x7f2564005f90 comp rx=0 tx=0).stop 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 msgr2=0x7f2574197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9f0 tx=0x7f256000edf0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 shutdown_connections 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f25580a2140 0x7f25580a4600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25741020a0 0x7f257419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25741016e0 0x7f2574197b50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 --2- 172.21.15.22:0/969975302 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25741004f0 0x7f2574197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 >> 172.21.15.22:0/969975302 conn(0x7f25740fba80 msgr2=0x7f25740fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 shutdown_connections 2024-03-24T06:39:49.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.116+0000 7f2579c00640 1 -- 172.21.15.22:0/969975302 wait complete. 2024-03-24T06:39:49.131 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:48 smithi022 ceph-mon[32003]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:49.135 INFO:teuthology.run_tasks:Running task sequential... 2024-03-24T06:39:49.145 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-24T06:39:49.145 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-24T06:39:49.146 INFO:teuthology.task.print:Enabling mgr modules 2024-03-24T06:39:49.146 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-24T06:39:49.146 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T06:39:49.147 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:39:49.147 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-24T06:39:49.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:48 smithi155 ceph-mon[36262]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 -- 172.21.15.22:0/3740585029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 msgr2=0x7ff7a81043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 --2- 172.21.15.22:0/3740585029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a81043d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff798009ca0 tx=0x7ff79802c100 comp rx=0 tx=0).stop 2024-03-24T06:39:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 -- 172.21.15.22:0/3740585029 shutdown_connections 2024-03-24T06:39:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 --2- 172.21.15.22:0/3740585029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8107240 0x7ff7a8109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 --2- 172.21.15.22:0/3740585029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8104910 0x7ff7a8106d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 --2- 172.21.15.22:0/3740585029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a81043d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.357+0000 7ff7adc00640 1 -- 172.21.15.22:0/3740585029 >> 172.21.15.22:0/3740585029 conn(0x7ff7a80fb9f0 msgr2=0x7ff7a80fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.358+0000 7ff7adc00640 1 -- 172.21.15.22:0/3740585029 shutdown_connections 2024-03-24T06:39:49.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.358+0000 7ff7adc00640 1 -- 172.21.15.22:0/3740585029 wait complete. 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 Processor -- start 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 -- start start 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8107240 0x7ff7a81a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81a1310 con 0x7ff7a8101fe0 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81a1480 con 0x7ff7a8104910 2024-03-24T06:39:49.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7adc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81a1640 con 0x7ff7a8107240 2024-03-24T06:39:49.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7a7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7a7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7ad200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8107240 0x7ff7a81a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7a7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35668/0 (socket says 172.21.15.22:35668) 2024-03-24T06:39:49.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.359+0000 7ff7a7400640 1 -- 172.21.15.22:0/1554357608 learned_addr learned my addr 172.21.15.22:0/1554357608 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 -- 172.21.15.22:0/1554357608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8107240 msgr2=0x7ff7a81a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8107240 0x7ff7a81a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 -- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 msgr2=0x7ff7a8199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a8199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 -- 172.21.15.22:0/1554357608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff798009900 con 0x7ff7a8104910 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.360+0000 7ff7a7400640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d770 tx=0x7ff79000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.361+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff790009c30 con 0x7ff7a8104910 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.361+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a81a1920 con 0x7ff7a8104910 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.361+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff79000de80 con 0x7ff7a8104910 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.361+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79000f6b0 con 0x7ff7a8104910 2024-03-24T06:39:49.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.361+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a81a1e30 con 0x7ff7a8104910 2024-03-24T06:39:49.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.362+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a8108d90 con 0x7ff7a8104910 2024-03-24T06:39:49.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.364+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ff790010460 con 0x7ff7a8104910 2024-03-24T06:39:49.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.366+0000 7ff7a4c00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 0x7ff78c0a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:49.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.366+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7ff7900cdd70 con 0x7ff7a8104910 2024-03-24T06:39:49.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.366+0000 7ff7a7e00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 0x7ff78c0a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:49.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.367+0000 7ff7a7e00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 0x7ff78c0a4570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff79802c710 tx=0x7ff7980324e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:49.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.374+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff790096a10 con 0x7ff7a8104910 2024-03-24T06:39:49.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:48 smithi143 ceph-mon[36520]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T06:39:49.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.634+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7ff7a802d1d0 con 0x7ff7a8104910 2024-03-24T06:39:49.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.990+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7ff790096de0 con 0x7ff7a8104910 2024-03-24T06:39:49.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.992+0000 7ff7a4c00640 1 -- 172.21.15.22:0/1554357608 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7ff790075e80 con 0x7ff7a8104910 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 msgr2=0x7ff78c0a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 0x7ff78c0a4570 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff79802c710 tx=0x7ff7980324e0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 msgr2=0x7ff7a8199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff79000d770 tx=0x7ff79000dc30 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 shutdown_connections 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7ff78c0a20b0 0x7ff78c0a4570 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7a8107240 0x7ff7a81a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7a8104910 0x7ff7a8199d60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 --2- 172.21.15.22:0/1554357608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7a8101fe0 0x7ff7a8199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:49.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 >> 172.21.15.22:0/1554357608 conn(0x7ff7a80fb9f0 msgr2=0x7ff7a81017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:49.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 shutdown_connections 2024-03-24T06:39:49.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:49.998+0000 7ff7adc00640 1 -- 172.21.15.22:0/1554357608 wait complete. 2024-03-24T06:39:50.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 -- 172.21.15.22:0/2904883802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c104b60 msgr2=0x7fa55c104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 --2- 172.21.15.22:0/2904883802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c104b60 0x7fa55c104fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009860 tx=0x7fa54c02bb70 comp rx=0 tx=0).stop 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 -- 172.21.15.22:0/2904883802 shutdown_connections 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 --2- 172.21.15.22:0/2904883802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c105520 0x7fa55c10bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 --2- 172.21.15.22:0/2904883802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c104b60 0x7fa55c104fe0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 --2- 172.21.15.22:0/2904883802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c103970 0x7fa55c103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 -- 172.21.15.22:0/2904883802 >> 172.21.15.22:0/2904883802 conn(0x7fa55c0fef00 msgr2=0x7fa55c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 -- 172.21.15.22:0/2904883802 shutdown_connections 2024-03-24T06:39:50.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.213+0000 7fa564800640 1 -- 172.21.15.22:0/2904883802 wait complete. 2024-03-24T06:39:50.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 Processor -- start 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 -- start start 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c104b60 0x7fa55c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 0x7fa55c1a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c1a1450 con 0x7fa55c105520 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c1a15c0 con 0x7fa55c104b60 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.214+0000 7fa564800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c1a1780 con 0x7fa55c103970 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33290/0 (socket says 172.21.15.22:33290) 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 -- 172.21.15.22:0/176263828 learned_addr learned my addr 172.21.15.22:0/176263828 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561400640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c104b60 0x7fa55c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa562800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 0x7fa55c1a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c104b60 msgr2=0x7fa55c199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c104b60 0x7fa55c199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 msgr2=0x7fa55c1a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 0x7fa55c1a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 -- 172.21.15.22:0/176263828 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa558009900 con 0x7fa55c103970 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa562800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 0x7fa55c1a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.215+0000 7fa561e00640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa55000e9c0 tx=0x7fa55000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.216+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550009700 con 0x7fa55c103970 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.216+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa55000cd80 con 0x7fa55c103970 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.216+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550010430 con 0x7fa55c103970 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.216+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa54c009490 con 0x7fa55c103970 2024-03-24T06:39:50.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.216+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55c1a1d60 con 0x7fa55c103970 2024-03-24T06:39:50.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.217+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55c10ac90 con 0x7fa55c103970 2024-03-24T06:39:50.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.221+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7fa550003680 con 0x7fa55c103970 2024-03-24T06:39:50.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.222+0000 7fa556a00640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 0x7fa5440a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.222+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7fa550014070 con 0x7fa55c103970 2024-03-24T06:39:50.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.222+0000 7fa561400640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 msgr2=0x7fa5440a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:50.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.222+0000 7fa561400640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 0x7fa5440a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:39:50.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.228+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa550097090 con 0x7fa55c103970 2024-03-24T06:39:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1554357608' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:49 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: ignoring --setuser ceph since I am not root 2024-03-24T06:39:50.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: ignoring --setgroup ceph since I am not root 2024-03-24T06:39:50.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-mgr[38180]: -- 172.21.15.155:0/3497786014 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1d8e461a0 con 0x55c1d8e1d000 2024-03-24T06:39:50.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:49 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1554357608' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:49 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setuser ceph since I am not root 2024-03-24T06:39:50.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: ignoring --setgroup ceph since I am not root 2024-03-24T06:39:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.423+0000 7fa561400640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 msgr2=0x7fa5440a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.423+0000 7fa561400640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 0x7fa5440a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:39:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:49 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1554357608' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:49 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T06:39:50.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: ignoring --setuser ceph since I am not root 2024-03-24T06:39:50.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: ignoring --setgroup ceph since I am not root 2024-03-24T06:39:50.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-mgr[37948]: -- 172.21.15.143:0/3852705925 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6d80c01a0 con 0x55b6d8096800 2024-03-24T06:39:50.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.495+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa55c10f350 con 0x7fa55c103970 2024-03-24T06:39:50.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.503+0000 7fa556a00640 1 -- 172.21.15.22:0/176263828 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7fa55009a750 con 0x7fa55c103970 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 msgr2=0x7fa5440a46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 0x7fa5440a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 msgr2=0x7fa55c199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa55000e9c0 tx=0x7fa55000ee80 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 shutdown_connections 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7fa5440a2230 0x7fa5440a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa55c105520 0x7fa55c1a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa55c104b60 0x7fa55c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 --2- 172.21.15.22:0/176263828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa55c103970 0x7fa55c199960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 >> 172.21.15.22:0/176263828 conn(0x7fa55c0fef00 msgr2=0x7fa55c107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 shutdown_connections 2024-03-24T06:39:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.506+0000 7fa564800640 1 -- 172.21.15.22:0/176263828 wait complete. 2024-03-24T06:39:50.525 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-24T06:39:50.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 -- 172.21.15.22:0/2857295924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c103970 msgr2=0x7f670c103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 --2- 172.21.15.22:0/2857295924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c103970 0x7f670c103d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6700009860 tx=0x7f670002bb70 comp rx=0 tx=0).stop 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 -- 172.21.15.22:0/2857295924 shutdown_connections 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 --2- 172.21.15.22:0/2857295924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c105520 0x7f670c10bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 --2- 172.21.15.22:0/2857295924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c104b60 0x7f670c104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 --2- 172.21.15.22:0/2857295924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c103970 0x7f670c103d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 -- 172.21.15.22:0/2857295924 >> 172.21.15.22:0/2857295924 conn(0x7f670c0fef00 msgr2=0x7f670c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 -- 172.21.15.22:0/2857295924 shutdown_connections 2024-03-24T06:39:50.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.716+0000 7f6714400640 1 -- 172.21.15.22:0/2857295924 wait complete. 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.717+0000 7f6714400640 1 Processor -- start 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.717+0000 7f6714400640 1 -- start start 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.717+0000 7f6714400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c103970 0x7f670c199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.717+0000 7f6714400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6714400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c105520 0x7f670c1a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6714400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a1410 con 0x7f670c103970 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6714400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a1580 con 0x7f670c105520 2024-03-24T06:39:50.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6714400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1a1740 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c103970 0x7f670c199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33310/0 (socket says 172.21.15.22:33310) 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 -- 172.21.15.22:0/4281565303 learned_addr learned my addr 172.21.15.22:0/4281565303 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6712400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c105520 0x7f670c1a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c105520 msgr2=0x7f670c1a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c105520 0x7f670c1a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c103970 msgr2=0x7f670c199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c103970 0x7f670c199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.718+0000 7f6711000640 1 -- 172.21.15.22:0/4281565303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6700009490 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f6711000640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f66f800e9c0 tx=0x7f66f800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f800ccc0 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66f800ce20 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8010430 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670c1a1a20 con 0x7f670c104b60 2024-03-24T06:39:50.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.719+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f670c1a1f80 con 0x7f670c104b60 2024-03-24T06:39:50.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.720+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670c10abc0 con 0x7f670c104b60 2024-03-24T06:39:50.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.723+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f66f8004f80 con 0x7f670c104b60 2024-03-24T06:39:50.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.725+0000 7f670a000640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 0x7f66f40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:50.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.725+0000 7f6711a00640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 msgr2=0x7f66f40a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:50.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.725+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f66f8014070 con 0x7f670c104b60 2024-03-24T06:39:50.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.725+0000 7f6711a00640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 0x7f66f40a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:39:50.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.732+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66f8096f20 con 0x7f670c104b60 2024-03-24T06:39:50.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.926+0000 7f6711a00640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 msgr2=0x7f66f40a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:50.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.926+0000 7f6711a00640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 0x7f66f40a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:39:50.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:50.566+0000 7f1791c9d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:39:50.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:50.694+0000 7f1791c9d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:39:50.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.934+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f670c10f370 con 0x7f670c104b60 2024-03-24T06:39:50.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.942+0000 7f670a000640 1 -- 172.21.15.22:0/4281565303 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f66f809a5e0 con 0x7f670c104b60 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 msgr2=0x7f66f40a46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 0x7f66f40a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 msgr2=0x7f670c199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f66f800e9c0 tx=0x7f66f800ee80 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 shutdown_connections 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f66f40a2230 0x7f66f40a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c105520 0x7f670c1a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f670c104b60 0x7f670c199e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 --2- 172.21.15.22:0/4281565303 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f670c103970 0x7f670c199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 >> 172.21.15.22:0/4281565303 conn(0x7f670c0fef00 msgr2=0x7f670c107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:50.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 shutdown_connections 2024-03-24T06:39:50.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:50.946+0000 7f6714400640 1 -- 172.21.15.22:0/4281565303 wait complete. 2024-03-24T06:39:50.971 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T06:39:50.984 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T06:39:50.984 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-24T06:39:50.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:50.985 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T06:39:51.031 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-24T06:39:51.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:51.032 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T06:39:51.144 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T06:39:51.145 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-mon[32003]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T06:39:51.145 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.145 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.145 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:50 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:50.875+0000 7ff483433200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:39:51.145 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:51 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:51.024+0000 7ff483433200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:39:51.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:51.157 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link delete ceph-brx 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:50.782+0000 7f71853f5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:50.877+0000 7f71853f5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-mon[36262]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:50 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.298 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-24T06:39:51.298 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T06:39:51.298 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T06:39:51.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T06:39:51.347 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T06:39:51.347 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-mon[36520]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T06:39:51.347 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.347 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:50 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' 2024-03-24T06:39:51.598 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:51 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:51.348+0000 7f1791c9d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:39:51.598 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:51 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:51.441+0000 7f1791c9d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:39:51.792 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:39:51.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:51 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:51.598+0000 7f1791c9d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:39:51.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:51 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:51.695+0000 7f1791c9d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:39:51.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:51 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:51.779+0000 7f1791c9d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:39:52.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:51 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:51.674+0000 7f71853f5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:39:52.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:51 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:51.768+0000 7f71853f5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:39:52.027 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:51 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:51.913+0000 7f71853f5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:39:52.229 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.013+0000 7ff483433200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T06:39:52.229 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.153+0000 7ff483433200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T06:39:52.354 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:52 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:52.046+0000 7f1791c9d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:39:52.354 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:52 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:52.172+0000 7f1791c9d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:52.354 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:52 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:52.259+0000 7f1791c9d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:39:52.485 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.353+0000 7ff483433200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T06:39:52.505 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.032+0000 7f71853f5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:39:52.506 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.150+0000 7f71853f5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:39:52.506 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.505+0000 7f71853f5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:39:52.678 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:52 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:52.355+0000 7f1791c9d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:39:52.678 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:52 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:52.535+0000 7f1791c9d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:39:52.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.660+0000 7f71853f5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:52.893 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.485+0000 7ff483433200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T06:39:52.893 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.600+0000 7ff483433200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T06:39:53.080 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.785+0000 7f71853f5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:39:53.080 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:52 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:52.900+0000 7f71853f5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:39:53.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:52.989+0000 7ff483433200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T06:39:53.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:53 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:53.141+0000 7ff483433200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:53.527 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:53 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:53.080+0000 7f71853f5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:39:53.585 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:53 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:53.266+0000 7ff483433200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T06:39:53.585 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:53 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:53.421+0000 7ff483433200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T06:39:53.898 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:53 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:53.682+0000 7ff483433200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T06:39:53.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:53 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:53.527+0000 7f1791c9d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:39:53.928 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:53 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:53.659+0000 7f1791c9d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 -- 172.21.15.22:0/3400349807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c102df0 msgr2=0x7f457c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 --2- 172.21.15.22:0/3400349807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c102df0 0x7f457c106e40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4568009990 tx=0x7f4568009ca0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 -- 172.21.15.22:0/3400349807 shutdown_connections 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 --2- 172.21.15.22:0/3400349807 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 --2- 172.21.15.22:0/3400349807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c107520 0x7f457c10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 --2- 172.21.15.22:0/3400349807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c102df0 0x7f457c106e40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 -- 172.21.15.22:0/3400349807 >> 172.21.15.22:0/3400349807 conn(0x7f457c0fdcb0 msgr2=0x7f457c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:54.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 -- 172.21.15.22:0/3400349807 shutdown_connections 2024-03-24T06:39:54.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.083+0000 7f4584600700 1 -- 172.21.15.22:0/3400349807 wait complete. 2024-03-24T06:39:54.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.084+0000 7f4584600700 1 Processor -- start 2024-03-24T06:39:54.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 -- start start 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 0x7f457c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f457c19aeb0 con 0x7f457c102440 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f457c19b020 con 0x7f457c102df0 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4584600700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f457c19b1e0 con 0x7f457c107520 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.085+0000 7f4581c00700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4581200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 0x7f457c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:54.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33316/0 (socket says 172.21.15.22:33316) 2024-03-24T06:39:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4581200700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 0x7f457c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35736/0 (socket says 172.21.15.22:35736) 2024-03-24T06:39:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 -- 172.21.15.22:0/487990071 learned_addr learned my addr 172.21.15.22:0/487990071 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:54.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 msgr2=0x7f457c1a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 0x7f457c1a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 msgr2=0x7f457c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4582600700 1 -- 172.21.15.22:0/487990071 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45680095f0 con 0x7f457c107520 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.086+0000 7f4581c00700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f4582600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f457400ef30 tx=0x7f457400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574018070 con 0x7f457c107520 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457400f040 con 0x7f457c107520 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457c19b460 con 0x7f457c107520 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457400cd50 con 0x7f457c107520 2024-03-24T06:39:54.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.087+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457c1a7660 con 0x7f457c107520 2024-03-24T06:39:54.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.088+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f457c103920 con 0x7f457c107520 2024-03-24T06:39:54.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.090+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) ==== 136773+0+0 (secure 0 0 0) 0x7f4574007500 con 0x7f457c107520 2024-03-24T06:39:54.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.093+0000 7f457a000700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 0x7f45640a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:54.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.093+0000 7f4581c00700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 msgr2=0x7f45640a46e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:54.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.093+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f45740cdb80 con 0x7f457c107520 2024-03-24T06:39:54.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.093+0000 7f4581c00700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 0x7f45640a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:39:54.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.095+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4574096800 con 0x7f457c107520 2024-03-24T06:39:54.290 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:54.003+0000 7f1791c9d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:39:54.290 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:54.093+0000 7f1791c9d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:39:54.290 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:54.290+0000 7f1791c9d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:54.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.294+0000 7f4581c00700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 msgr2=0x7f45640a46e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:54.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.294+0000 7f4581c00700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 0x7f45640a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:39:54.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.406+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f457c19bce0 con 0x7f457c107520 2024-03-24T06:39:54.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.407+0000 7f457a000700 1 -- 172.21.15.22:0/487990071 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f4574099ec0 con 0x7f457c107520 2024-03-24T06:39:54.407 INFO:teuthology.orchestra.run.smithi022.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T06:39:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.409+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 msgr2=0x7f45640a46e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:39:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.409+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 0x7f45640a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.409+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 msgr2=0x7f457c19a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.409+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f457400ef30 tx=0x7f457400c3a0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 shutdown_connections 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f457c102440 0x7f457c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f45640a2220 0x7f45640a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f457c107520 0x7f457c19a800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 --2- 172.21.15.22:0/487990071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c102df0 0x7f457c1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 >> 172.21.15.22:0/487990071 conn(0x7f457c0fdcb0 msgr2=0x7f457c0fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 shutdown_connections 2024-03-24T06:39:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:54.410+0000 7f4584600700 1 -- 172.21.15.22:0/487990071 wait complete. 2024-03-24T06:39:54.459 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:54 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:54.189+0000 7f71853f5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:39:54.459 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:54 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:54.362+0000 7f71853f5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:39:54.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:54 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/487990071' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:39:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/487990071' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:39:54.678 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:54.477+0000 7f1791c9d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:39:54.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:54 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/487990071' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T06:39:55.030 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:54 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:54.762+0000 7f1791c9d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:55.135 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:54 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:54.784+0000 7f71853f5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:39:55.135 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:54 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:54.906+0000 7f71853f5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:39:55.135 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:55 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:55.135+0000 7f71853f5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:55.278 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T06:39:55.278 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi022.front.sepia.ceph.com 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T06:39:55.279 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T06:39:55.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:55.279 DEBUG:teuthology.orchestra.run.smithi022:> ip addr 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 ::1/128 scope host 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9a brd ff:ff:ff:ff:ff:ff 2024-03-24T06:39:55.326 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f0 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9b brd ff:ff:ff:ff:ff:ff 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f1 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e4 brd ff:ff:ff:ff:ff:ff 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f0 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: inet 172.21.15.22/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft 41888sec preferred_lft 41888sec 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e5 brd ff:ff:ff:ff:ff:ff 2024-03-24T06:39:55.327 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f1 2024-03-24T06:39:55.328 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add name ceph-brx type bridge 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr flush dev ceph-brx 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set ceph-brx up 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T06:39:55.328 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:39:55.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:55 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:55.031+0000 7f1791c9d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:55.428 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:55 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:55.269+0000 7f1791c9d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:39:55.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:55.495 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:55 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:55.154+0000 7ff483433200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T06:39:55.495 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:55 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:55.354+0000 7ff483433200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T06:39:55.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:55.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:55.653 DEBUG:teuthology.orchestra.run.smithi022:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T06:39:55.723 INFO:teuthology.orchestra.run.smithi022.stdout:1 2024-03-24T06:39:55.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:55.727 DEBUG:teuthology.orchestra.run.smithi022:> route 2024-03-24T06:39:55.773 INFO:teuthology.orchestra.run.smithi022.stdout:Kernel IP routing table 2024-03-24T06:39:55.773 INFO:teuthology.orchestra.run.smithi022.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T06:39:55.773 INFO:teuthology.orchestra.run.smithi022.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T06:39:55.773 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T06:39:55.774 INFO:teuthology.orchestra.run.smithi022.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T06:39:55.774 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:39:55.775 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:39:55.775 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T06:39:55.775 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T06:39:55.775 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T06:39:55.775 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:39:55.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:55 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:55.398+0000 7f71853f5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:39:55.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:55.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:56.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:56.008 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T06:39:56.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:56.051 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list-id 2024-03-24T06:39:56.083 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:55 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:55.779+0000 7f71853f5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:56.146 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:39:56.146 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:39:56.146 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T06:39:56.146 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T06:39:56.146 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:39:56.151 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:55 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:55.882+0000 7ff483433200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T06:39:56.151 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:56 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:56.016+0000 7ff483433200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T06:39:56.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:56.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:56.334 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T06:39:56.334 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T06:39:56.335 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:39:56.358 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:56 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:56.083+0000 7f71853f5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:56.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:56.531 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:56 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:56.342+0000 7ff483433200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:56.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:56.632 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T06:39:56.632 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T06:39:56.633 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set brx.0 up 2024-03-24T06:39:56.633 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T06:39:56.633 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T06:39:56.678 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:56 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:56.190+0000 7f1791c9d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:39:56.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:56.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:56 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:56.358+0000 7f71853f5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:39:56.782 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:56 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:56.661+0000 7ff483433200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T06:39:56.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:56.862 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T06:39:56.862 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T06:39:56.901 INFO:teuthology.orchestra.run.smithi022.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T06:39:56.901 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T06:39:56.901 DEBUG:teuthology.orchestra.run.smithi022:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T06:39:56.988 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-24T06:39:56.988 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-24T06:39:56.988 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/465aaf92-975d-4dec-a92a-7131150f7a52 2024-03-24T06:39:56.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:56.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/465aaf92-975d-4dec-a92a-7131150f7a52 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-24T06:39:57.154 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-24T06:39:57.154 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-24T06:39:57.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:56 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:56.880+0000 7f1791c9d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:39:57.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:56 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:56.979+0000 7f1791c9d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:39:57.398 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:57.079+0000 7ff483433200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: Standby manager daemon y restarted 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: Standby manager daemon y started 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:39:57.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y[37944]: 2024-03-24T06:39:57.194+0000 7f1791c9d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: Standby manager daemon y restarted 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: Standby manager daemon y started 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:39:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:57 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:57.327+0000 7f71853f5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: Standby manager daemon y restarted 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: Standby manager daemon y started 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T06:39:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.143:0/962628206' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:57.795 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:57.458+0000 7ff483433200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T06:39:57.842 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-24T06:39:57.842 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-24T06:39:57.842 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: resolved to: "172.21.15.22:6789,172.21.15.143:6789,172.21.15.155:6789" 2024-03-24T06:39:57.843 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: trying mount with new device syntax: 0@7bca48fe-e9a7-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/465aaf92-975d-4dec-a92a-7131150f7a52 2024-03-24T06:39:57.843 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.22:6789/172.21.15.143:6789/172.21.15.155:6789" will pass to kernel 2024-03-24T06:39:57.848 INFO:teuthology.orchestra.run.smithi022.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T06:39:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: You just mounted a file system that supports labels which does not 2024-03-24T06:39:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T06:39:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T06:39:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T06:39:57.852 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T06:39:57.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:39:57.852 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T06:39:57.942 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T06:39:58.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T06:39:58.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:58.052 INFO:teuthology.orchestra.run.smithi022.stdout:client25513 2024-03-24T06:39:58.052 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:39:58.052 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T06:39:58.093 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T06:39:58.148 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:57.795+0000 7ff483433200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T06:39:58.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T06:39:58.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:58.238 INFO:teuthology.orchestra.run.smithi022.stdout:client25513 2024-03-24T06:39:58.238 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T06:39:58.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:57 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:57.980+0000 7f71853f5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:39:58.277 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:58 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:58.079+0000 7f71853f5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:39:58.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T06:39:58.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:58.387 INFO:teuthology.orchestra.run.smithi022.stdout:client25513 2024-03-24T06:39:58.387 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715.client25513/status') 2024-03-24T06:39:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T06:39:58.512 INFO:teuthology.orchestra.run.smithi022.stdout:instance: client.25513 (3)172.21.15.22:0/491579536 2024-03-24T06:39:58.513 INFO:teuthology.orchestra.run.smithi022.stdout:blocklisted: false 2024-03-24T06:39:58.513 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records in 2024-03-24T06:39:58.513 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records out 2024-03-24T06:39:58.513 INFO:teuthology.orchestra.run.smithi022.stderr:70 bytes copied, 7.0922e-05 s, 987 kB/s 2024-03-24T06:39:58.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T06:39:58.525 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T06:39:58.537 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T06:39:58.537 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:39:58.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-24T06:39:58.607 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:58 smithi022 ceph-mon[32003]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-24T06:39:58.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:58 smithi143 ceph-mon[36520]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-24T06:39:58.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 -- 172.21.15.22:0/1259731472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8102250 msgr2=0x7f7ef8104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:58.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 --2- 172.21.15.22:0/1259731472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8102250 0x7f7ef8104640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009900 tx=0x7f7ee802bc50 comp rx=0 tx=0).stop 2024-03-24T06:39:58.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 -- 172.21.15.22:0/1259731472 shutdown_connections 2024-03-24T06:39:58.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 --2- 172.21.15.22:0/1259731472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:58.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 --2- 172.21.15.22:0/1259731472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8104b80 0x7f7ef8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:58.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 --2- 172.21.15.22:0/1259731472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8102250 0x7f7ef8104640 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:58.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 -- 172.21.15.22:0/1259731472 >> 172.21.15.22:0/1259731472 conn(0x7f7ef80fbba0 msgr2=0x7f7ef80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:58.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 -- 172.21.15.22:0/1259731472 shutdown_connections 2024-03-24T06:39:58.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.743+0000 7f7efe000640 1 -- 172.21.15.22:0/1259731472 wait complete. 2024-03-24T06:39:58.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.744+0000 7f7efe000640 1 Processor -- start 2024-03-24T06:39:58.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.744+0000 7f7efe000640 1 -- start start 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8102250 0x7f7ef8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 0x7f7ef8199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81a1470 con 0x7f7ef8102250 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81a15e0 con 0x7f7ef81074b0 2024-03-24T06:39:58.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7efe000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef81a17a0 con 0x7f7ef8104b80 2024-03-24T06:39:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 0x7f7ef8199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8102250 0x7f7ef8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:57278/0 (socket says 172.21.15.22:57278) 2024-03-24T06:39:58.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 -- 172.21.15.22:0/2676090079 learned_addr learned my addr 172.21.15.22:0/2676090079 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef6a00640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 0x7f7ef8199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:53926/0 (socket says 172.21.15.22:53926) 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 msgr2=0x7f7ef8199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 0x7f7ef8199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8102250 msgr2=0x7f7ef8199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8102250 0x7f7ef8199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:58.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.745+0000 7f7ef7e00640 1 -- 172.21.15.22:0/2676090079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef0009840 con 0x7f7ef81074b0 2024-03-24T06:39:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7ef7e00640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000efd0 tx=0x7f7ef000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0009b40 con 0x7f7ef81074b0 2024-03-24T06:39:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ef0003b70 con 0x7f7ef81074b0 2024-03-24T06:39:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee8009560 con 0x7f7ef81074b0 2024-03-24T06:39:58.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00108f0 con 0x7f7ef81074b0 2024-03-24T06:39:58.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.746+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a1e00 con 0x7f7ef81074b0 2024-03-24T06:39:58.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.747+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef8108fa0 con 0x7f7ef81074b0 2024-03-24T06:39:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.749+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 20) ==== 136773+0+0 (secure 0 0 0) 0x7f7ef0003ce0 con 0x7f7ef81074b0 2024-03-24T06:39:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.750+0000 7f7eefe00640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 0x7f7edc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.750+0000 7f7ef7400640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 msgr2=0x7f7edc0a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.750+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f7ef0014070 con 0x7f7ef81074b0 2024-03-24T06:39:58.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.750+0000 7f7ef7400640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 0x7f7edc0a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:39:58.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.754+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ef0097210 con 0x7f7ef81074b0 2024-03-24T06:39:58.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:58 smithi155 ceph-mon[36262]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-24T06:39:58.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:39:58 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z[38140]: 2024-03-24T06:39:58.347+0000 7f71853f5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:39:58.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.951+0000 7f7ef7400640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 msgr2=0x7f7edc0a46f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:58.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.951+0000 7f7ef7400640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 0x7f7edc0a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:39:58.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:58.995+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f7ef802d130 con 0x7f7ef81074b0 2024-03-24T06:39:59.287 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:58 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:58.927+0000 7ff483433200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T06:39:59.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.314+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 21) ==== 136773+0+0 (secure 0 0 0) 0x7f7ef00975e0 con 0x7f7ef81074b0 2024-03-24T06:39:59.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.315+0000 7f7eefe00640 1 -- 172.21.15.22:0/2676090079 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f7ef004a850 con 0x7f7ef81074b0 2024-03-24T06:39:59.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 msgr2=0x7f7edc0a46f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 0x7f7edc0a46f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 msgr2=0x7f7ef81a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000efd0 tx=0x7f7ef000c6a0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 shutdown_connections 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f7edc0a2230 0x7f7edc0a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ef81074b0 0x7f7ef81a0f30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef8104b80 0x7f7ef8199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 --2- 172.21.15.22:0/2676090079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ef8102250 0x7f7ef8199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 >> 172.21.15.22:0/2676090079 conn(0x7f7ef80fbba0 msgr2=0x7f7ef81019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:59.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.320+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 shutdown_connections 2024-03-24T06:39:59.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.321+0000 7f7efe000640 1 -- 172.21.15.22:0/2676090079 wait complete. 2024-03-24T06:39:59.345 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T06:39:59.357 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-24T06:39:59.378 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 -- 172.21.15.22:0/4244660008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 msgr2=0x7f133410ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 --2- 172.21.15.22:0/4244660008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f133410ff40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f132c009cd0 tx=0x7f132c02c0f0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 -- 172.21.15.22:0/4244660008 shutdown_connections 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 --2- 172.21.15.22:0/4244660008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f133410ff40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 --2- 172.21.15.22:0/4244660008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f13340745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 --2- 172.21.15.22:0/4244660008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f1334073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 -- 172.21.15.22:0/4244660008 >> 172.21.15.22:0/4244660008 conn(0x7f13340fcaf0 msgr2=0x7f13340fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:59.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 -- 172.21.15.22:0/4244660008 shutdown_connections 2024-03-24T06:39:59.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.569+0000 7f1338a00640 1 -- 172.21.15.22:0/4244660008 wait complete. 2024-03-24T06:39:59.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.570+0000 7f1338a00640 1 Processor -- start 2024-03-24T06:39:59.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.570+0000 7f1338a00640 1 -- start start 2024-03-24T06:39:59.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f133419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:59.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f133419e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f13341a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f13341a56c0 con 0x7f1334074170 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f13341a5830 con 0x7f133410db50 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1338a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13341a59f0 con 0x7f1334075750 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1332a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f133419e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f13341a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1332a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f133419e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:53958/0 (socket says 172.21.15.22:53958) 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1332a00640 1 -- 172.21.15.22:0/1329576888 learned_addr learned my addr 172.21.15.22:0/1329576888 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333400640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f133419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:39:59.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 msgr2=0x7f133419e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f133419e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 msgr2=0x7f133419dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f133419dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333e00640 1 -- 172.21.15.22:0/1329576888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c009900 con 0x7f133410db50 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.571+0000 7f1333400640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f133419dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f1333e00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f13341a5180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f132c009bf0 tx=0x7f132c0093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c0329b0 con 0x7f133410db50 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f132c003e50 con 0x7f133410db50 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13341a5c70 con 0x7f133410db50 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c004800 con 0x7f133410db50 2024-03-24T06:39:59.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.572+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13341a6140 con 0x7f133410db50 2024-03-24T06:39:59.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.573+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1334103c10 con 0x7f133410db50 2024-03-24T06:39:59.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.576+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) ==== 136773+0+0 (secure 0 0 0) 0x7f132c00a040 con 0x7f133410db50 2024-03-24T06:39:59.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.577+0000 7f132be00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 0x7f13180a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:39:59.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.578+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f132c0f09c0 con 0x7f133410db50 2024-03-24T06:39:59.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.578+0000 7f1333400640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 msgr2=0x7f13180a4650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:59.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.578+0000 7f1333400640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 0x7f13180a4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:39:59.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.582+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f132c0b9640 con 0x7f133410db50 2024-03-24T06:39:59.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: Standby manager daemon z restarted 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: Standby manager daemon z started 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2676090079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: Standby manager daemon z restarted 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: Standby manager daemon z started 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2676090079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:39:59 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: Standby manager daemon z restarted 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: Standby manager daemon z started 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='mgr.? 172.21.15.155:0/1693320380' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2676090079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:39:59 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T06:39:59.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.779+0000 7f1333400640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 msgr2=0x7f13180a4650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:39:59.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.779+0000 7f1333400640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 0x7f13180a4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T06:39:59.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.782+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f133402d1d0 con 0x7f133410db50 2024-03-24T06:39:59.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.784+0000 7f132be00640 1 -- 172.21.15.22:0/1329576888 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24711 (secure 0 0 0) 0x7f132c0bcd00 con 0x7f133410db50 2024-03-24T06:39:59.784 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:39:59.784 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":93,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:38:57.766765+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:39:59.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 msgr2=0x7f13180a4650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:39:59.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 0x7f13180a4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 msgr2=0x7f13341a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:39:59.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f13341a5180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f132c009bf0 tx=0x7f132c0093d0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 shutdown_connections 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f13180a2190 0x7f13180a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f133410db50 0x7f13341a5180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1334075750 0x7f133419e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 --2- 172.21.15.22:0/1329576888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1334074170 0x7f133419dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 >> 172.21.15.22:0/1329576888 conn(0x7f13340fcaf0 msgr2=0x7f1334100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 shutdown_connections 2024-03-24T06:39:59.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.786+0000 7f1338a00640 1 -- 172.21.15.22:0/1329576888 wait complete. 2024-03-24T06:39:59.803 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:39:59.803 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.999+0000 7f5ef2800640 1 -- 172.21.15.22:0/272228223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec107400 msgr2=0x7f5eec109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:39:59.999+0000 7f5ef2800640 1 --2- 172.21.15.22:0/272228223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec107400 0x7f5eec109850 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009ca0 tx=0x7f5ee402c0f0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 -- 172.21.15.22:0/272228223 shutdown_connections 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 --2- 172.21.15.22:0/272228223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec107400 0x7f5eec109850 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 --2- 172.21.15.22:0/272228223 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec104ad0 0x7f5eec106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 --2- 172.21.15.22:0/272228223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec1021a0 0x7f5eec104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 -- 172.21.15.22:0/272228223 >> 172.21.15.22:0/272228223 conn(0x7f5eec0fbbb0 msgr2=0x7f5eec0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 -- 172.21.15.22:0/272228223 shutdown_connections 2024-03-24T06:40:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.000+0000 7f5ef2800640 1 -- 172.21.15.22:0/272228223 wait complete. 2024-03-24T06:40:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 Processor -- start 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 -- start start 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec107400 0x7f5eec1a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a1460 con 0x7f5eec107400 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a15d0 con 0x7f5eec1021a0 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5ef2800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a1790 con 0x7f5eec104ad0 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eeaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eeb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eeb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:57316/0 (socket says 172.21.15.22:57316) 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eeaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:53986/0 (socket says 172.21.15.22:53986) 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eeb400640 1 -- 172.21.15.22:0/831848965 learned_addr learned my addr 172.21.15.22:0/831848965 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.001+0000 7f5eebe00640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec107400 0x7f5eec1a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 -- 172.21.15.22:0/831848965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 msgr2=0x7f5eec199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 -- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec107400 msgr2=0x7f5eec1a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec107400 0x7f5eec1a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 -- 172.21.15.22:0/831848965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee4009900 con 0x7f5eec1021a0 2024-03-24T06:40:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeaa00640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5eeb400640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00e9f0 tx=0x7f5edc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc009720 con 0x7f5eec1021a0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5edc00cd80 con 0x7f5eec1021a0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eec1a1a70 con 0x7f5eec1021a0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.002+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc010690 con 0x7f5eec1021a0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.003+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eec1a1f80 con 0x7f5eec1021a0 2024-03-24T06:40:00.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.004+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eec108ef0 con 0x7f5eec1021a0 2024-03-24T06:40:00.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.005+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 21) ==== 136773+0+0 (secure 0 0 0) 0x7f5edc0046c0 con 0x7f5eec1021a0 2024-03-24T06:40:00.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.007+0000 7f5ee3e00640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 0x7f5ed00a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.007+0000 7f5eeaa00640 1 -- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 msgr2=0x7f5ed00a4650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/503368228 2024-03-24T06:40:00.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.007+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9121+0+0 (secure 0 0 0) 0x7f5edc014070 con 0x7f5eec1021a0 2024-03-24T06:40:00.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.007+0000 7f5eeaa00640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 0x7f5ed00a4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T06:40:00.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.011+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5edc096fd0 con 0x7f5eec1021a0 2024-03-24T06:40:00.107 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:59.668+0000 7ff483433200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T06:40:00.107 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:39:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:39:59.808+0000 7ff483433200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T06:40:00.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.131+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 22) ==== 136441+0+0 (secure 0 0 0) 0x7f5edc0973a0 con 0x7f5eec1021a0 2024-03-24T06:40:00.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.131+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 msgr2=0x7f5ed00a4650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:40:00.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.131+0000 7f5ee3e00640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 0x7f5ed00a4650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.243+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5eec02d130 con 0x7f5eec1021a0 2024-03-24T06:40:00.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.244+0000 7f5ee3e00640 1 -- 172.21.15.22:0/831848965 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f5edc0ca680 con 0x7f5eec1021a0 2024-03-24T06:40:00.244 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:40:00.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 msgr2=0x7f5eec199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00e9f0 tx=0x7f5edc00edf0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 shutdown_connections 2024-03-24T06:40:00.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:6800/503368228,v1:172.21.15.22:6801/503368228] conn(0x7f5ed00a2190 0x7f5ed00a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5eec107400 0x7f5eec1a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5eec104ad0 0x7f5eec199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 --2- 172.21.15.22:0/831848965 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5eec1021a0 0x7f5eec199970 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 >> 172.21.15.22:0/831848965 conn(0x7f5eec0fbbb0 msgr2=0x7f5eec1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 shutdown_connections 2024-03-24T06:40:00.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.246+0000 7f5ef2800640 1 -- 172.21.15.22:0/831848965 wait complete. 2024-03-24T06:40:00.267 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:40:00.359 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.107+0000 7ff483433200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1329576888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: overall HEALTH_OK 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: Active manager daemon x restarted 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: Activating manager daemon x 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T06:40:00.360 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: mgrmap e22: x(active, starting, since 0.0210566s), standbys: y, z 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:40:00.361 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:40:00.362 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:40:00.363 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: Manager daemon x is now available 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.364 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.365 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/831848965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:00.365 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.365 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.365 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.365 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 -- 172.21.15.22:0/3229297038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c4101610 msgr2=0x7ff1c4101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 --2- 172.21.15.22:0/3229297038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c4101610 0x7ff1c4101a30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009900 tx=0x7ff1b402bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 -- 172.21.15.22:0/3229297038 shutdown_connections 2024-03-24T06:40:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 --2- 172.21.15.22:0/3229297038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c41031c0 0x7ff1c4109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 --2- 172.21.15.22:0/3229297038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 --2- 172.21.15.22:0/3229297038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c4101610 0x7ff1c4101a30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 -- 172.21.15.22:0/3229297038 >> 172.21.15.22:0/3229297038 conn(0x7ff1c40fcb80 msgr2=0x7ff1c40fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:00.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 -- 172.21.15.22:0/3229297038 shutdown_connections 2024-03-24T06:40:00.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.510+0000 7ff1c9600640 1 -- 172.21.15.22:0/3229297038 wait complete. 2024-03-24T06:40:00.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.511+0000 7ff1c9600640 1 Processor -- start 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.511+0000 7ff1c9600640 1 -- start start 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 0x7ff1c4110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 0x7ff1c41ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c4117d70 con 0x7ff1c4102800 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c4117ee0 con 0x7ff1c4101610 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c9600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c41ae3b0 con 0x7ff1c41031c0 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 0x7ff1c4110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4110d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58030/0 (socket says 172.21.15.22:58030) 2024-03-24T06:40:00.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c3400640 1 -- 172.21.15.22:0/2564707626 learned_addr learned my addr 172.21.15.22:0/2564707626 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.512+0000 7ff1c8c00640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 0x7ff1c41ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 -- 172.21.15.22:0/2564707626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 msgr2=0x7ff1c41ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 0x7ff1c41ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 -- 172.21.15.22:0/2564707626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 msgr2=0x7ff1c4110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 -- 172.21.15.22:0/2564707626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b4009560 con 0x7ff1c4101610 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c8c00640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 0x7ff1c41ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c3e00640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 0x7ff1c4110810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036040 tx=0x7ff1b40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40044e0 con 0x7ff1c4101610 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b40054e0 con 0x7ff1c4101610 2024-03-24T06:40:00.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.513+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c41ae5b0 con 0x7ff1c4101610 2024-03-24T06:40:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.514+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4003e90 con 0x7ff1c4101610 2024-03-24T06:40:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.514+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c41aeae0 con 0x7ff1c4101610 2024-03-24T06:40:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.515+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c41088d0 con 0x7ff1c4101610 2024-03-24T06:40:00.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.517+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 22) ==== 136441+0+0 (secure 0 0 0) 0x7ff1b40035f0 con 0x7ff1c4101610 2024-03-24T06:40:00.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.518+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff1b40f04d0 con 0x7ff1c4101610 2024-03-24T06:40:00.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.522+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1b40b94e0 con 0x7ff1c4101610 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1329576888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: overall HEALTH_OK 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: Active manager daemon x restarted 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: Activating manager daemon x 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: mgrmap e22: x(active, starting, since 0.0210566s), standbys: y, z 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:40:00.524 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:40:00.525 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:40:00.526 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: Manager daemon x is now available 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/831848965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:00.527 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.528 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.528 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.528 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:00 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.567 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T06:40:00.567 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:40:00.567 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-24T06:40:00.567 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1329576888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: overall HEALTH_OK 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: Active manager daemon x restarted 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: Activating manager daemon x 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: mgrmap e22: x(active, starting, since 0.0210566s), standbys: y, z 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:40:00.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T06:40:00.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T06:40:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: Manager daemon x is now available 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/831848965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:00 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T06:40:00.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.747+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1c402d130 con 0x7ff1c4101610 2024-03-24T06:40:00.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.748+0000 7ff1c0c00640 1 -- 172.21.15.22:0/2564707626 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7ff1b4043020 con 0x7ff1c4101610 2024-03-24T06:40:00.748 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:00.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.750+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 msgr2=0x7ff1c4110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.750+0000 7ff1c9600640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 0x7ff1c4110810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036040 tx=0x7ff1b40042f0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.750+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 shutdown_connections 2024-03-24T06:40:00.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1c41031c0 0x7ff1c41ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1c4102800 0x7ff1c4110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 --2- 172.21.15.22:0/2564707626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1c4101610 0x7ff1c4110810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 >> 172.21.15.22:0/2564707626 conn(0x7ff1c40fcb80 msgr2=0x7ff1c4105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:00.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 shutdown_connections 2024-03-24T06:40:00.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.751+0000 7ff1c9600640 1 -- 172.21.15.22:0/2564707626 wait complete. 2024-03-24T06:40:00.768 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d9a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d9a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.963 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d9a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.964 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d9a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.964 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:40:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:40:00.671+0000 7ff3d9a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:40:00.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 -- 172.21.15.22:0/4091927151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 msgr2=0x7fab9c073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 --2- 172.21.15.22:0/4091927151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c073ba0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009900 tx=0x7fab8c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:00.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 -- 172.21.15.22:0/4091927151 shutdown_connections 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 --2- 172.21.15.22:0/4091927151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c10ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 --2- 172.21.15.22:0/4091927151 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 --2- 172.21.15.22:0/4091927151 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c073ba0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 -- 172.21.15.22:0/4091927151 >> 172.21.15.22:0/4091927151 conn(0x7fab9c0fcb80 msgr2=0x7fab9c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 -- 172.21.15.22:0/4091927151 shutdown_connections 2024-03-24T06:40:00.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.967+0000 7fab9be00640 1 -- 172.21.15.22:0/4091927151 wait complete. 2024-03-24T06:40:00.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 Processor -- start 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 -- start start 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c19dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c1a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c1a57b0 con 0x7fab9c074170 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c1a5920 con 0x7fab9c10dbe0 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c1a5ae0 con 0x7fab9c075750 2024-03-24T06:40:00.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.969+0000 7fab9aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c19dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c19dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58072/0 (socket says 172.21.15.22:58072) 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c1a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9aa00640 1 -- 172.21.15.22:0/4163132525 learned_addr learned my addr 172.21.15.22:0/4163132525 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 -- 172.21.15.22:0/4163132525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 msgr2=0x7fab9c19e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c19e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 -- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 msgr2=0x7fab9c19dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c19dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 -- 172.21.15.22:0/4163132525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab94009840 con 0x7fab9c10dbe0 2024-03-24T06:40:00.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9a000640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.970+0000 7fab9b400640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c1a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fab9400ec50 tx=0x7fab9400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.971+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94003b70 con 0x7fab9c10dbe0 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.971+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9400ce70 con 0x7fab9c10dbe0 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.971+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94010830 con 0x7fab9c10dbe0 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.971+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8c009560 con 0x7fab9c10dbe0 2024-03-24T06:40:00.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.971+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c1a6140 con 0x7fab9c10dbe0 2024-03-24T06:40:00.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.972+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c103db0 con 0x7fab9c10dbe0 2024-03-24T06:40:00.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.974+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 22) ==== 136441+0+0 (secure 0 0 0) 0x7fab94004020 con 0x7fab9c10dbe0 2024-03-24T06:40:00.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.976+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fab94014070 con 0x7fab9c10dbe0 2024-03-24T06:40:00.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:00.979+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab94097130 con 0x7fab9c10dbe0 2024-03-24T06:40:01.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.140+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 23) ==== 136568+0+0 (secure 0 0 0) 0x7fab940042d0 con 0x7fab9c10dbe0 2024-03-24T06:40:01.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.140+0000 7fab93400640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 0x7fab800ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.150+0000 7fab9aa00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 0x7fab800ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.154+0000 7fab9aa00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 0x7fab800ac8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab8c006010 tx=0x7fab8c02c810 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:01.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.254+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fab9c02d1d0 con 0x7fab9c10dbe0 2024-03-24T06:40:01.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.254+0000 7fab93400640 1 -- 172.21.15.22:0/4163132525 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fab9408cc80 con 0x7fab9c10dbe0 2024-03-24T06:40:01.254 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:40:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 msgr2=0x7fab800ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 0x7fab800ac8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab8c006010 tx=0x7fab8c02c810 comp rx=0 tx=0).stop 2024-03-24T06:40:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 msgr2=0x7fab9c1a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c1a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fab9400ec50 tx=0x7fab9400c6a0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 shutdown_connections 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fab800aa4d0 0x7fab800ac8c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab9c10dbe0 0x7fab9c1a5240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab9c075750 0x7fab9c19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 --2- 172.21.15.22:0/4163132525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab9c074170 0x7fab9c19dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 >> 172.21.15.22:0/4163132525 conn(0x7fab9c0fcb80 msgr2=0x7fab9c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 shutdown_connections 2024-03-24T06:40:01.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.257+0000 7fab9be00640 1 -- 172.21.15.22:0/4163132525 wait complete. 2024-03-24T06:40:01.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:40:01.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 -- 172.21.15.22:0/1569383961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c101610 msgr2=0x7fb56c101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 --2- 172.21.15.22:0/1569383961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c101610 0x7fb56c101a30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009900 tx=0x7fb55c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:01.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 -- 172.21.15.22:0/1569383961 shutdown_connections 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 --2- 172.21.15.22:0/1569383961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c1031c0 0x7fb56c109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 --2- 172.21.15.22:0/1569383961 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb56c102800 0x7fb56c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 --2- 172.21.15.22:0/1569383961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c101610 0x7fb56c101a30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 -- 172.21.15.22:0/1569383961 >> 172.21.15.22:0/1569383961 conn(0x7fb56c0fcb80 msgr2=0x7fb56c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 -- 172.21.15.22:0/1569383961 shutdown_connections 2024-03-24T06:40:01.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.469+0000 7fb571000640 1 -- 172.21.15.22:0/1569383961 wait complete. 2024-03-24T06:40:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.470+0000 7fb571000640 1 Processor -- start 2024-03-24T06:40:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.470+0000 7fb571000640 1 -- start start 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c102800 0x7fb56c199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb56c1031c0 0x7fb56c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1a1330 con 0x7fb56c1031c0 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1a14a0 con 0x7fb56c102800 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb571000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c1a1660 con 0x7fb56c101610 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54182/0 (socket says 172.21.15.22:54182) 2024-03-24T06:40:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 -- 172.21.15.22:0/4124560072 learned_addr learned my addr 172.21.15.22:0/4124560072 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56aa00640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c102800 0x7fb56c199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 -- 172.21.15.22:0/4124560072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c102800 msgr2=0x7fb56c199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c102800 0x7fb56c199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 -- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb56c1031c0 msgr2=0x7fb56c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb56c1031c0 0x7fb56c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.471+0000 7fb56b400640 1 -- 172.21.15.22:0/4124560072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb55c009560 con 0x7fb56c101610 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb56b400640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0043d0 con 0x7fb56c101610 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c0052d0 con 0x7fb56c101610 2024-03-24T06:40:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c1a18e0 con 0x7fb56c101610 2024-03-24T06:40:01.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c003ce0 con 0x7fb56c101610 2024-03-24T06:40:01.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.472+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c1a1e40 con 0x7fb56c101610 2024-03-24T06:40:01.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.474+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c10d070 con 0x7fb56c101610 2024-03-24T06:40:01.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.477+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 23) ==== 136568+0+0 (secure 0 0 0) 0x7fb55c004df0 con 0x7fb56c101610 2024-03-24T06:40:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.478+0000 7fb563e00640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 0x7fb5500a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.478+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb55c0f0470 con 0x7fb56c101610 2024-03-24T06:40:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.478+0000 7fb56aa00640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 0x7fb5500a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.479+0000 7fb56aa00640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 0x7fb5500a4400 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb56c19ad50 tx=0x7fb558005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:01.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.484+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb55c0b92b0 con 0x7fb56c101610 2024-03-24T06:40:01.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:01 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2564707626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:01 smithi022 ceph-mon[32003]: mgrmap e23: x(active, since 1.02416s), standbys: y, z 2024-03-24T06:40:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:01 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:01 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4163132525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:01 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:01 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2564707626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:01 smithi143 ceph-mon[36520]: mgrmap e23: x(active, since 1.02416s), standbys: y, z 2024-03-24T06:40:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:01 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:01 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4163132525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:01 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.682+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb56c1a2100 con 0x7fb56c101610 2024-03-24T06:40:01.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.683+0000 7fb563e00640 1 -- 172.21.15.22:0/4124560072 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb55c0bc970 con 0x7fb56c101610 2024-03-24T06:40:01.683 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 msgr2=0x7fb5500a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 0x7fb5500a4400 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb56c19ad50 tx=0x7fb558005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 msgr2=0x7fb56c199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c005e80 comp rx=0 tx=0).stop 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 shutdown_connections 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5500a1f40 0x7fb5500a4400 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb56c1031c0 0x7fb56c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb56c102800 0x7fb56c199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 --2- 172.21.15.22:0/4124560072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb56c101610 0x7fb56c199840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.685+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 >> 172.21.15.22:0/4124560072 conn(0x7fb56c0fcb80 msgr2=0x7fb56c073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:01.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.686+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 shutdown_connections 2024-03-24T06:40:01.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.686+0000 7fb571000640 1 -- 172.21.15.22:0/4124560072 wait complete. 2024-03-24T06:40:01.705 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:40:01.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:01 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2564707626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:01 smithi155 ceph-mon[36262]: mgrmap e23: x(active, since 1.02416s), standbys: y, z 2024-03-24T06:40:01.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:01 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:01 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4163132525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:01.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:01 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 -- 172.21.15.22:0/3330114998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 msgr2=0x7f11fc074550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 --2- 172.21.15.22:0/3330114998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc074550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009900 tx=0x7f11e802bcd0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 -- 172.21.15.22:0/3330114998 shutdown_connections 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 --2- 172.21.15.22:0/3330114998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc109420 0x7f11fc10b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 --2- 172.21.15.22:0/3330114998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc074550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 --2- 172.21.15.22:0/3330114998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc0756d0 0x7f11fc073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.913+0000 7f1203000640 1 -- 172.21.15.22:0/3330114998 >> 172.21.15.22:0/3330114998 conn(0x7f11fc0fcae0 msgr2=0x7f11fc0fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.914+0000 7f1203000640 1 -- 172.21.15.22:0/3330114998 shutdown_connections 2024-03-24T06:40:01.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.914+0000 7f1203000640 1 -- 172.21.15.22:0/3330114998 wait complete. 2024-03-24T06:40:01.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 Processor -- start 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 -- start start 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 0x7f11fc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 0x7f11fc1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a12d0 con 0x7f11fc0740f0 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a1440 con 0x7f11fc0756d0 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1203000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc1a1600 con 0x7f11fc109420 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1201c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1201200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 0x7f11fc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1201200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 0x7f11fc199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:57488/0 (socket says 172.21.15.22:57488) 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1201200640 1 -- 172.21.15.22:0/1977010546 learned_addr learned my addr 172.21.15.22:0/1977010546 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1201c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc1997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58104/0 (socket says 172.21.15.22:58104) 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.915+0000 7f1202600640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 0x7f11fc1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 -- 172.21.15.22:0/1977010546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 msgr2=0x7f11fc199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 0x7f11fc199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 -- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 msgr2=0x7f11fc1997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc1997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 -- 172.21.15.22:0/1977010546 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8009560 con 0x7f11fc109420 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1202600640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 0x7f11fc1a0d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009fd0 tx=0x7f11f400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f400ce20 con 0x7f11fc109420 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.916+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc1a18e0 con 0x7f11fc109420 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.917+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11f4005020 con 0x7f11fc109420 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.917+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4010690 con 0x7f11fc109420 2024-03-24T06:40:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.917+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc1a1df0 con 0x7f11fc109420 2024-03-24T06:40:01.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.918+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc1081e0 con 0x7f11fc109420 2024-03-24T06:40:01.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.920+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 23) ==== 136568+0+0 (secure 0 0 0) 0x7f11f4005500 con 0x7f11fc109420 2024-03-24T06:40:01.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.922+0000 7f11fa000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 0x7f11e40a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:01.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.922+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f11f4014070 con 0x7f11fc109420 2024-03-24T06:40:01.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.922+0000 7f1201c00640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 0x7f11e40a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:01.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.923+0000 7f1201c00640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 0x7f11e40a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009bd0 tx=0x7f11f0005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:01.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:01.929+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f11f40972d0 con 0x7f11fc109420 2024-03-24T06:40:02.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.163+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11fc02d130 con 0x7f11fc109420 2024-03-24T06:40:02.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.165+0000 7f11fa000640 1 -- 172.21.15.22:0/1977010546 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25085 (secure 0 0 0) 0x7f11f4005190 con 0x7f11fc109420 2024-03-24T06:40:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:02.166 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":94,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:40:00.112797+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 msgr2=0x7f11e40a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 0x7f11e40a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009bd0 tx=0x7f11f0005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 msgr2=0x7f11fc1a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 0x7f11fc1a0d90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009fd0 tx=0x7f11f400ef10 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 shutdown_connections 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11e40a1ec0 0x7f11e40a4380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11fc109420 0x7f11fc1a0d90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11fc0756d0 0x7f11fc199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 --2- 172.21.15.22:0/1977010546 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11fc0740f0 0x7f11fc1997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 >> 172.21.15.22:0/1977010546 conn(0x7f11fc0fcae0 msgr2=0x7f11fc1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:02.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 shutdown_connections 2024-03-24T06:40:02.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.169+0000 7f1203000640 1 -- 172.21.15.22:0/1977010546 wait complete. 2024-03-24T06:40:02.188 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:40:02.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:40:02.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.391+0000 7f9055e00640 1 -- 172.21.15.22:0/1619050838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050101720 msgr2=0x7f9050101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.391+0000 7f9055e00640 1 --2- 172.21.15.22:0/1619050838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050101720 0x7f9050101b40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9040009900 tx=0x7f904002bc50 comp rx=0 tx=0).stop 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 -- 172.21.15.22:0/1619050838 shutdown_connections 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 --2- 172.21.15.22:0/1619050838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90501032d0 0x7f9050109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 --2- 172.21.15.22:0/1619050838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9050102910 0x7f9050102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 --2- 172.21.15.22:0/1619050838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050101720 0x7f9050101b40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 -- 172.21.15.22:0/1619050838 >> 172.21.15.22:0/1619050838 conn(0x7f90500fccb0 msgr2=0x7f90500ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 -- 172.21.15.22:0/1619050838 shutdown_connections 2024-03-24T06:40:02.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 -- 172.21.15.22:0/1619050838 wait complete. 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.392+0000 7f9055e00640 1 Processor -- start 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 -- start start 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 0x7f90500754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 0x7f9050073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90501032d0 0x7f9050074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f90500747a0 con 0x7f90501032d0 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9050074910 con 0x7f9050101720 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f9055e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f90501a1880 con 0x7f9050102910 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f904e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 0x7f9050073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f904ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 0x7f90500754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f904e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 0x7f9050073b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54228/0 (socket says 172.21.15.22:54228) 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f904e000640 1 -- 172.21.15.22:0/2736451135 learned_addr learned my addr 172.21.15.22:0/2736451135 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.393+0000 7f904f400640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90501032d0 0x7f9050074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 -- 172.21.15.22:0/2736451135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 msgr2=0x7f9050073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 0x7f9050073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 -- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90501032d0 msgr2=0x7f9050074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90501032d0 0x7f9050074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 -- 172.21.15.22:0/2736451135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9040009560 con 0x7f9050101720 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f904ea00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 0x7f90500754c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9040036040 tx=0x7f9040005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90400043d0 con 0x7f9050101720 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90400052d0 con 0x7f9050101720 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90501a1ad0 con 0x7f9050101720 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.394+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040003cd0 con 0x7f9050101720 2024-03-24T06:40:02.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.395+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90501a1fe0 con 0x7f9050101720 2024-03-24T06:40:02.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.396+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9050108ae0 con 0x7f9050101720 2024-03-24T06:40:02.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.397+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 23) ==== 136568+0+0 (secure 0 0 0) 0x7f9040005440 con 0x7f9050101720 2024-03-24T06:40:02.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.400+0000 7f9054a00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 0x7f90340a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.400+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f90400efec0 con 0x7f9050101720 2024-03-24T06:40:02.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.400+0000 7f904e000640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 0x7f90340a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.401+0000 7f904e000640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 0x7f90340a4380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9050075230 tx=0x7f903c005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:02.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.403+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f90400b8cc0 con 0x7f9050101720 2024-03-24T06:40:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.610+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f905002d1d0 con 0x7f9050101720 2024-03-24T06:40:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.610+0000 7f9054a00640 1 -- 172.21.15.22:0/2736451135 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f90400bc380 con 0x7f9050101720 2024-03-24T06:40:02.610 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 msgr2=0x7f90340a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 0x7f90340a4380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9050075230 tx=0x7f903c005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 msgr2=0x7f90500754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 0x7f90500754c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9040036040 tx=0x7f9040005e50 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 shutdown_connections 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f90340a1ec0 0x7f90340a4380 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90501032d0 0x7f9050074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9050102910 0x7f9050073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 --2- 172.21.15.22:0/2736451135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9050101720 0x7f90500754c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 >> 172.21.15.22:0/2736451135 conn(0x7f90500fccb0 msgr2=0x7f9050105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 shutdown_connections 2024-03-24T06:40:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.614+0000 7f9055e00640 1 -- 172.21.15.22:0/2736451135 wait complete. 2024-03-24T06:40:02.631 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:40:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: [24/Mar/2024:06:40:01] ENGINE Bus STARTING 2024-03-24T06:40:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: [24/Mar/2024:06:40:01] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T06:40:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: [24/Mar/2024:06:40:01] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: [24/Mar/2024:06:40:01] ENGINE Bus STARTED 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4124560072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:02 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1977010546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:02.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 -- 172.21.15.22:0/3445123725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0ffec0 msgr2=0x7fcfac1002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 --2- 172.21.15.22:0/3445123725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1002e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009900 tx=0x7fcf9c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 -- 172.21.15.22:0/3445123725 shutdown_connections 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 --2- 172.21.15.22:0/3445123725 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac1011c0 0x7fcfac101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 --2- 172.21.15.22:0/3445123725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 --2- 172.21.15.22:0/3445123725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1002e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 -- 172.21.15.22:0/3445123725 >> 172.21.15.22:0/3445123725 conn(0x7fcfac0fba80 msgr2=0x7fcfac0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 -- 172.21.15.22:0/3445123725 shutdown_connections 2024-03-24T06:40:02.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.842+0000 7fcfb1800640 1 -- 172.21.15.22:0/3445123725 wait complete. 2024-03-24T06:40:02.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 Processor -- start 2024-03-24T06:40:02.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 -- start start 2024-03-24T06:40:02.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac1a1390 con 0x7fcfac0ffec0 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac1a1500 con 0x7fcfac100820 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.843+0000 7fcfb1800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac1a16c0 con 0x7fcfac1011c0 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfa3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfabe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54242/0 (socket says 172.21.15.22:54242) 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfa3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:57520/0 (socket says 172.21.15.22:57520) 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 -- 172.21.15.22:0/1544400246 learned_addr learned my addr 172.21.15.22:0/1544400246 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfabe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58154/0 (socket says 172.21.15.22:58154) 2024-03-24T06:40:02.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 -- 172.21.15.22:0/1544400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 msgr2=0x7fcfac199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 -- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 msgr2=0x7fcfac1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.844+0000 7fcfb0e00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfb0e00640 1 -- 172.21.15.22:0/1544400246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa4009840 con 0x7fcfac1011c0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfa3e00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfb0e00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400efd0 tx=0x7fcfa400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4009b40 con 0x7fcfac1011c0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa4003b70 con 0x7fcfac1011c0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9c009560 con 0x7fcfac1011c0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa40108f0 con 0x7fcfac1011c0 2024-03-24T06:40:02.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.845+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfac1a1d20 con 0x7fcfac1011c0 2024-03-24T06:40:02.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.847+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfac10ae00 con 0x7fcfac1011c0 2024-03-24T06:40:02.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.848+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 23) ==== 136568+0+0 (secure 0 0 0) 0x7fcfa4003ce0 con 0x7fcfac1011c0 2024-03-24T06:40:02.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.850+0000 7fcfa9600640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 0x7fcf900a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:02.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.850+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fcfa4014070 con 0x7fcfac1011c0 2024-03-24T06:40:02.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.850+0000 7fcfabe00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 0x7fcf900a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:02.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.851+0000 7fcfabe00640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 0x7fcf900a4400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009230 tx=0x7fcf9c005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:02.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.854+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcfa40970a0 con 0x7fcfac1011c0 2024-03-24T06:40:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: [24/Mar/2024:06:40:01] ENGINE Bus STARTING 2024-03-24T06:40:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: [24/Mar/2024:06:40:01] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T06:40:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: [24/Mar/2024:06:40:01] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: [24/Mar/2024:06:40:01] ENGINE Bus STARTED 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4124560072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:02.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:02 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1977010546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: [24/Mar/2024:06:40:01] ENGINE Bus STARTING 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: [24/Mar/2024:06:40:01] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: [24/Mar/2024:06:40:01] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: [24/Mar/2024:06:40:01] ENGINE Bus STARTED 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4124560072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:02.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:02.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:02 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1977010546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:02.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:02.930+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mgrmap(e 24) ==== 136876+0+0 (secure 0 0 0) 0x7fcfa4097470 con 0x7fcfac1011c0 2024-03-24T06:40:03.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.083+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcfac10a870 con 0x7fcfac1011c0 2024-03-24T06:40:03.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.083+0000 7fcfa9600640 1 -- 172.21.15.22:0/1544400246 <== mon.1 v2:172.21.15.155:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fcfa4090060 con 0x7fcfac1011c0 2024-03-24T06:40:03.083 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 msgr2=0x7fcf900a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 0x7fcf900a4400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009230 tx=0x7fcf9c005fd0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 msgr2=0x7fcfac1a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400efd0 tx=0x7fcfa400c6a0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 shutdown_connections 2024-03-24T06:40:03.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf900a1f40 0x7fcf900a4400 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfac1011c0 0x7fcfac1a0e50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfac100820 0x7fcfac199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 --2- 172.21.15.22:0/1544400246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac0ffec0 0x7fcfac1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 >> 172.21.15.22:0/1544400246 conn(0x7fcfac0fba80 msgr2=0x7fcfac0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 shutdown_connections 2024-03-24T06:40:03.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.086+0000 7fcfb1800640 1 -- 172.21.15.22:0/1544400246 wait complete. 2024-03-24T06:40:03.107 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 -- 172.21.15.22:0/725113935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c102930 msgr2=0x7f8a8c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 --2- 172.21.15.22:0/725113935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c102930 0x7f8a8c102db0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009900 tx=0x7f8a7802bc50 comp rx=0 tx=0).stop 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 -- 172.21.15.22:0/725113935 shutdown_connections 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 --2- 172.21.15.22:0/725113935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c1032f0 0x7f8a8c109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 --2- 172.21.15.22:0/725113935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c102930 0x7f8a8c102db0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 --2- 172.21.15.22:0/725113935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c101740 0x7f8a8c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 -- 172.21.15.22:0/725113935 >> 172.21.15.22:0/725113935 conn(0x7f8a8c0fccb0 msgr2=0x7f8a8c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 -- 172.21.15.22:0/725113935 shutdown_connections 2024-03-24T06:40:03.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.308+0000 7f8a94000640 1 -- 172.21.15.22:0/725113935 wait complete. 2024-03-24T06:40:03.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 Processor -- start 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 -- start start 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c102930 0x7f8a8c197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c1032f0 0x7f8a8c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c19f2d0 con 0x7f8a8c101740 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c19f440 con 0x7f8a8c102930 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a94000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8c19f600 con 0x7f8a8c1032f0 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a91600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.309+0000 7f8a90c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c102930 0x7f8a8c197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58168/0 (socket says 172.21.15.22:58168) 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 -- 172.21.15.22:0/1419302433 learned_addr learned my addr 172.21.15.22:0/1419302433 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a92000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c1032f0 0x7f8a8c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 -- 172.21.15.22:0/1419302433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c1032f0 msgr2=0x7f8a8c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c1032f0 0x7f8a8c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 -- 172.21.15.22:0/1419302433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c102930 msgr2=0x7f8a8c197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c102930 0x7f8a8c197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 -- 172.21.15.22:0/1419302433 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a78009560 con 0x7f8a8c101740 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a91600640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000e9c0 tx=0x7f8a8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.310+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80009700 con 0x7f8a8c101740 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.311+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8c19f8e0 con 0x7f8a8c101740 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.311+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a8000cd80 con 0x7f8a8c101740 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.311+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80010430 con 0x7f8a8c101740 2024-03-24T06:40:03.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.311+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8c19fe10 con 0x7f8a8c101740 2024-03-24T06:40:03.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.312+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a8c108a60 con 0x7f8a8c101740 2024-03-24T06:40:03.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.314+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 24) ==== 136876+0+0 (secure 0 0 0) 0x7f8a80003680 con 0x7f8a8c101740 2024-03-24T06:40:03.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.315+0000 7f8a8a000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 0x7f8a740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.315+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8a80014070 con 0x7f8a8c101740 2024-03-24T06:40:03.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.316+0000 7f8a90c00640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 0x7f8a740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.317+0000 7f8a90c00640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 0x7f8a740a46f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a78000c00 tx=0x7f8a7802c800 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:03.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.320+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8a800970e0 con 0x7f8a8c101740 2024-03-24T06:40:03.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.528+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a8c10d170 con 0x7f8a8c101740 2024-03-24T06:40:03.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.529+0000 7f8a8a000640 1 -- 172.21.15.22:0/1419302433 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f8a8c10d170 con 0x7f8a8c101740 2024-03-24T06:40:03.529 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 msgr2=0x7f8a740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 0x7f8a740a46f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a78000c00 tx=0x7f8a7802c800 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 msgr2=0x7f8a8c1977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000e9c0 tx=0x7f8a8000ee80 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 shutdown_connections 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8a740a2230 0x7f8a740a46f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a8c1032f0 0x7f8a8c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a8c102930 0x7f8a8c197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 --2- 172.21.15.22:0/1419302433 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8a8c101740 0x7f8a8c1977e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 >> 172.21.15.22:0/1419302433 conn(0x7f8a8c0fccb0 msgr2=0x7f8a8c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 shutdown_connections 2024-03-24T06:40:03.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.532+0000 7f8a94000640 1 -- 172.21.15.22:0/1419302433 wait complete. 2024-03-24T06:40:03.552 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:40:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2736451135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-24T06:40:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1544400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:03 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 -- 172.21.15.22:0/2594899122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc102910 msgr2=0x7f4ecc102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 --2- 172.21.15.22:0/2594899122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc102910 0x7f4ecc102d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009900 tx=0x7f4eb802bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 -- 172.21.15.22:0/2594899122 shutdown_connections 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 --2- 172.21.15.22:0/2594899122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 --2- 172.21.15.22:0/2594899122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc102910 0x7f4ecc102d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 --2- 172.21.15.22:0/2594899122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc101720 0x7f4ecc101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 -- 172.21.15.22:0/2594899122 >> 172.21.15.22:0/2594899122 conn(0x7f4ecc0fccb0 msgr2=0x7f4ecc0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 -- 172.21.15.22:0/2594899122 shutdown_connections 2024-03-24T06:40:03.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.807+0000 7f4ed4400640 1 -- 172.21.15.22:0/2594899122 wait complete. 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 Processor -- start 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 -- start start 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 0x7f4ecc1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc102910 0x7f4ecc197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc19f2e0 con 0x7f4ecc101720 2024-03-24T06:40:03.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc19f450 con 0x7f4ecc102910 2024-03-24T06:40:03.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.808+0000 7f4ed4400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4ecc19f610 con 0x7f4ecc1032d0 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 0x7f4ecc1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 0x7f4ecc1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58184/0 (socket says 172.21.15.22:58184) 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed1a00640 1 -- 172.21.15.22:0/3931527791 learned_addr learned my addr 172.21.15.22:0/3931527791 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed1000640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc102910 0x7f4ecc197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 -- 172.21.15.22:0/3931527791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc102910 msgr2=0x7f4ecc197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc102910 0x7f4ecc197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 -- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 msgr2=0x7f4ecc1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:03.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 0x7f4ecc1977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.809+0000 7f4ed2400640 1 -- 172.21.15.22:0/3931527791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec0009840 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4ed2400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc19eda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e970 tx=0x7f4ec400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4009620 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec400ccc0 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4004a00 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb8009560 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.810+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ecc073cf0 con 0x7f4ecc1032d0 2024-03-24T06:40:03.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.812+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ecc108a40 con 0x7f4ecc1032d0 2024-03-24T06:40:03.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.813+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) ==== 136876+0+0 (secure 0 0 0) 0x7f4ec4005020 con 0x7f4ecc1032d0 2024-03-24T06:40:03.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.815+0000 7f4eca000640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 0x7f4eb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:03.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.815+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4ec4014070 con 0x7f4ecc1032d0 2024-03-24T06:40:03.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.815+0000 7f4ed1a00640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 0x7f4eb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:03.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.816+0000 7f4ed1a00640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 0x7f4eb40a46f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0005c40 tx=0x7f4ec0009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:03.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:03.818+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ec4096c70 con 0x7f4ecc1032d0 2024-03-24T06:40:03.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2736451135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-24T06:40:03.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1544400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:03 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2736451135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1544400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:03 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.033+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4ecc02d130 con 0x7f4ecc1032d0 2024-03-24T06:40:04.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.034+0000 7f4eca000640 1 -- 172.21.15.22:0/3931527791 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f4ec409a330 con 0x7f4ecc1032d0 2024-03-24T06:40:04.034 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 msgr2=0x7f4eb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 0x7f4eb40a46f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0005c40 tx=0x7f4ec0009500 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 msgr2=0x7f4ecc19eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc19eda0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e970 tx=0x7f4ec400ee30 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 shutdown_connections 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4eb40a2230 0x7f4eb40a46f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ecc1032d0 0x7f4ecc19eda0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ecc102910 0x7f4ecc197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 --2- 172.21.15.22:0/3931527791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ecc101720 0x7f4ecc1977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 >> 172.21.15.22:0/3931527791 conn(0x7f4ecc0fccb0 msgr2=0x7f4ecc105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.036+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 shutdown_connections 2024-03-24T06:40:04.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.037+0000 7f4ed4400640 1 -- 172.21.15.22:0/3931527791 wait complete. 2024-03-24T06:40:04.055 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-24T06:40:04.056 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 -- 172.21.15.22:0/1468257210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff184074110 msgr2=0x7ff184074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 --2- 172.21.15.22:0/1468257210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff184074110 0x7ff184074570 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff170009900 tx=0x7ff17002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 -- 172.21.15.22:0/1468257210 shutdown_connections 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 --2- 172.21.15.22:0/1468257210 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff18410ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 --2- 172.21.15.22:0/1468257210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff184074110 0x7ff184074570 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 --2- 172.21.15.22:0/1468257210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1840756f0 0x7ff184073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 -- 172.21.15.22:0/1468257210 >> 172.21.15.22:0/1468257210 conn(0x7ff1840fccc0 msgr2=0x7ff1840ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:04.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 -- 172.21.15.22:0/1468257210 shutdown_connections 2024-03-24T06:40:04.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.296+0000 7ff18a000640 1 -- 172.21.15.22:0/1468257210 wait complete. 2024-03-24T06:40:04.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 Processor -- start 2024-03-24T06:40:04.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 -- start start 2024-03-24T06:40:04.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 0x7ff184199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff1841a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841a14f0 con 0x7ff1841095e0 2024-03-24T06:40:04.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841a1660 con 0x7ff1840756f0 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.298+0000 7ff18a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff1841a1820 con 0x7ff184074110 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 0x7ff184199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54286/0 (socket says 172.21.15.22:54286) 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183400640 1 -- 172.21.15.22:0/3978919001 learned_addr learned my addr 172.21.15.22:0/3978919001 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:04.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183e00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff1841a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 -- 172.21.15.22:0/3978919001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 msgr2=0x7ff184199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 -- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 msgr2=0x7ff1841a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff1841a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff182a00640 1 -- 172.21.15.22:0/3978919001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff170009560 con 0x7ff1840756f0 2024-03-24T06:40:04.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183400640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.299+0000 7ff183e00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff1841a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff182a00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 0x7ff184199f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff17002c210 tx=0x7ff170005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17002dc20 con 0x7ff1840756f0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff17002dd80 con 0x7ff1840756f0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1841a1aa0 con 0x7ff1840756f0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170003bc0 con 0x7ff1840756f0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.300+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1841a1fd0 con 0x7ff1840756f0 2024-03-24T06:40:04.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.301+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1841083a0 con 0x7ff1840756f0 2024-03-24T06:40:04.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.304+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 24) ==== 136876+0+0 (secure 0 0 0) 0x7ff170003d20 con 0x7ff1840756f0 2024-03-24T06:40:04.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.306+0000 7ff17be00640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 0x7ff1680a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.306+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff1700f0920 con 0x7ff1840756f0 2024-03-24T06:40:04.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.306+0000 7ff183400640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 0x7ff1680a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.307+0000 7ff183400640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 0x7ff1680a4770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff174011550 tx=0x7ff174005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:04.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.313+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1700b9390 con 0x7ff1840756f0 2024-03-24T06:40:04.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.487+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7ff1700b9760 con 0x7ff1840756f0 2024-03-24T06:40:04.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.613+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff18410f380 con 0x7ff1840756f0 2024-03-24T06:40:04.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.614+0000 7ff17be00640 1 -- 172.21.15.22:0/3978919001 <== mon.2 v2:172.21.15.143:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11939 (secure 0 0 0) 0x7ff1700de020 con 0x7ff1840756f0 2024-03-24T06:40:04.614 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:04.614 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:39:58.997320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:40:04.614 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 29 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 msgr2=0x7ff1680a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 0x7ff1680a4770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff174011550 tx=0x7ff174005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 msgr2=0x7ff184199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 0x7ff184199f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff17002c210 tx=0x7ff170005d70 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 shutdown_connections 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff1680a22b0 0x7ff1680a4770 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff1841095e0 0x7ff1841a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1840756f0 0x7ff184199f40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 --2- 172.21.15.22:0/3978919001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184074110 0x7ff184199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 >> 172.21.15.22:0/3978919001 conn(0x7ff1840fccc0 msgr2=0x7ff184105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:04.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 shutdown_connections 2024-03-24T06:40:04.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.618+0000 7ff18a000640 1 -- 172.21.15.22:0/3978919001 wait complete. 2024-03-24T06:40:04.639 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-24T06:40:04.639 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-24T06:40:04.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1419302433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3931527791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.711 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:04 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1419302433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3931527791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:04 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 -- 172.21.15.22:0/256421557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4103950 msgr2=0x7f92a4103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 --2- 172.21.15.22:0/256421557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4103950 0x7f92a4103d70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9298009900 tx=0x7f929802bc40 comp rx=0 tx=0).stop 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 -- 172.21.15.22:0/256421557 shutdown_connections 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 --2- 172.21.15.22:0/256421557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4105500 0x7f92a410bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 --2- 172.21.15.22:0/256421557 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4104b40 0x7f92a4104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 --2- 172.21.15.22:0/256421557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4103950 0x7f92a4103d70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 -- 172.21.15.22:0/256421557 >> 172.21.15.22:0/256421557 conn(0x7f92a40feec0 msgr2=0x7f92a4101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 -- 172.21.15.22:0/256421557 shutdown_connections 2024-03-24T06:40:04.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.852+0000 7f92ac200640 1 -- 172.21.15.22:0/256421557 wait complete. 2024-03-24T06:40:04.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.853+0000 7f92ac200640 1 Processor -- start 2024-03-24T06:40:04.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 -- start start 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 0x7f92a41999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4105500 0x7f92a41a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f92a41a14c0 con 0x7f92a4103950 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92a41a1630 con 0x7f92a4105500 2024-03-24T06:40:04.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92ac200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f92a41a17f0 con 0x7f92a4104b40 2024-03-24T06:40:04.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92a9800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 0x7f92a41999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92a8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92a8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:54306/0 (socket says 172.21.15.22:54306) 2024-03-24T06:40:04.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92a8e00640 1 -- 172.21.15.22:0/1419712502 learned_addr learned my addr 172.21.15.22:0/1419712502 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:04.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92a9800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 0x7f92a41999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58212/0 (socket says 172.21.15.22:58212) 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.854+0000 7f92aa200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4105500 0x7f92a41a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 -- 172.21.15.22:0/1419712502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4105500 msgr2=0x7f92a41a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4105500 0x7f92a41a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 -- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 msgr2=0x7f92a41999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 0x7f92a41999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 -- 172.21.15.22:0/1419712502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9298009560 con 0x7f92a4104b40 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.855+0000 7f92a8e00640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f929000d950 tx=0x7f929000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.856+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290009c30 con 0x7f92a4104b40 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.856+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9290009d90 con 0x7f92a4104b40 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.856+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a4075bb0 con 0x7f92a4104b40 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.856+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000f7e0 con 0x7f92a4104b40 2024-03-24T06:40:04.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.856+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a4076110 con 0x7f92a4104b40 2024-03-24T06:40:04.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.857+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a410ac70 con 0x7f92a4104b40 2024-03-24T06:40:04.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.859+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f9290010ad0 con 0x7f92a4104b40 2024-03-24T06:40:04.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.861+0000 7f92a2000640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 0x7f928c0a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:04.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.861+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f92900ce1a0 con 0x7f92a4104b40 2024-03-24T06:40:04.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.861+0000 7f92a9800640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 0x7f928c0a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:04.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.862+0000 7f92a9800640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 0x7f928c0a4800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f929802c250 tx=0x7f929802c7f0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:04.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:04.870+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f92900969b0 con 0x7f92a4104b40 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1419302433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3931527791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:04.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:04 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.248+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92a402d240 con 0x7f928c0a2340 2024-03-24T06:40:05.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.261+0000 7f92a2000640 1 -- 172.21.15.22:0/1419712502 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339455 (secure 0 0 0) 0x7f92a402d240 con 0x7f928c0a2340 2024-03-24T06:40:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:05.267 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 msgr2=0x7f928c0a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 0x7f928c0a4800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f929802c250 tx=0x7f929802c7f0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 msgr2=0x7f92a4199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f929000d950 tx=0x7f929000de10 comp rx=0 tx=0).stop 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 shutdown_connections 2024-03-24T06:40:05.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f928c0a2340 0x7f928c0a4800 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92a4105500 0x7f92a41a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92a4104b40 0x7f92a4199f10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 --2- 172.21.15.22:0/1419712502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92a4103950 0x7f92a41999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 >> 172.21.15.22:0/1419712502 conn(0x7f92a40feec0 msgr2=0x7f92a4107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 shutdown_connections 2024-03-24T06:40:05.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.267+0000 7f92ac200640 1 -- 172.21.15.22:0/1419712502 wait complete. 2024-03-24T06:40:05.290 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T06:40:04.142233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191251,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1735,"num_read_kb":1040,"num_write":547,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181380,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":933,"ondisk_log_size":933,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389820,"kb_used_data":15424,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552004,"statfs":{"total":1151940427776,"available":1151541252096,"internally_reserved":0,"allocated":15794176,"data_stored":8969942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.977269"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807561+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:40:00.807561+0000","last_peered":"2024-03-24T06:40:00.807561+0000","last_clean":"2024-03-24T06:40:00.807561+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:40:00.807561+0000","last_undegraded":"2024-03-24T06:40:00.807561+0000","last_fullsized":"2024-03-24T06:40:00.807561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808126+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:40:00.808126+0000","last_peered":"2024-03-24T06:40:00.808126+0000","last_clean":"2024-03-24T06:40:00.808126+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:40:00.808126+0000","last_undegraded":"2024-03-24T06:40:00.808126+0000","last_fullsized":"2024-03-24T06:40:00.808126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807855+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:40:00.807855+0000","last_peered":"2024-03-24T06:40:00.807855+0000","last_clean":"2024-03-24T06:40:00.807855+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:40:00.807855+0000","last_undegraded":"2024-03-24T06:40:00.807855+0000","last_fullsized":"2024-03-24T06:40:00.807855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697661+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:40:00.697661+0000","last_peered":"2024-03-24T06:40:00.697661+0000","last_clean":"2024-03-24T06:40:00.697661+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:40:00.697661+0000","last_undegraded":"2024-03-24T06:40:00.697661+0000","last_fullsized":"2024-03-24T06:40:00.697661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281757+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:40:00.281757+0000","last_peered":"2024-03-24T06:40:00.281757+0000","last_clean":"2024-03-24T06:40:00.281757+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:40:00.281757+0000","last_undegraded":"2024-03-24T06:40:00.281757+0000","last_fullsized":"2024-03-24T06:40:00.281757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123976+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:40:00.123976+0000","last_peered":"2024-03-24T06:40:00.123976+0000","last_clean":"2024-03-24T06:40:00.123976+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:40:00.123976+0000","last_undegraded":"2024-03-24T06:40:00.123976+0000","last_fullsized":"2024-03-24T06:40:00.123976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652893+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:40:00.652893+0000","last_peered":"2024-03-24T06:40:00.652893+0000","last_clean":"2024-03-24T06:40:00.652893+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:40:00.652893+0000","last_undegraded":"2024-03-24T06:40:00.652893+0000","last_fullsized":"2024-03-24T06:40:00.652893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123459+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:40:00.123459+0000","last_peered":"2024-03-24T06:40:00.123459+0000","last_clean":"2024-03-24T06:40:00.123459+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:40:00.123459+0000","last_undegraded":"2024-03-24T06:40:00.123459+0000","last_fullsized":"2024-03-24T06:40:00.123459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688245+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:40:00.688245+0000","last_peered":"2024-03-24T06:40:00.688245+0000","last_clean":"2024-03-24T06:40:00.688245+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:40:00.688245+0000","last_undegraded":"2024-03-24T06:40:00.688245+0000","last_fullsized":"2024-03-24T06:40:00.688245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124396+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:40:00.124396+0000","last_peered":"2024-03-24T06:40:00.124396+0000","last_clean":"2024-03-24T06:40:00.124396+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:40:00.124396+0000","last_undegraded":"2024-03-24T06:40:00.124396+0000","last_fullsized":"2024-03-24T06:40:00.124396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653258+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:40:00.653258+0000","last_peered":"2024-03-24T06:40:00.653258+0000","last_clean":"2024-03-24T06:40:00.653258+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:40:00.653258+0000","last_undegraded":"2024-03-24T06:40:00.653258+0000","last_fullsized":"2024-03-24T06:40:00.653258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697898+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:40:00.697898+0000","last_peered":"2024-03-24T06:40:00.697898+0000","last_clean":"2024-03-24T06:40:00.697898+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:40:00.697898+0000","last_undegraded":"2024-03-24T06:40:00.697898+0000","last_fullsized":"2024-03-24T06:40:00.697898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680331+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:40:00.680331+0000","last_peered":"2024-03-24T06:40:00.680331+0000","last_clean":"2024-03-24T06:40:00.680331+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:40:00.680331+0000","last_undegraded":"2024-03-24T06:40:00.680331+0000","last_fullsized":"2024-03-24T06:40:00.680331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807397+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:40:00.807397+0000","last_peered":"2024-03-24T06:40:00.807397+0000","last_clean":"2024-03-24T06:40:00.807397+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:40:00.807397+0000","last_undegraded":"2024-03-24T06:40:00.807397+0000","last_fullsized":"2024-03-24T06:40:00.807397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807500+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:40:00.807500+0000","last_peered":"2024-03-24T06:40:00.807500+0000","last_clean":"2024-03-24T06:40:00.807500+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:40:00.807500+0000","last_undegraded":"2024-03-24T06:40:00.807500+0000","last_fullsized":"2024-03-24T06:40:00.807500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687644+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:40:00.687644+0000","last_peered":"2024-03-24T06:40:00.687644+0000","last_clean":"2024-03-24T06:40:00.687644+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:40:00.687644+0000","last_undegraded":"2024-03-24T06:40:00.687644+0000","last_fullsized":"2024-03-24T06:40:00.687644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807536+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:40:00.807536+0000","last_peered":"2024-03-24T06:40:00.807536+0000","last_clean":"2024-03-24T06:40:00.807536+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:40:00.807536+0000","last_undegraded":"2024-03-24T06:40:00.807536+0000","last_fullsized":"2024-03-24T06:40:00.807536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.278167+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:40:00.278167+0000","last_peered":"2024-03-24T06:40:00.278167+0000","last_clean":"2024-03-24T06:40:00.278167+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:40:00.278167+0000","last_undegraded":"2024-03-24T06:40:00.278167+0000","last_fullsized":"2024-03-24T06:40:00.278167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124626+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:40:00.124626+0000","last_peered":"2024-03-24T06:40:00.124626+0000","last_clean":"2024-03-24T06:40:00.124626+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:40:00.124626+0000","last_undegraded":"2024-03-24T06:40:00.124626+0000","last_fullsized":"2024-03-24T06:40:00.124626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123859+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:40:00.123859+0000","last_peered":"2024-03-24T06:40:00.123859+0000","last_clean":"2024-03-24T06:40:00.123859+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:40:00.123859+0000","last_undegraded":"2024-03-24T06:40:00.123859+0000","last_fullsized":"2024-03-24T06:40:00.123859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":197,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.651139+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:40:00.651139+0000","last_peered":"2024-03-24T06:40:00.651139+0000","last_clean":"2024-03-24T06:40:00.651139+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:40:00.651139+0000","last_undegraded":"2024-03-24T06:40:00.651139+0000","last_fullsized":"2024-03-24T06:40:00.651139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124742+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:40:00.124742+0000","last_peered":"2024-03-24T06:40:00.124742+0000","last_clean":"2024-03-24T06:40:00.124742+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:40:00.124742+0000","last_undegraded":"2024-03-24T06:40:00.124742+0000","last_fullsized":"2024-03-24T06:40:00.124742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697277+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:40:00.697277+0000","last_peered":"2024-03-24T06:40:00.697277+0000","last_clean":"2024-03-24T06:40:00.697277+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:40:00.697277+0000","last_undegraded":"2024-03-24T06:40:00.697277+0000","last_fullsized":"2024-03-24T06:40:00.697277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125229+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:40:00.125229+0000","last_peered":"2024-03-24T06:40:00.125229+0000","last_clean":"2024-03-24T06:40:00.125229+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:40:00.125229+0000","last_undegraded":"2024-03-24T06:40:00.125229+0000","last_fullsized":"2024-03-24T06:40:00.125229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808502+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:40:00.808502+0000","last_peered":"2024-03-24T06:40:00.808502+0000","last_clean":"2024-03-24T06:40:00.808502+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:40:00.808502+0000","last_undegraded":"2024-03-24T06:40:00.808502+0000","last_fullsized":"2024-03-24T06:40:00.808502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697506+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:40:00.697506+0000","last_peered":"2024-03-24T06:40:00.697506+0000","last_clean":"2024-03-24T06:40:00.697506+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:40:00.697506+0000","last_undegraded":"2024-03-24T06:40:00.697506+0000","last_fullsized":"2024-03-24T06:40:00.697506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282154+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:40:00.282154+0000","last_peered":"2024-03-24T06:40:00.282154+0000","last_clean":"2024-03-24T06:40:00.282154+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:40:00.282154+0000","last_undegraded":"2024-03-24T06:40:00.282154+0000","last_fullsized":"2024-03-24T06:40:00.282154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687945+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:40:00.687945+0000","last_peered":"2024-03-24T06:40:00.687945+0000","last_clean":"2024-03-24T06:40:00.687945+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:40:00.687945+0000","last_undegraded":"2024-03-24T06:40:00.687945+0000","last_fullsized":"2024-03-24T06:40:00.687945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697300+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:40:00.697300+0000","last_peered":"2024-03-24T06:40:00.697300+0000","last_clean":"2024-03-24T06:40:00.697300+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:40:00.697300+0000","last_undegraded":"2024-03-24T06:40:00.697300+0000","last_fullsized":"2024-03-24T06:40:00.697300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281817+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:40:00.281817+0000","last_peered":"2024-03-24T06:40:00.281817+0000","last_clean":"2024-03-24T06:40:00.281817+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:40:00.281817+0000","last_undegraded":"2024-03-24T06:40:00.281817+0000","last_fullsized":"2024-03-24T06:40:00.281817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291734+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:40:00.291734+0000","last_peered":"2024-03-24T06:40:00.291734+0000","last_clean":"2024-03-24T06:40:00.291734+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:40:00.291734+0000","last_undegraded":"2024-03-24T06:40:00.291734+0000","last_fullsized":"2024-03-24T06:40:00.291734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124183+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:40:00.124183+0000","last_peered":"2024-03-24T06:40:00.124183+0000","last_clean":"2024-03-24T06:40:00.124183+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:40:00.124183+0000","last_undegraded":"2024-03-24T06:40:00.124183+0000","last_fullsized":"2024-03-24T06:40:00.124183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959636+0000","last_change":"2024-03-24T06:37:56.839629+0000","last_active":"2024-03-24T06:40:00.959636+0000","last_peered":"2024-03-24T06:40:00.959636+0000","last_clean":"2024-03-24T06:40:00.959636+0000","last_became_active":"2024-03-24T06:37:56.832648+0000","last_became_peered":"2024-03-24T06:37:56.832648+0000","last_unstale":"2024-03-24T06:40:00.959636+0000","last_undegraded":"2024-03-24T06:40:00.959636+0000","last_fullsized":"2024-03-24T06:40:00.959636+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:57.270140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281637+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:40:00.281637+0000","last_peered":"2024-03-24T06:40:00.281637+0000","last_clean":"2024-03-24T06:40:00.281637+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:40:00.281637+0000","last_undegraded":"2024-03-24T06:40:00.281637+0000","last_fullsized":"2024-03-24T06:40:00.281637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124228+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:40:00.124228+0000","last_peered":"2024-03-24T06:40:00.124228+0000","last_clean":"2024-03-24T06:40:00.124228+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:40:00.124228+0000","last_undegraded":"2024-03-24T06:40:00.124228+0000","last_fullsized":"2024-03-24T06:40:00.124228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123925+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:40:00.123925+0000","last_peered":"2024-03-24T06:40:00.123925+0000","last_clean":"2024-03-24T06:40:00.123925+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:40:00.123925+0000","last_undegraded":"2024-03-24T06:40:00.123925+0000","last_fullsized":"2024-03-24T06:40:00.123925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291895+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:40:00.291895+0000","last_peered":"2024-03-24T06:40:00.291895+0000","last_clean":"2024-03-24T06:40:00.291895+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:40:00.291895+0000","last_undegraded":"2024-03-24T06:40:00.291895+0000","last_fullsized":"2024-03-24T06:40:00.291895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807182+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:40:00.807182+0000","last_peered":"2024-03-24T06:40:00.807182+0000","last_clean":"2024-03-24T06:40:00.807182+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:40:00.807182+0000","last_undegraded":"2024-03-24T06:40:00.807182+0000","last_fullsized":"2024-03-24T06:40:00.807182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679807+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:40:00.679807+0000","last_peered":"2024-03-24T06:40:00.679807+0000","last_clean":"2024-03-24T06:40:00.679807+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:40:00.679807+0000","last_undegraded":"2024-03-24T06:40:00.679807+0000","last_fullsized":"2024-03-24T06:40:00.679807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123726+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:40:00.123726+0000","last_peered":"2024-03-24T06:40:00.123726+0000","last_clean":"2024-03-24T06:40:00.123726+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:40:00.123726+0000","last_undegraded":"2024-03-24T06:40:00.123726+0000","last_fullsized":"2024-03-24T06:40:00.123726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808088+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:40:00.808088+0000","last_peered":"2024-03-24T06:40:00.808088+0000","last_clean":"2024-03-24T06:40:00.808088+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:40:00.808088+0000","last_undegraded":"2024-03-24T06:40:00.808088+0000","last_fullsized":"2024-03-24T06:40:00.808088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291185+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:40:00.291185+0000","last_peered":"2024-03-24T06:40:00.291185+0000","last_clean":"2024-03-24T06:40:00.291185+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:40:00.291185+0000","last_undegraded":"2024-03-24T06:40:00.291185+0000","last_fullsized":"2024-03-24T06:40:00.291185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277299+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:40:00.277299+0000","last_peered":"2024-03-24T06:40:00.277299+0000","last_clean":"2024-03-24T06:40:00.277299+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:40:00.277299+0000","last_undegraded":"2024-03-24T06:40:00.277299+0000","last_fullsized":"2024-03-24T06:40:00.277299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123735+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:40:00.123735+0000","last_peered":"2024-03-24T06:40:00.123735+0000","last_clean":"2024-03-24T06:40:00.123735+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:40:00.123735+0000","last_undegraded":"2024-03-24T06:40:00.123735+0000","last_fullsized":"2024-03-24T06:40:00.123735+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":334,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.590647+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:40:00.590647+0000","last_peered":"2024-03-24T06:40:00.590647+0000","last_clean":"2024-03-24T06:40:00.590647+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:40:00.590647+0000","last_undegraded":"2024-03-24T06:40:00.590647+0000","last_fullsized":"2024-03-24T06:40:00.590647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":296,"num_read_kb":296,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688179+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:40:00.688179+0000","last_peered":"2024-03-24T06:40:00.688179+0000","last_clean":"2024-03-24T06:40:00.688179+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:40:00.688179+0000","last_undegraded":"2024-03-24T06:40:00.688179+0000","last_fullsized":"2024-03-24T06:40:00.688179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281341+0000","last_change":"2024-03-24T06:37:56.831960+0000","last_active":"2024-03-24T06:40:00.281341+0000","last_peered":"2024-03-24T06:40:00.281341+0000","last_clean":"2024-03-24T06:40:00.281341+0000","last_became_active":"2024-03-24T06:37:56.831524+0000","last_became_peered":"2024-03-24T06:37:56.831524+0000","last_unstale":"2024-03-24T06:40:00.281341+0000","last_undegraded":"2024-03-24T06:40:00.281341+0000","last_fullsized":"2024-03-24T06:40:00.281341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:28.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653238+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:40:00.653238+0000","last_peered":"2024-03-24T06:40:00.653238+0000","last_clean":"2024-03-24T06:40:00.653238+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:40:00.653238+0000","last_undegraded":"2024-03-24T06:40:00.653238+0000","last_fullsized":"2024-03-24T06:40:00.653238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808357+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:40:00.808357+0000","last_peered":"2024-03-24T06:40:00.808357+0000","last_clean":"2024-03-24T06:40:00.808357+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:40:00.808357+0000","last_undegraded":"2024-03-24T06:40:00.808357+0000","last_fullsized":"2024-03-24T06:40:00.808357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697632+0000","last_change":"2024-03-24T06:37:57.056640+0000","last_active":"2024-03-24T06:40:00.697632+0000","last_peered":"2024-03-24T06:40:00.697632+0000","last_clean":"2024-03-24T06:40:00.697632+0000","last_became_active":"2024-03-24T06:37:57.056087+0000","last_became_peered":"2024-03-24T06:37:57.056087+0000","last_unstale":"2024-03-24T06:40:00.697632+0000","last_undegraded":"2024-03-24T06:40:00.697632+0000","last_fullsized":"2024-03-24T06:40:00.697632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:46.933123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291609+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:40:00.291609+0000","last_peered":"2024-03-24T06:40:00.291609+0000","last_clean":"2024-03-24T06:40:00.291609+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:40:00.291609+0000","last_undegraded":"2024-03-24T06:40:00.291609+0000","last_fullsized":"2024-03-24T06:40:00.291609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688042+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:40:00.688042+0000","last_peered":"2024-03-24T06:40:00.688042+0000","last_clean":"2024-03-24T06:40:00.688042+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:40:00.688042+0000","last_undegraded":"2024-03-24T06:40:00.688042+0000","last_fullsized":"2024-03-24T06:40:00.688042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277348+0000","last_change":"2024-03-24T06:38:58.624331+0000","last_active":"2024-03-24T06:40:00.277348+0000","last_peered":"2024-03-24T06:40:00.277348+0000","last_clean":"2024-03-24T06:40:00.277348+0000","last_became_active":"2024-03-24T06:38:58.623562+0000","last_became_peered":"2024-03-24T06:38:58.623562+0000","last_unstale":"2024-03-24T06:40:00.277348+0000","last_undegraded":"2024-03-24T06:40:00.277348+0000","last_fullsized":"2024-03-24T06:40:00.277348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:19:32.355669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":377,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959384+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:40:00.959384+0000","last_peered":"2024-03-24T06:40:00.959384+0000","last_clean":"2024-03-24T06:40:00.959384+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:40:00.959384+0000","last_undegraded":"2024-03-24T06:40:00.959384+0000","last_fullsized":"2024-03-24T06:40:00.959384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67932,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":304,"num_read_kb":219,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281881+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:40:00.281881+0000","last_peered":"2024-03-24T06:40:00.281881+0000","last_clean":"2024-03-24T06:40:00.281881+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:40:00.281881+0000","last_undegraded":"2024-03-24T06:40:00.281881+0000","last_fullsized":"2024-03-24T06:40:00.281881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291994+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:40:00.291994+0000","last_peered":"2024-03-24T06:40:00.291994+0000","last_clean":"2024-03-24T06:40:00.291994+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:40:00.291994+0000","last_undegraded":"2024-03-24T06:40:00.291994+0000","last_fullsized":"2024-03-24T06:40:00.291994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124117+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:40:00.124117+0000","last_peered":"2024-03-24T06:40:00.124117+0000","last_clean":"2024-03-24T06:40:00.124117+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:40:00.124117+0000","last_undegraded":"2024-03-24T06:40:00.124117+0000","last_fullsized":"2024-03-24T06:40:00.124117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679797+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:40:00.679797+0000","last_peered":"2024-03-24T06:40:00.679797+0000","last_clean":"2024-03-24T06:40:00.679797+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:40:00.679797+0000","last_undegraded":"2024-03-24T06:40:00.679797+0000","last_fullsized":"2024-03-24T06:40:00.679797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807095+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:40:00.807095+0000","last_peered":"2024-03-24T06:40:00.807095+0000","last_clean":"2024-03-24T06:40:00.807095+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:40:00.807095+0000","last_undegraded":"2024-03-24T06:40:00.807095+0000","last_fullsized":"2024-03-24T06:40:00.807095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291586+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:40:00.291586+0000","last_peered":"2024-03-24T06:40:00.291586+0000","last_clean":"2024-03-24T06:40:00.291586+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:40:00.291586+0000","last_undegraded":"2024-03-24T06:40:00.291586+0000","last_fullsized":"2024-03-24T06:40:00.291586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653539+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:40:00.653539+0000","last_peered":"2024-03-24T06:40:00.653539+0000","last_clean":"2024-03-24T06:40:00.653539+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:40:00.653539+0000","last_undegraded":"2024-03-24T06:40:00.653539+0000","last_fullsized":"2024-03-24T06:40:00.653539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279380+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:40:00.279380+0000","last_peered":"2024-03-24T06:40:00.279380+0000","last_clean":"2024-03-24T06:40:00.279380+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:40:00.279380+0000","last_undegraded":"2024-03-24T06:40:00.279380+0000","last_fullsized":"2024-03-24T06:40:00.279380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697530+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:40:00.697530+0000","last_peered":"2024-03-24T06:40:00.697530+0000","last_clean":"2024-03-24T06:40:00.697530+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:40:00.697530+0000","last_undegraded":"2024-03-24T06:40:00.697530+0000","last_fullsized":"2024-03-24T06:40:00.697530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":151,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652957+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:40:00.652957+0000","last_peered":"2024-03-24T06:40:00.652957+0000","last_clean":"2024-03-24T06:40:00.652957+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:40:00.652957+0000","last_undegraded":"2024-03-24T06:40:00.652957+0000","last_fullsized":"2024-03-24T06:40:00.652957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681557+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:40:00.681557+0000","last_peered":"2024-03-24T06:40:00.681557+0000","last_clean":"2024-03-24T06:40:00.681557+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:40:00.681557+0000","last_undegraded":"2024-03-24T06:40:00.681557+0000","last_fullsized":"2024-03-24T06:40:00.681557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281346+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:40:00.281346+0000","last_peered":"2024-03-24T06:40:00.281346+0000","last_clean":"2024-03-24T06:40:00.281346+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:40:00.281346+0000","last_undegraded":"2024-03-24T06:40:00.281346+0000","last_fullsized":"2024-03-24T06:40:00.281346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291241+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:40:00.291241+0000","last_peered":"2024-03-24T06:40:00.291241+0000","last_clean":"2024-03-24T06:40:00.291241+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:40:00.291241+0000","last_undegraded":"2024-03-24T06:40:00.291241+0000","last_fullsized":"2024-03-24T06:40:00.291241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291201+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:40:00.291201+0000","last_peered":"2024-03-24T06:40:00.291201+0000","last_clean":"2024-03-24T06:40:00.291201+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:40:00.291201+0000","last_undegraded":"2024-03-24T06:40:00.291201+0000","last_fullsized":"2024-03-24T06:40:00.291201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807189+0000","last_change":"2024-03-24T06:37:56.837415+0000","last_active":"2024-03-24T06:40:00.807189+0000","last_peered":"2024-03-24T06:40:00.807189+0000","last_clean":"2024-03-24T06:40:00.807189+0000","last_became_active":"2024-03-24T06:37:56.832240+0000","last_became_peered":"2024-03-24T06:37:56.832240+0000","last_unstale":"2024-03-24T06:40:00.807189+0000","last_undegraded":"2024-03-24T06:40:00.807189+0000","last_fullsized":"2024-03-24T06:40:00.807189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:44.867297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807274+0000","last_change":"2024-03-24T06:37:56.826934+0000","last_active":"2024-03-24T06:40:00.807274+0000","last_peered":"2024-03-24T06:40:00.807274+0000","last_clean":"2024-03-24T06:40:00.807274+0000","last_became_active":"2024-03-24T06:37:56.826574+0000","last_became_peered":"2024-03-24T06:37:56.826574+0000","last_unstale":"2024-03-24T06:40:00.807274+0000","last_undegraded":"2024-03-24T06:40:00.807274+0000","last_fullsized":"2024-03-24T06:40:00.807274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:52.406936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281632+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:40:00.281632+0000","last_peered":"2024-03-24T06:40:00.281632+0000","last_clean":"2024-03-24T06:40:00.281632+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:40:00.281632+0000","last_undegraded":"2024-03-24T06:40:00.281632+0000","last_fullsized":"2024-03-24T06:40:00.281632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277730+0000","last_change":"2024-03-24T06:37:56.832165+0000","last_active":"2024-03-24T06:40:00.277730+0000","last_peered":"2024-03-24T06:40:00.277730+0000","last_clean":"2024-03-24T06:40:00.277730+0000","last_became_active":"2024-03-24T06:37:56.831670+0000","last_became_peered":"2024-03-24T06:37:56.831670+0000","last_unstale":"2024-03-24T06:40:00.277730+0000","last_undegraded":"2024-03-24T06:40:00.277730+0000","last_fullsized":"2024-03-24T06:40:00.277730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.201188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124107+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:40:00.124107+0000","last_peered":"2024-03-24T06:40:00.124107+0000","last_clean":"2024-03-24T06:40:00.124107+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:40:00.124107+0000","last_undegraded":"2024-03-24T06:40:00.124107+0000","last_fullsized":"2024-03-24T06:40:00.124107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680650+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:40:00.680650+0000","last_peered":"2024-03-24T06:40:00.680650+0000","last_clean":"2024-03-24T06:40:00.680650+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:40:00.680650+0000","last_undegraded":"2024-03-24T06:40:00.680650+0000","last_fullsized":"2024-03-24T06:40:00.680650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687830+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:40:00.687830+0000","last_peered":"2024-03-24T06:40:00.687830+0000","last_clean":"2024-03-24T06:40:00.687830+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:40:00.687830+0000","last_undegraded":"2024-03-24T06:40:00.687830+0000","last_fullsized":"2024-03-24T06:40:00.687830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123953+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:40:00.123953+0000","last_peered":"2024-03-24T06:40:00.123953+0000","last_clean":"2024-03-24T06:40:00.123953+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:40:00.123953+0000","last_undegraded":"2024-03-24T06:40:00.123953+0000","last_fullsized":"2024-03-24T06:40:00.123953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681686+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:40:00.681686+0000","last_peered":"2024-03-24T06:40:00.681686+0000","last_clean":"2024-03-24T06:40:00.681686+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:40:00.681686+0000","last_undegraded":"2024-03-24T06:40:00.681686+0000","last_fullsized":"2024-03-24T06:40:00.681686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.292031+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:40:00.292031+0000","last_peered":"2024-03-24T06:40:00.292031+0000","last_clean":"2024-03-24T06:40:00.292031+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:40:00.292031+0000","last_undegraded":"2024-03-24T06:40:00.292031+0000","last_fullsized":"2024-03-24T06:40:00.292031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":187,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679720+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:40:00.679720+0000","last_peered":"2024-03-24T06:40:00.679720+0000","last_clean":"2024-03-24T06:40:00.679720+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:40:00.679720+0000","last_undegraded":"2024-03-24T06:40:00.679720+0000","last_fullsized":"2024-03-24T06:40:00.679720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653101+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:40:00.653101+0000","last_peered":"2024-03-24T06:40:00.653101+0000","last_clean":"2024-03-24T06:40:00.653101+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:40:00.653101+0000","last_undegraded":"2024-03-24T06:40:00.653101+0000","last_fullsized":"2024-03-24T06:40:00.653101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124031+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:40:00.124031+0000","last_peered":"2024-03-24T06:40:00.124031+0000","last_clean":"2024-03-24T06:40:00.124031+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:40:00.124031+0000","last_undegraded":"2024-03-24T06:40:00.124031+0000","last_fullsized":"2024-03-24T06:40:00.124031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697561+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:40:00.697561+0000","last_peered":"2024-03-24T06:40:00.697561+0000","last_clean":"2024-03-24T06:40:00.697561+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:40:00.697561+0000","last_undegraded":"2024-03-24T06:40:00.697561+0000","last_fullsized":"2024-03-24T06:40:00.697561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652830+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:40:00.652830+0000","last_peered":"2024-03-24T06:40:00.652830+0000","last_clean":"2024-03-24T06:40:00.652830+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:40:00.652830+0000","last_undegraded":"2024-03-24T06:40:00.652830+0000","last_fullsized":"2024-03-24T06:40:00.652830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291596+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:40:00.291596+0000","last_peered":"2024-03-24T06:40:00.291596+0000","last_clean":"2024-03-24T06:40:00.291596+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:40:00.291596+0000","last_undegraded":"2024-03-24T06:40:00.291596+0000","last_fullsized":"2024-03-24T06:40:00.291596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679511+0000","last_change":"2024-03-24T06:37:56.831122+0000","last_active":"2024-03-24T06:40:00.679511+0000","last_peered":"2024-03-24T06:40:00.679511+0000","last_clean":"2024-03-24T06:40:00.679511+0000","last_became_active":"2024-03-24T06:37:56.830642+0000","last_became_peered":"2024-03-24T06:37:56.830642+0000","last_unstale":"2024-03-24T06:40:00.679511+0000","last_undegraded":"2024-03-24T06:40:00.679511+0000","last_fullsized":"2024-03-24T06:40:00.679511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.204542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652901+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:40:00.652901+0000","last_peered":"2024-03-24T06:40:00.652901+0000","last_clean":"2024-03-24T06:40:00.652901+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:40:00.652901+0000","last_undegraded":"2024-03-24T06:40:00.652901+0000","last_fullsized":"2024-03-24T06:40:00.652901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806953+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:40:00.806953+0000","last_peered":"2024-03-24T06:40:00.806953+0000","last_clean":"2024-03-24T06:40:00.806953+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:40:00.806953+0000","last_undegraded":"2024-03-24T06:40:00.806953+0000","last_fullsized":"2024-03-24T06:40:00.806953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282064+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:40:00.282064+0000","last_peered":"2024-03-24T06:40:00.282064+0000","last_clean":"2024-03-24T06:40:00.282064+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:40:00.282064+0000","last_undegraded":"2024-03-24T06:40:00.282064+0000","last_fullsized":"2024-03-24T06:40:00.282064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680823+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:40:00.680823+0000","last_peered":"2024-03-24T06:40:00.680823+0000","last_clean":"2024-03-24T06:40:00.680823+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:40:00.680823+0000","last_undegraded":"2024-03-24T06:40:00.680823+0000","last_fullsized":"2024-03-24T06:40:00.680823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123597+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:40:00.123597+0000","last_peered":"2024-03-24T06:40:00.123597+0000","last_clean":"2024-03-24T06:40:00.123597+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:40:00.123597+0000","last_undegraded":"2024-03-24T06:40:00.123597+0000","last_fullsized":"2024-03-24T06:40:00.123597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.457728+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:40:00.457728+0000","last_peered":"2024-03-24T06:40:00.457728+0000","last_clean":"2024-03-24T06:40:00.457728+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:40:00.457728+0000","last_undegraded":"2024-03-24T06:40:00.457728+0000","last_fullsized":"2024-03-24T06:40:00.457728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":85,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124952+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:40:00.124952+0000","last_peered":"2024-03-24T06:40:00.124952+0000","last_clean":"2024-03-24T06:40:00.124952+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:40:00.124952+0000","last_undegraded":"2024-03-24T06:40:00.124952+0000","last_fullsized":"2024-03-24T06:40:00.124952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123851+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:40:00.123851+0000","last_peered":"2024-03-24T06:40:00.123851+0000","last_clean":"2024-03-24T06:40:00.123851+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:40:00.123851+0000","last_undegraded":"2024-03-24T06:40:00.123851+0000","last_fullsized":"2024-03-24T06:40:00.123851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697902+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:40:00.697902+0000","last_peered":"2024-03-24T06:40:00.697902+0000","last_clean":"2024-03-24T06:40:00.697902+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:40:00.697902+0000","last_undegraded":"2024-03-24T06:40:00.697902+0000","last_fullsized":"2024-03-24T06:40:00.697902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808204+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:40:00.808204+0000","last_peered":"2024-03-24T06:40:00.808204+0000","last_clean":"2024-03-24T06:40:00.808204+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:40:00.808204+0000","last_undegraded":"2024-03-24T06:40:00.808204+0000","last_fullsized":"2024-03-24T06:40:00.808204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697694+0000","last_change":"2024-03-24T06:37:57.056479+0000","last_active":"2024-03-24T06:40:00.697694+0000","last_peered":"2024-03-24T06:40:00.697694+0000","last_clean":"2024-03-24T06:40:00.697694+0000","last_became_active":"2024-03-24T06:37:57.055993+0000","last_became_peered":"2024-03-24T06:37:57.055993+0000","last_unstale":"2024-03-24T06:40:00.697694+0000","last_undegraded":"2024-03-24T06:40:00.697694+0000","last_fullsized":"2024-03-24T06:40:00.697694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:12.473494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687639+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:40:00.687639+0000","last_peered":"2024-03-24T06:40:00.687639+0000","last_clean":"2024-03-24T06:40:00.687639+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:40:00.687639+0000","last_undegraded":"2024-03-24T06:40:00.687639+0000","last_fullsized":"2024-03-24T06:40:00.687639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281408+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:40:00.281408+0000","last_peered":"2024-03-24T06:40:00.281408+0000","last_clean":"2024-03-24T06:40:00.281408+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:40:00.281408+0000","last_undegraded":"2024-03-24T06:40:00.281408+0000","last_fullsized":"2024-03-24T06:40:00.281408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279502+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:40:00.279502+0000","last_peered":"2024-03-24T06:40:00.279502+0000","last_clean":"2024-03-24T06:40:00.279502+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:40:00.279502+0000","last_undegraded":"2024-03-24T06:40:00.279502+0000","last_fullsized":"2024-03-24T06:40:00.279502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123941+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:40:00.123941+0000","last_peered":"2024-03-24T06:40:00.123941+0000","last_clean":"2024-03-24T06:40:00.123941+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:40:00.123941+0000","last_undegraded":"2024-03-24T06:40:00.123941+0000","last_fullsized":"2024-03-24T06:40:00.123941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806870+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:40:00.806870+0000","last_peered":"2024-03-24T06:40:00.806870+0000","last_clean":"2024-03-24T06:40:00.806870+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:40:00.806870+0000","last_undegraded":"2024-03-24T06:40:00.806870+0000","last_fullsized":"2024-03-24T06:40:00.806870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697505+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:40:00.697505+0000","last_peered":"2024-03-24T06:40:00.697505+0000","last_clean":"2024-03-24T06:40:00.697505+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:40:00.697505+0000","last_undegraded":"2024-03-24T06:40:00.697505+0000","last_fullsized":"2024-03-24T06:40:00.697505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123999+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:40:00.123999+0000","last_peered":"2024-03-24T06:40:00.123999+0000","last_clean":"2024-03-24T06:40:00.123999+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:40:00.123999+0000","last_undegraded":"2024-03-24T06:40:00.123999+0000","last_fullsized":"2024-03-24T06:40:00.123999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125506+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:40:00.125506+0000","last_peered":"2024-03-24T06:40:00.125506+0000","last_clean":"2024-03-24T06:40:00.125506+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:40:00.125506+0000","last_undegraded":"2024-03-24T06:40:00.125506+0000","last_fullsized":"2024-03-24T06:40:00.125506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697583+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:40:00.697583+0000","last_peered":"2024-03-24T06:40:00.697583+0000","last_clean":"2024-03-24T06:40:00.697583+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:40:00.697583+0000","last_undegraded":"2024-03-24T06:40:00.697583+0000","last_fullsized":"2024-03-24T06:40:00.697583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281619+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:40:00.281619+0000","last_peered":"2024-03-24T06:40:00.281619+0000","last_clean":"2024-03-24T06:40:00.281619+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:40:00.281619+0000","last_undegraded":"2024-03-24T06:40:00.281619+0000","last_fullsized":"2024-03-24T06:40:00.281619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653331+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:40:00.653331+0000","last_peered":"2024-03-24T06:40:00.653331+0000","last_clean":"2024-03-24T06:40:00.653331+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:40:00.653331+0000","last_undegraded":"2024-03-24T06:40:00.653331+0000","last_fullsized":"2024-03-24T06:40:00.653331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687653+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:40:00.687653+0000","last_peered":"2024-03-24T06:40:00.687653+0000","last_clean":"2024-03-24T06:40:00.687653+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:40:00.687653+0000","last_undegraded":"2024-03-24T06:40:00.687653+0000","last_fullsized":"2024-03-24T06:40:00.687653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124472+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:40:00.124472+0000","last_peered":"2024-03-24T06:40:00.124472+0000","last_clean":"2024-03-24T06:40:00.124472+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:40:00.124472+0000","last_undegraded":"2024-03-24T06:40:00.124472+0000","last_fullsized":"2024-03-24T06:40:00.124472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653414+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:40:00.653414+0000","last_peered":"2024-03-24T06:40:00.653414+0000","last_clean":"2024-03-24T06:40:00.653414+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:40:00.653414+0000","last_undegraded":"2024-03-24T06:40:00.653414+0000","last_fullsized":"2024-03-24T06:40:00.653414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807599+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:40:00.807599+0000","last_peered":"2024-03-24T06:40:00.807599+0000","last_clean":"2024-03-24T06:40:00.807599+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:40:00.807599+0000","last_undegraded":"2024-03-24T06:40:00.807599+0000","last_fullsized":"2024-03-24T06:40:00.807599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.454352+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:40:00.454352+0000","last_peered":"2024-03-24T06:40:00.454352+0000","last_clean":"2024-03-24T06:40:00.454352+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:40:00.454352+0000","last_undegraded":"2024-03-24T06:40:00.454352+0000","last_fullsized":"2024-03-24T06:40:00.454352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653114+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:40:00.653114+0000","last_peered":"2024-03-24T06:40:00.653114+0000","last_clean":"2024-03-24T06:40:00.653114+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:40:00.653114+0000","last_undegraded":"2024-03-24T06:40:00.653114+0000","last_fullsized":"2024-03-24T06:40:00.653114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.290992+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:40:00.290992+0000","last_peered":"2024-03-24T06:40:00.290992+0000","last_clean":"2024-03-24T06:40:00.290992+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:40:00.290992+0000","last_undegraded":"2024-03-24T06:40:00.290992+0000","last_fullsized":"2024-03-24T06:40:00.290992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681423+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:40:00.681423+0000","last_peered":"2024-03-24T06:40:00.681423+0000","last_clean":"2024-03-24T06:40:00.681423+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:40:00.681423+0000","last_undegraded":"2024-03-24T06:40:00.681423+0000","last_fullsized":"2024-03-24T06:40:00.681423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653402+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:40:00.653402+0000","last_peered":"2024-03-24T06:40:00.653402+0000","last_clean":"2024-03-24T06:40:00.653402+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:40:00.653402+0000","last_undegraded":"2024-03-24T06:40:00.653402+0000","last_fullsized":"2024-03-24T06:40:00.653402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806949+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:40:00.806949+0000","last_peered":"2024-03-24T06:40:00.806949+0000","last_clean":"2024-03-24T06:40:00.806949+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:40:00.806949+0000","last_undegraded":"2024-03-24T06:40:00.806949+0000","last_fullsized":"2024-03-24T06:40:00.806949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124150+0000","last_change":"2024-03-24T06:37:56.832171+0000","last_active":"2024-03-24T06:40:00.124150+0000","last_peered":"2024-03-24T06:40:00.124150+0000","last_clean":"2024-03-24T06:40:00.124150+0000","last_became_active":"2024-03-24T06:37:56.831734+0000","last_became_peered":"2024-03-24T06:37:56.831734+0000","last_unstale":"2024-03-24T06:40:00.124150+0000","last_undegraded":"2024-03-24T06:40:00.124150+0000","last_fullsized":"2024-03-24T06:40:00.124150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:19.788831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123937+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:40:00.123937+0000","last_peered":"2024-03-24T06:40:00.123937+0000","last_clean":"2024-03-24T06:40:00.123937+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:40:00.123937+0000","last_undegraded":"2024-03-24T06:40:00.123937+0000","last_fullsized":"2024-03-24T06:40:00.123937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807833+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:40:00.807833+0000","last_peered":"2024-03-24T06:40:00.807833+0000","last_clean":"2024-03-24T06:40:00.807833+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:40:00.807833+0000","last_undegraded":"2024-03-24T06:40:00.807833+0000","last_fullsized":"2024-03-24T06:40:00.807833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680980+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:40:00.680980+0000","last_peered":"2024-03-24T06:40:00.680980+0000","last_clean":"2024-03-24T06:40:00.680980+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:40:00.680980+0000","last_undegraded":"2024-03-24T06:40:00.680980+0000","last_fullsized":"2024-03-24T06:40:00.680980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807380+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:40:00.807380+0000","last_peered":"2024-03-24T06:40:00.807380+0000","last_clean":"2024-03-24T06:40:00.807380+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:40:00.807380+0000","last_undegraded":"2024-03-24T06:40:00.807380+0000","last_fullsized":"2024-03-24T06:40:00.807380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124693+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:40:00.124693+0000","last_peered":"2024-03-24T06:40:00.124693+0000","last_clean":"2024-03-24T06:40:00.124693+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:40:00.124693+0000","last_undegraded":"2024-03-24T06:40:00.124693+0000","last_fullsized":"2024-03-24T06:40:00.124693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125145+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:40:00.125145+0000","last_peered":"2024-03-24T06:40:00.125145+0000","last_clean":"2024-03-24T06:40:00.125145+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:40:00.125145+0000","last_undegraded":"2024-03-24T06:40:00.125145+0000","last_fullsized":"2024-03-24T06:40:00.125145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291432+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:40:00.291432+0000","last_peered":"2024-03-24T06:40:00.291432+0000","last_clean":"2024-03-24T06:40:00.291432+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:40:00.291432+0000","last_undegraded":"2024-03-24T06:40:00.291432+0000","last_fullsized":"2024-03-24T06:40:00.291432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124080+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:40:00.124080+0000","last_peered":"2024-03-24T06:40:00.124080+0000","last_clean":"2024-03-24T06:40:00.124080+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:40:00.124080+0000","last_undegraded":"2024-03-24T06:40:00.124080+0000","last_fullsized":"2024-03-24T06:40:00.124080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687904+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:40:00.687904+0000","last_peered":"2024-03-24T06:40:00.687904+0000","last_clean":"2024-03-24T06:40:00.687904+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:40:00.687904+0000","last_undegraded":"2024-03-24T06:40:00.687904+0000","last_fullsized":"2024-03-24T06:40:00.687904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680146+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:40:00.680146+0000","last_peered":"2024-03-24T06:40:00.680146+0000","last_clean":"2024-03-24T06:40:00.680146+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:40:00.680146+0000","last_undegraded":"2024-03-24T06:40:00.680146+0000","last_fullsized":"2024-03-24T06:40:00.680146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807199+0000","last_change":"2024-03-24T06:37:56.828731+0000","last_active":"2024-03-24T06:40:00.807199+0000","last_peered":"2024-03-24T06:40:00.807199+0000","last_clean":"2024-03-24T06:40:00.807199+0000","last_became_active":"2024-03-24T06:37:56.828384+0000","last_became_peered":"2024-03-24T06:37:56.828384+0000","last_unstale":"2024-03-24T06:40:00.807199+0000","last_undegraded":"2024-03-24T06:40:00.807199+0000","last_fullsized":"2024-03-24T06:40:00.807199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:26.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600649,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1599,"num_read_kb":921,"num_write":364,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383730,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":520134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29060,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716092,"statfs":{"total":95995035648,"available":95965278208,"internally_reserved":0,"allocated":1908736,"data_stored":1310142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":524138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070532,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1116832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495695,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":578353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":587788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1556480,"data_stored":984363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757333,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37136,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708016,"statfs":{"total":95995035648,"available":95957008384,"internally_reserved":0,"allocated":1843200,"data_stored":1254242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":201268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:40:05.294 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 -- 172.21.15.22:0/4082377358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c104b40 msgr2=0x7f351c104fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 --2- 172.21.15.22:0/4082377358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c104b40 0x7f351c104fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3508009900 tx=0x7f350802bc50 comp rx=0 tx=0).stop 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 -- 172.21.15.22:0/4082377358 shutdown_connections 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 --2- 172.21.15.22:0/4082377358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c105500 0x7f351c10bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 --2- 172.21.15.22:0/4082377358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c104b40 0x7f351c104fc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 --2- 172.21.15.22:0/4082377358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 -- 172.21.15.22:0/4082377358 >> 172.21.15.22:0/4082377358 conn(0x7f351c0fef00 msgr2=0x7f351c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:05.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 -- 172.21.15.22:0/4082377358 shutdown_connections 2024-03-24T06:40:05.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.538+0000 7f3523400640 1 -- 172.21.15.22:0/4082377358 wait complete. 2024-03-24T06:40:05.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.539+0000 7f3523400640 1 Processor -- start 2024-03-24T06:40:05.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 -- start start 2024-03-24T06:40:05.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:05.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 0x7f351c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:05.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 0x7f351c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:05.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1a14d0 con 0x7f351c104b40 2024-03-24T06:40:05.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1a1640 con 0x7f351c103950 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3523400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f351c1a1800 con 0x7f351c105500 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3520a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3520a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37022/0 (socket says 172.21.15.22:37022) 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3520a00640 1 -- 172.21.15.22:0/2962811408 learned_addr learned my addr 172.21.15.22:0/2962811408 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f351be00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 0x7f351c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.540+0000 7f3521400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 0x7f351c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 -- 172.21.15.22:0/2962811408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 msgr2=0x7f351c1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 0x7f351c1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 -- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 msgr2=0x7f351c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 0x7f351c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 -- 172.21.15.22:0/2962811408 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3510009490 con 0x7f351c103950 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3521400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 0x7f351c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f351be00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 0x7f351c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:05.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3520a00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f351000e9c0 tx=0x7f351000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000cde0 con 0x7f351c103950 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3510010910 con 0x7f351c103950 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510018a10 con 0x7f351c103950 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.541+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3508009560 con 0x7f351c103950 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.542+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351c1a1de0 con 0x7f351c103950 2024-03-24T06:40:05.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.543+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f351c10acf0 con 0x7f351c103950 2024-03-24T06:40:05.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.545+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f3510010a80 con 0x7f351c103950 2024-03-24T06:40:05.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.546+0000 7f3519600640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 0x7f35040a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:05.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.546+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3510014070 con 0x7f351c103950 2024-03-24T06:40:05.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.546+0000 7f351be00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 0x7f35040a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:05.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.547+0000 7f351be00640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 0x7f35040a4800 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3508006010 tx=0x7f3508005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3978919001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:40:05.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:05 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:40:05.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.555+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3510097500 con 0x7f351c103950 2024-03-24T06:40:05.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.752+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f351c02d240 con 0x7f35040a2340 2024-03-24T06:40:05.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.768+0000 7f3519600640 1 -- 172.21.15.22:0/2962811408 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339455 (secure 0 0 0) 0x7f351c02d240 con 0x7f35040a2340 2024-03-24T06:40:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:05.776 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:40:05.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 msgr2=0x7f35040a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 0x7f35040a4800 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3508006010 tx=0x7f3508005a20 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 msgr2=0x7f351c1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f351000e9c0 tx=0x7f351000ee80 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 shutdown_connections 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35040a2340 0x7f35040a4800 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f351c105500 0x7f351c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f351c104b40 0x7f351c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 --2- 172.21.15.22:0/2962811408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c103950 0x7f351c1999e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.776+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 >> 172.21.15.22:0/2962811408 conn(0x7f351c0fef00 msgr2=0x7f351c107ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.777+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 shutdown_connections 2024-03-24T06:40:05.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:05.777+0000 7f3523400640 1 -- 172.21.15.22:0/2962811408 wait complete. 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3978919001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:40:05.780 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:05 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:40:05.795 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T06:40:04.142233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191251,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1735,"num_read_kb":1040,"num_write":547,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181380,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":933,"ondisk_log_size":933,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389820,"kb_used_data":15424,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552004,"statfs":{"total":1151940427776,"available":1151541252096,"internally_reserved":0,"allocated":15794176,"data_stored":8969942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.977269"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807561+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:40:00.807561+0000","last_peered":"2024-03-24T06:40:00.807561+0000","last_clean":"2024-03-24T06:40:00.807561+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:40:00.807561+0000","last_undegraded":"2024-03-24T06:40:00.807561+0000","last_fullsized":"2024-03-24T06:40:00.807561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808126+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:40:00.808126+0000","last_peered":"2024-03-24T06:40:00.808126+0000","last_clean":"2024-03-24T06:40:00.808126+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:40:00.808126+0000","last_undegraded":"2024-03-24T06:40:00.808126+0000","last_fullsized":"2024-03-24T06:40:00.808126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807855+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:40:00.807855+0000","last_peered":"2024-03-24T06:40:00.807855+0000","last_clean":"2024-03-24T06:40:00.807855+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:40:00.807855+0000","last_undegraded":"2024-03-24T06:40:00.807855+0000","last_fullsized":"2024-03-24T06:40:00.807855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697661+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:40:00.697661+0000","last_peered":"2024-03-24T06:40:00.697661+0000","last_clean":"2024-03-24T06:40:00.697661+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:40:00.697661+0000","last_undegraded":"2024-03-24T06:40:00.697661+0000","last_fullsized":"2024-03-24T06:40:00.697661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281757+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:40:00.281757+0000","last_peered":"2024-03-24T06:40:00.281757+0000","last_clean":"2024-03-24T06:40:00.281757+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:40:00.281757+0000","last_undegraded":"2024-03-24T06:40:00.281757+0000","last_fullsized":"2024-03-24T06:40:00.281757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123976+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:40:00.123976+0000","last_peered":"2024-03-24T06:40:00.123976+0000","last_clean":"2024-03-24T06:40:00.123976+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:40:00.123976+0000","last_undegraded":"2024-03-24T06:40:00.123976+0000","last_fullsized":"2024-03-24T06:40:00.123976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652893+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:40:00.652893+0000","last_peered":"2024-03-24T06:40:00.652893+0000","last_clean":"2024-03-24T06:40:00.652893+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:40:00.652893+0000","last_undegraded":"2024-03-24T06:40:00.652893+0000","last_fullsized":"2024-03-24T06:40:00.652893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123459+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:40:00.123459+0000","last_peered":"2024-03-24T06:40:00.123459+0000","last_clean":"2024-03-24T06:40:00.123459+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:40:00.123459+0000","last_undegraded":"2024-03-24T06:40:00.123459+0000","last_fullsized":"2024-03-24T06:40:00.123459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688245+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:40:00.688245+0000","last_peered":"2024-03-24T06:40:00.688245+0000","last_clean":"2024-03-24T06:40:00.688245+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:40:00.688245+0000","last_undegraded":"2024-03-24T06:40:00.688245+0000","last_fullsized":"2024-03-24T06:40:00.688245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124396+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:40:00.124396+0000","last_peered":"2024-03-24T06:40:00.124396+0000","last_clean":"2024-03-24T06:40:00.124396+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:40:00.124396+0000","last_undegraded":"2024-03-24T06:40:00.124396+0000","last_fullsized":"2024-03-24T06:40:00.124396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653258+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:40:00.653258+0000","last_peered":"2024-03-24T06:40:00.653258+0000","last_clean":"2024-03-24T06:40:00.653258+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:40:00.653258+0000","last_undegraded":"2024-03-24T06:40:00.653258+0000","last_fullsized":"2024-03-24T06:40:00.653258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697898+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:40:00.697898+0000","last_peered":"2024-03-24T06:40:00.697898+0000","last_clean":"2024-03-24T06:40:00.697898+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:40:00.697898+0000","last_undegraded":"2024-03-24T06:40:00.697898+0000","last_fullsized":"2024-03-24T06:40:00.697898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680331+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:40:00.680331+0000","last_peered":"2024-03-24T06:40:00.680331+0000","last_clean":"2024-03-24T06:40:00.680331+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:40:00.680331+0000","last_undegraded":"2024-03-24T06:40:00.680331+0000","last_fullsized":"2024-03-24T06:40:00.680331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807397+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:40:00.807397+0000","last_peered":"2024-03-24T06:40:00.807397+0000","last_clean":"2024-03-24T06:40:00.807397+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:40:00.807397+0000","last_undegraded":"2024-03-24T06:40:00.807397+0000","last_fullsized":"2024-03-24T06:40:00.807397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807500+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:40:00.807500+0000","last_peered":"2024-03-24T06:40:00.807500+0000","last_clean":"2024-03-24T06:40:00.807500+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:40:00.807500+0000","last_undegraded":"2024-03-24T06:40:00.807500+0000","last_fullsized":"2024-03-24T06:40:00.807500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687644+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:40:00.687644+0000","last_peered":"2024-03-24T06:40:00.687644+0000","last_clean":"2024-03-24T06:40:00.687644+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:40:00.687644+0000","last_undegraded":"2024-03-24T06:40:00.687644+0000","last_fullsized":"2024-03-24T06:40:00.687644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807536+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:40:00.807536+0000","last_peered":"2024-03-24T06:40:00.807536+0000","last_clean":"2024-03-24T06:40:00.807536+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:40:00.807536+0000","last_undegraded":"2024-03-24T06:40:00.807536+0000","last_fullsized":"2024-03-24T06:40:00.807536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.278167+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:40:00.278167+0000","last_peered":"2024-03-24T06:40:00.278167+0000","last_clean":"2024-03-24T06:40:00.278167+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:40:00.278167+0000","last_undegraded":"2024-03-24T06:40:00.278167+0000","last_fullsized":"2024-03-24T06:40:00.278167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124626+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:40:00.124626+0000","last_peered":"2024-03-24T06:40:00.124626+0000","last_clean":"2024-03-24T06:40:00.124626+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:40:00.124626+0000","last_undegraded":"2024-03-24T06:40:00.124626+0000","last_fullsized":"2024-03-24T06:40:00.124626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123859+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:40:00.123859+0000","last_peered":"2024-03-24T06:40:00.123859+0000","last_clean":"2024-03-24T06:40:00.123859+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:40:00.123859+0000","last_undegraded":"2024-03-24T06:40:00.123859+0000","last_fullsized":"2024-03-24T06:40:00.123859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":197,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.651139+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:40:00.651139+0000","last_peered":"2024-03-24T06:40:00.651139+0000","last_clean":"2024-03-24T06:40:00.651139+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:40:00.651139+0000","last_undegraded":"2024-03-24T06:40:00.651139+0000","last_fullsized":"2024-03-24T06:40:00.651139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124742+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:40:00.124742+0000","last_peered":"2024-03-24T06:40:00.124742+0000","last_clean":"2024-03-24T06:40:00.124742+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:40:00.124742+0000","last_undegraded":"2024-03-24T06:40:00.124742+0000","last_fullsized":"2024-03-24T06:40:00.124742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697277+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:40:00.697277+0000","last_peered":"2024-03-24T06:40:00.697277+0000","last_clean":"2024-03-24T06:40:00.697277+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:40:00.697277+0000","last_undegraded":"2024-03-24T06:40:00.697277+0000","last_fullsized":"2024-03-24T06:40:00.697277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125229+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:40:00.125229+0000","last_peered":"2024-03-24T06:40:00.125229+0000","last_clean":"2024-03-24T06:40:00.125229+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:40:00.125229+0000","last_undegraded":"2024-03-24T06:40:00.125229+0000","last_fullsized":"2024-03-24T06:40:00.125229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808502+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:40:00.808502+0000","last_peered":"2024-03-24T06:40:00.808502+0000","last_clean":"2024-03-24T06:40:00.808502+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:40:00.808502+0000","last_undegraded":"2024-03-24T06:40:00.808502+0000","last_fullsized":"2024-03-24T06:40:00.808502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697506+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:40:00.697506+0000","last_peered":"2024-03-24T06:40:00.697506+0000","last_clean":"2024-03-24T06:40:00.697506+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:40:00.697506+0000","last_undegraded":"2024-03-24T06:40:00.697506+0000","last_fullsized":"2024-03-24T06:40:00.697506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282154+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:40:00.282154+0000","last_peered":"2024-03-24T06:40:00.282154+0000","last_clean":"2024-03-24T06:40:00.282154+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:40:00.282154+0000","last_undegraded":"2024-03-24T06:40:00.282154+0000","last_fullsized":"2024-03-24T06:40:00.282154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687945+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:40:00.687945+0000","last_peered":"2024-03-24T06:40:00.687945+0000","last_clean":"2024-03-24T06:40:00.687945+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:40:00.687945+0000","last_undegraded":"2024-03-24T06:40:00.687945+0000","last_fullsized":"2024-03-24T06:40:00.687945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697300+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:40:00.697300+0000","last_peered":"2024-03-24T06:40:00.697300+0000","last_clean":"2024-03-24T06:40:00.697300+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:40:00.697300+0000","last_undegraded":"2024-03-24T06:40:00.697300+0000","last_fullsized":"2024-03-24T06:40:00.697300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281817+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:40:00.281817+0000","last_peered":"2024-03-24T06:40:00.281817+0000","last_clean":"2024-03-24T06:40:00.281817+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:40:00.281817+0000","last_undegraded":"2024-03-24T06:40:00.281817+0000","last_fullsized":"2024-03-24T06:40:00.281817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291734+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:40:00.291734+0000","last_peered":"2024-03-24T06:40:00.291734+0000","last_clean":"2024-03-24T06:40:00.291734+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:40:00.291734+0000","last_undegraded":"2024-03-24T06:40:00.291734+0000","last_fullsized":"2024-03-24T06:40:00.291734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124183+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:40:00.124183+0000","last_peered":"2024-03-24T06:40:00.124183+0000","last_clean":"2024-03-24T06:40:00.124183+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:40:00.124183+0000","last_undegraded":"2024-03-24T06:40:00.124183+0000","last_fullsized":"2024-03-24T06:40:00.124183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959636+0000","last_change":"2024-03-24T06:37:56.839629+0000","last_active":"2024-03-24T06:40:00.959636+0000","last_peered":"2024-03-24T06:40:00.959636+0000","last_clean":"2024-03-24T06:40:00.959636+0000","last_became_active":"2024-03-24T06:37:56.832648+0000","last_became_peered":"2024-03-24T06:37:56.832648+0000","last_unstale":"2024-03-24T06:40:00.959636+0000","last_undegraded":"2024-03-24T06:40:00.959636+0000","last_fullsized":"2024-03-24T06:40:00.959636+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:57.270140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281637+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:40:00.281637+0000","last_peered":"2024-03-24T06:40:00.281637+0000","last_clean":"2024-03-24T06:40:00.281637+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:40:00.281637+0000","last_undegraded":"2024-03-24T06:40:00.281637+0000","last_fullsized":"2024-03-24T06:40:00.281637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124228+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:40:00.124228+0000","last_peered":"2024-03-24T06:40:00.124228+0000","last_clean":"2024-03-24T06:40:00.124228+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:40:00.124228+0000","last_undegraded":"2024-03-24T06:40:00.124228+0000","last_fullsized":"2024-03-24T06:40:00.124228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123925+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:40:00.123925+0000","last_peered":"2024-03-24T06:40:00.123925+0000","last_clean":"2024-03-24T06:40:00.123925+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:40:00.123925+0000","last_undegraded":"2024-03-24T06:40:00.123925+0000","last_fullsized":"2024-03-24T06:40:00.123925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291895+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:40:00.291895+0000","last_peered":"2024-03-24T06:40:00.291895+0000","last_clean":"2024-03-24T06:40:00.291895+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:40:00.291895+0000","last_undegraded":"2024-03-24T06:40:00.291895+0000","last_fullsized":"2024-03-24T06:40:00.291895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807182+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:40:00.807182+0000","last_peered":"2024-03-24T06:40:00.807182+0000","last_clean":"2024-03-24T06:40:00.807182+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:40:00.807182+0000","last_undegraded":"2024-03-24T06:40:00.807182+0000","last_fullsized":"2024-03-24T06:40:00.807182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679807+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:40:00.679807+0000","last_peered":"2024-03-24T06:40:00.679807+0000","last_clean":"2024-03-24T06:40:00.679807+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:40:00.679807+0000","last_undegraded":"2024-03-24T06:40:00.679807+0000","last_fullsized":"2024-03-24T06:40:00.679807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123726+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:40:00.123726+0000","last_peered":"2024-03-24T06:40:00.123726+0000","last_clean":"2024-03-24T06:40:00.123726+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:40:00.123726+0000","last_undegraded":"2024-03-24T06:40:00.123726+0000","last_fullsized":"2024-03-24T06:40:00.123726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808088+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:40:00.808088+0000","last_peered":"2024-03-24T06:40:00.808088+0000","last_clean":"2024-03-24T06:40:00.808088+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:40:00.808088+0000","last_undegraded":"2024-03-24T06:40:00.808088+0000","last_fullsized":"2024-03-24T06:40:00.808088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291185+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:40:00.291185+0000","last_peered":"2024-03-24T06:40:00.291185+0000","last_clean":"2024-03-24T06:40:00.291185+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:40:00.291185+0000","last_undegraded":"2024-03-24T06:40:00.291185+0000","last_fullsized":"2024-03-24T06:40:00.291185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277299+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:40:00.277299+0000","last_peered":"2024-03-24T06:40:00.277299+0000","last_clean":"2024-03-24T06:40:00.277299+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:40:00.277299+0000","last_undegraded":"2024-03-24T06:40:00.277299+0000","last_fullsized":"2024-03-24T06:40:00.277299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123735+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:40:00.123735+0000","last_peered":"2024-03-24T06:40:00.123735+0000","last_clean":"2024-03-24T06:40:00.123735+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:40:00.123735+0000","last_undegraded":"2024-03-24T06:40:00.123735+0000","last_fullsized":"2024-03-24T06:40:00.123735+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":334,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.590647+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:40:00.590647+0000","last_peered":"2024-03-24T06:40:00.590647+0000","last_clean":"2024-03-24T06:40:00.590647+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:40:00.590647+0000","last_undegraded":"2024-03-24T06:40:00.590647+0000","last_fullsized":"2024-03-24T06:40:00.590647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":296,"num_read_kb":296,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688179+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:40:00.688179+0000","last_peered":"2024-03-24T06:40:00.688179+0000","last_clean":"2024-03-24T06:40:00.688179+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:40:00.688179+0000","last_undegraded":"2024-03-24T06:40:00.688179+0000","last_fullsized":"2024-03-24T06:40:00.688179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281341+0000","last_change":"2024-03-24T06:37:56.831960+0000","last_active":"2024-03-24T06:40:00.281341+0000","last_peered":"2024-03-24T06:40:00.281341+0000","last_clean":"2024-03-24T06:40:00.281341+0000","last_became_active":"2024-03-24T06:37:56.831524+0000","last_became_peered":"2024-03-24T06:37:56.831524+0000","last_unstale":"2024-03-24T06:40:00.281341+0000","last_undegraded":"2024-03-24T06:40:00.281341+0000","last_fullsized":"2024-03-24T06:40:00.281341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:28.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653238+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:40:00.653238+0000","last_peered":"2024-03-24T06:40:00.653238+0000","last_clean":"2024-03-24T06:40:00.653238+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:40:00.653238+0000","last_undegraded":"2024-03-24T06:40:00.653238+0000","last_fullsized":"2024-03-24T06:40:00.653238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808357+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:40:00.808357+0000","last_peered":"2024-03-24T06:40:00.808357+0000","last_clean":"2024-03-24T06:40:00.808357+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:40:00.808357+0000","last_undegraded":"2024-03-24T06:40:00.808357+0000","last_fullsized":"2024-03-24T06:40:00.808357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697632+0000","last_change":"2024-03-24T06:37:57.056640+0000","last_active":"2024-03-24T06:40:00.697632+0000","last_peered":"2024-03-24T06:40:00.697632+0000","last_clean":"2024-03-24T06:40:00.697632+0000","last_became_active":"2024-03-24T06:37:57.056087+0000","last_became_peered":"2024-03-24T06:37:57.056087+0000","last_unstale":"2024-03-24T06:40:00.697632+0000","last_undegraded":"2024-03-24T06:40:00.697632+0000","last_fullsized":"2024-03-24T06:40:00.697632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:46.933123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291609+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:40:00.291609+0000","last_peered":"2024-03-24T06:40:00.291609+0000","last_clean":"2024-03-24T06:40:00.291609+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:40:00.291609+0000","last_undegraded":"2024-03-24T06:40:00.291609+0000","last_fullsized":"2024-03-24T06:40:00.291609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688042+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:40:00.688042+0000","last_peered":"2024-03-24T06:40:00.688042+0000","last_clean":"2024-03-24T06:40:00.688042+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:40:00.688042+0000","last_undegraded":"2024-03-24T06:40:00.688042+0000","last_fullsized":"2024-03-24T06:40:00.688042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277348+0000","last_change":"2024-03-24T06:38:58.624331+0000","last_active":"2024-03-24T06:40:00.277348+0000","last_peered":"2024-03-24T06:40:00.277348+0000","last_clean":"2024-03-24T06:40:00.277348+0000","last_became_active":"2024-03-24T06:38:58.623562+0000","last_became_peered":"2024-03-24T06:38:58.623562+0000","last_unstale":"2024-03-24T06:40:00.277348+0000","last_undegraded":"2024-03-24T06:40:00.277348+0000","last_fullsized":"2024-03-24T06:40:00.277348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:19:32.355669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":377,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959384+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:40:00.959384+0000","last_peered":"2024-03-24T06:40:00.959384+0000","last_clean":"2024-03-24T06:40:00.959384+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:40:00.959384+0000","last_undegraded":"2024-03-24T06:40:00.959384+0000","last_fullsized":"2024-03-24T06:40:00.959384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67932,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":304,"num_read_kb":219,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281881+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:40:00.281881+0000","last_peered":"2024-03-24T06:40:00.281881+0000","last_clean":"2024-03-24T06:40:00.281881+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:40:00.281881+0000","last_undegraded":"2024-03-24T06:40:00.281881+0000","last_fullsized":"2024-03-24T06:40:00.281881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291994+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:40:00.291994+0000","last_peered":"2024-03-24T06:40:00.291994+0000","last_clean":"2024-03-24T06:40:00.291994+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:40:00.291994+0000","last_undegraded":"2024-03-24T06:40:00.291994+0000","last_fullsized":"2024-03-24T06:40:00.291994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124117+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:40:00.124117+0000","last_peered":"2024-03-24T06:40:00.124117+0000","last_clean":"2024-03-24T06:40:00.124117+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:40:00.124117+0000","last_undegraded":"2024-03-24T06:40:00.124117+0000","last_fullsized":"2024-03-24T06:40:00.124117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679797+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:40:00.679797+0000","last_peered":"2024-03-24T06:40:00.679797+0000","last_clean":"2024-03-24T06:40:00.679797+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:40:00.679797+0000","last_undegraded":"2024-03-24T06:40:00.679797+0000","last_fullsized":"2024-03-24T06:40:00.679797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807095+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:40:00.807095+0000","last_peered":"2024-03-24T06:40:00.807095+0000","last_clean":"2024-03-24T06:40:00.807095+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:40:00.807095+0000","last_undegraded":"2024-03-24T06:40:00.807095+0000","last_fullsized":"2024-03-24T06:40:00.807095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291586+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:40:00.291586+0000","last_peered":"2024-03-24T06:40:00.291586+0000","last_clean":"2024-03-24T06:40:00.291586+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:40:00.291586+0000","last_undegraded":"2024-03-24T06:40:00.291586+0000","last_fullsized":"2024-03-24T06:40:00.291586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653539+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:40:00.653539+0000","last_peered":"2024-03-24T06:40:00.653539+0000","last_clean":"2024-03-24T06:40:00.653539+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:40:00.653539+0000","last_undegraded":"2024-03-24T06:40:00.653539+0000","last_fullsized":"2024-03-24T06:40:00.653539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279380+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:40:00.279380+0000","last_peered":"2024-03-24T06:40:00.279380+0000","last_clean":"2024-03-24T06:40:00.279380+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:40:00.279380+0000","last_undegraded":"2024-03-24T06:40:00.279380+0000","last_fullsized":"2024-03-24T06:40:00.279380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697530+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:40:00.697530+0000","last_peered":"2024-03-24T06:40:00.697530+0000","last_clean":"2024-03-24T06:40:00.697530+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:40:00.697530+0000","last_undegraded":"2024-03-24T06:40:00.697530+0000","last_fullsized":"2024-03-24T06:40:00.697530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":151,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652957+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:40:00.652957+0000","last_peered":"2024-03-24T06:40:00.652957+0000","last_clean":"2024-03-24T06:40:00.652957+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:40:00.652957+0000","last_undegraded":"2024-03-24T06:40:00.652957+0000","last_fullsized":"2024-03-24T06:40:00.652957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681557+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:40:00.681557+0000","last_peered":"2024-03-24T06:40:00.681557+0000","last_clean":"2024-03-24T06:40:00.681557+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:40:00.681557+0000","last_undegraded":"2024-03-24T06:40:00.681557+0000","last_fullsized":"2024-03-24T06:40:00.681557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281346+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:40:00.281346+0000","last_peered":"2024-03-24T06:40:00.281346+0000","last_clean":"2024-03-24T06:40:00.281346+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:40:00.281346+0000","last_undegraded":"2024-03-24T06:40:00.281346+0000","last_fullsized":"2024-03-24T06:40:00.281346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291241+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:40:00.291241+0000","last_peered":"2024-03-24T06:40:00.291241+0000","last_clean":"2024-03-24T06:40:00.291241+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:40:00.291241+0000","last_undegraded":"2024-03-24T06:40:00.291241+0000","last_fullsized":"2024-03-24T06:40:00.291241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291201+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:40:00.291201+0000","last_peered":"2024-03-24T06:40:00.291201+0000","last_clean":"2024-03-24T06:40:00.291201+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:40:00.291201+0000","last_undegraded":"2024-03-24T06:40:00.291201+0000","last_fullsized":"2024-03-24T06:40:00.291201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807189+0000","last_change":"2024-03-24T06:37:56.837415+0000","last_active":"2024-03-24T06:40:00.807189+0000","last_peered":"2024-03-24T06:40:00.807189+0000","last_clean":"2024-03-24T06:40:00.807189+0000","last_became_active":"2024-03-24T06:37:56.832240+0000","last_became_peered":"2024-03-24T06:37:56.832240+0000","last_unstale":"2024-03-24T06:40:00.807189+0000","last_undegraded":"2024-03-24T06:40:00.807189+0000","last_fullsized":"2024-03-24T06:40:00.807189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:44.867297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807274+0000","last_change":"2024-03-24T06:37:56.826934+0000","last_active":"2024-03-24T06:40:00.807274+0000","last_peered":"2024-03-24T06:40:00.807274+0000","last_clean":"2024-03-24T06:40:00.807274+0000","last_became_active":"2024-03-24T06:37:56.826574+0000","last_became_peered":"2024-03-24T06:37:56.826574+0000","last_unstale":"2024-03-24T06:40:00.807274+0000","last_undegraded":"2024-03-24T06:40:00.807274+0000","last_fullsized":"2024-03-24T06:40:00.807274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:52.406936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281632+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:40:00.281632+0000","last_peered":"2024-03-24T06:40:00.281632+0000","last_clean":"2024-03-24T06:40:00.281632+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:40:00.281632+0000","last_undegraded":"2024-03-24T06:40:00.281632+0000","last_fullsized":"2024-03-24T06:40:00.281632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277730+0000","last_change":"2024-03-24T06:37:56.832165+0000","last_active":"2024-03-24T06:40:00.277730+0000","last_peered":"2024-03-24T06:40:00.277730+0000","last_clean":"2024-03-24T06:40:00.277730+0000","last_became_active":"2024-03-24T06:37:56.831670+0000","last_became_peered":"2024-03-24T06:37:56.831670+0000","last_unstale":"2024-03-24T06:40:00.277730+0000","last_undegraded":"2024-03-24T06:40:00.277730+0000","last_fullsized":"2024-03-24T06:40:00.277730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.201188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124107+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:40:00.124107+0000","last_peered":"2024-03-24T06:40:00.124107+0000","last_clean":"2024-03-24T06:40:00.124107+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:40:00.124107+0000","last_undegraded":"2024-03-24T06:40:00.124107+0000","last_fullsized":"2024-03-24T06:40:00.124107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680650+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:40:00.680650+0000","last_peered":"2024-03-24T06:40:00.680650+0000","last_clean":"2024-03-24T06:40:00.680650+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:40:00.680650+0000","last_undegraded":"2024-03-24T06:40:00.680650+0000","last_fullsized":"2024-03-24T06:40:00.680650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687830+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:40:00.687830+0000","last_peered":"2024-03-24T06:40:00.687830+0000","last_clean":"2024-03-24T06:40:00.687830+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:40:00.687830+0000","last_undegraded":"2024-03-24T06:40:00.687830+0000","last_fullsized":"2024-03-24T06:40:00.687830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123953+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:40:00.123953+0000","last_peered":"2024-03-24T06:40:00.123953+0000","last_clean":"2024-03-24T06:40:00.123953+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:40:00.123953+0000","last_undegraded":"2024-03-24T06:40:00.123953+0000","last_fullsized":"2024-03-24T06:40:00.123953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681686+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:40:00.681686+0000","last_peered":"2024-03-24T06:40:00.681686+0000","last_clean":"2024-03-24T06:40:00.681686+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:40:00.681686+0000","last_undegraded":"2024-03-24T06:40:00.681686+0000","last_fullsized":"2024-03-24T06:40:00.681686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.292031+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:40:00.292031+0000","last_peered":"2024-03-24T06:40:00.292031+0000","last_clean":"2024-03-24T06:40:00.292031+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:40:00.292031+0000","last_undegraded":"2024-03-24T06:40:00.292031+0000","last_fullsized":"2024-03-24T06:40:00.292031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":187,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679720+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:40:00.679720+0000","last_peered":"2024-03-24T06:40:00.679720+0000","last_clean":"2024-03-24T06:40:00.679720+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:40:00.679720+0000","last_undegraded":"2024-03-24T06:40:00.679720+0000","last_fullsized":"2024-03-24T06:40:00.679720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653101+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:40:00.653101+0000","last_peered":"2024-03-24T06:40:00.653101+0000","last_clean":"2024-03-24T06:40:00.653101+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:40:00.653101+0000","last_undegraded":"2024-03-24T06:40:00.653101+0000","last_fullsized":"2024-03-24T06:40:00.653101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124031+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:40:00.124031+0000","last_peered":"2024-03-24T06:40:00.124031+0000","last_clean":"2024-03-24T06:40:00.124031+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:40:00.124031+0000","last_undegraded":"2024-03-24T06:40:00.124031+0000","last_fullsized":"2024-03-24T06:40:00.124031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697561+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:40:00.697561+0000","last_peered":"2024-03-24T06:40:00.697561+0000","last_clean":"2024-03-24T06:40:00.697561+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:40:00.697561+0000","last_undegraded":"2024-03-24T06:40:00.697561+0000","last_fullsized":"2024-03-24T06:40:00.697561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652830+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:40:00.652830+0000","last_peered":"2024-03-24T06:40:00.652830+0000","last_clean":"2024-03-24T06:40:00.652830+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:40:00.652830+0000","last_undegraded":"2024-03-24T06:40:00.652830+0000","last_fullsized":"2024-03-24T06:40:00.652830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291596+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:40:00.291596+0000","last_peered":"2024-03-24T06:40:00.291596+0000","last_clean":"2024-03-24T06:40:00.291596+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:40:00.291596+0000","last_undegraded":"2024-03-24T06:40:00.291596+0000","last_fullsized":"2024-03-24T06:40:00.291596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679511+0000","last_change":"2024-03-24T06:37:56.831122+0000","last_active":"2024-03-24T06:40:00.679511+0000","last_peered":"2024-03-24T06:40:00.679511+0000","last_clean":"2024-03-24T06:40:00.679511+0000","last_became_active":"2024-03-24T06:37:56.830642+0000","last_became_peered":"2024-03-24T06:37:56.830642+0000","last_unstale":"2024-03-24T06:40:00.679511+0000","last_undegraded":"2024-03-24T06:40:00.679511+0000","last_fullsized":"2024-03-24T06:40:00.679511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.204542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652901+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:40:00.652901+0000","last_peered":"2024-03-24T06:40:00.652901+0000","last_clean":"2024-03-24T06:40:00.652901+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:40:00.652901+0000","last_undegraded":"2024-03-24T06:40:00.652901+0000","last_fullsized":"2024-03-24T06:40:00.652901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806953+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:40:00.806953+0000","last_peered":"2024-03-24T06:40:00.806953+0000","last_clean":"2024-03-24T06:40:00.806953+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:40:00.806953+0000","last_undegraded":"2024-03-24T06:40:00.806953+0000","last_fullsized":"2024-03-24T06:40:00.806953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282064+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:40:00.282064+0000","last_peered":"2024-03-24T06:40:00.282064+0000","last_clean":"2024-03-24T06:40:00.282064+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:40:00.282064+0000","last_undegraded":"2024-03-24T06:40:00.282064+0000","last_fullsized":"2024-03-24T06:40:00.282064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680823+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:40:00.680823+0000","last_peered":"2024-03-24T06:40:00.680823+0000","last_clean":"2024-03-24T06:40:00.680823+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:40:00.680823+0000","last_undegraded":"2024-03-24T06:40:00.680823+0000","last_fullsized":"2024-03-24T06:40:00.680823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123597+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:40:00.123597+0000","last_peered":"2024-03-24T06:40:00.123597+0000","last_clean":"2024-03-24T06:40:00.123597+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:40:00.123597+0000","last_undegraded":"2024-03-24T06:40:00.123597+0000","last_fullsized":"2024-03-24T06:40:00.123597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.457728+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:40:00.457728+0000","last_peered":"2024-03-24T06:40:00.457728+0000","last_clean":"2024-03-24T06:40:00.457728+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:40:00.457728+0000","last_undegraded":"2024-03-24T06:40:00.457728+0000","last_fullsized":"2024-03-24T06:40:00.457728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":85,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124952+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:40:00.124952+0000","last_peered":"2024-03-24T06:40:00.124952+0000","last_clean":"2024-03-24T06:40:00.124952+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:40:00.124952+0000","last_undegraded":"2024-03-24T06:40:00.124952+0000","last_fullsized":"2024-03-24T06:40:00.124952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123851+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:40:00.123851+0000","last_peered":"2024-03-24T06:40:00.123851+0000","last_clean":"2024-03-24T06:40:00.123851+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:40:00.123851+0000","last_undegraded":"2024-03-24T06:40:00.123851+0000","last_fullsized":"2024-03-24T06:40:00.123851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697902+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:40:00.697902+0000","last_peered":"2024-03-24T06:40:00.697902+0000","last_clean":"2024-03-24T06:40:00.697902+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:40:00.697902+0000","last_undegraded":"2024-03-24T06:40:00.697902+0000","last_fullsized":"2024-03-24T06:40:00.697902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808204+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:40:00.808204+0000","last_peered":"2024-03-24T06:40:00.808204+0000","last_clean":"2024-03-24T06:40:00.808204+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:40:00.808204+0000","last_undegraded":"2024-03-24T06:40:00.808204+0000","last_fullsized":"2024-03-24T06:40:00.808204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697694+0000","last_change":"2024-03-24T06:37:57.056479+0000","last_active":"2024-03-24T06:40:00.697694+0000","last_peered":"2024-03-24T06:40:00.697694+0000","last_clean":"2024-03-24T06:40:00.697694+0000","last_became_active":"2024-03-24T06:37:57.055993+0000","last_became_peered":"2024-03-24T06:37:57.055993+0000","last_unstale":"2024-03-24T06:40:00.697694+0000","last_undegraded":"2024-03-24T06:40:00.697694+0000","last_fullsized":"2024-03-24T06:40:00.697694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:12.473494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687639+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:40:00.687639+0000","last_peered":"2024-03-24T06:40:00.687639+0000","last_clean":"2024-03-24T06:40:00.687639+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:40:00.687639+0000","last_undegraded":"2024-03-24T06:40:00.687639+0000","last_fullsized":"2024-03-24T06:40:00.687639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281408+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:40:00.281408+0000","last_peered":"2024-03-24T06:40:00.281408+0000","last_clean":"2024-03-24T06:40:00.281408+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:40:00.281408+0000","last_undegraded":"2024-03-24T06:40:00.281408+0000","last_fullsized":"2024-03-24T06:40:00.281408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279502+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:40:00.279502+0000","last_peered":"2024-03-24T06:40:00.279502+0000","last_clean":"2024-03-24T06:40:00.279502+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:40:00.279502+0000","last_undegraded":"2024-03-24T06:40:00.279502+0000","last_fullsized":"2024-03-24T06:40:00.279502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123941+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:40:00.123941+0000","last_peered":"2024-03-24T06:40:00.123941+0000","last_clean":"2024-03-24T06:40:00.123941+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:40:00.123941+0000","last_undegraded":"2024-03-24T06:40:00.123941+0000","last_fullsized":"2024-03-24T06:40:00.123941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806870+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:40:00.806870+0000","last_peered":"2024-03-24T06:40:00.806870+0000","last_clean":"2024-03-24T06:40:00.806870+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:40:00.806870+0000","last_undegraded":"2024-03-24T06:40:00.806870+0000","last_fullsized":"2024-03-24T06:40:00.806870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697505+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:40:00.697505+0000","last_peered":"2024-03-24T06:40:00.697505+0000","last_clean":"2024-03-24T06:40:00.697505+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:40:00.697505+0000","last_undegraded":"2024-03-24T06:40:00.697505+0000","last_fullsized":"2024-03-24T06:40:00.697505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123999+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:40:00.123999+0000","last_peered":"2024-03-24T06:40:00.123999+0000","last_clean":"2024-03-24T06:40:00.123999+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:40:00.123999+0000","last_undegraded":"2024-03-24T06:40:00.123999+0000","last_fullsized":"2024-03-24T06:40:00.123999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125506+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:40:00.125506+0000","last_peered":"2024-03-24T06:40:00.125506+0000","last_clean":"2024-03-24T06:40:00.125506+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:40:00.125506+0000","last_undegraded":"2024-03-24T06:40:00.125506+0000","last_fullsized":"2024-03-24T06:40:00.125506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697583+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:40:00.697583+0000","last_peered":"2024-03-24T06:40:00.697583+0000","last_clean":"2024-03-24T06:40:00.697583+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:40:00.697583+0000","last_undegraded":"2024-03-24T06:40:00.697583+0000","last_fullsized":"2024-03-24T06:40:00.697583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281619+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:40:00.281619+0000","last_peered":"2024-03-24T06:40:00.281619+0000","last_clean":"2024-03-24T06:40:00.281619+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:40:00.281619+0000","last_undegraded":"2024-03-24T06:40:00.281619+0000","last_fullsized":"2024-03-24T06:40:00.281619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653331+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:40:00.653331+0000","last_peered":"2024-03-24T06:40:00.653331+0000","last_clean":"2024-03-24T06:40:00.653331+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:40:00.653331+0000","last_undegraded":"2024-03-24T06:40:00.653331+0000","last_fullsized":"2024-03-24T06:40:00.653331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687653+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:40:00.687653+0000","last_peered":"2024-03-24T06:40:00.687653+0000","last_clean":"2024-03-24T06:40:00.687653+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:40:00.687653+0000","last_undegraded":"2024-03-24T06:40:00.687653+0000","last_fullsized":"2024-03-24T06:40:00.687653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124472+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:40:00.124472+0000","last_peered":"2024-03-24T06:40:00.124472+0000","last_clean":"2024-03-24T06:40:00.124472+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:40:00.124472+0000","last_undegraded":"2024-03-24T06:40:00.124472+0000","last_fullsized":"2024-03-24T06:40:00.124472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653414+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:40:00.653414+0000","last_peered":"2024-03-24T06:40:00.653414+0000","last_clean":"2024-03-24T06:40:00.653414+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:40:00.653414+0000","last_undegraded":"2024-03-24T06:40:00.653414+0000","last_fullsized":"2024-03-24T06:40:00.653414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807599+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:40:00.807599+0000","last_peered":"2024-03-24T06:40:00.807599+0000","last_clean":"2024-03-24T06:40:00.807599+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:40:00.807599+0000","last_undegraded":"2024-03-24T06:40:00.807599+0000","last_fullsized":"2024-03-24T06:40:00.807599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.454352+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:40:00.454352+0000","last_peered":"2024-03-24T06:40:00.454352+0000","last_clean":"2024-03-24T06:40:00.454352+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:40:00.454352+0000","last_undegraded":"2024-03-24T06:40:00.454352+0000","last_fullsized":"2024-03-24T06:40:00.454352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653114+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:40:00.653114+0000","last_peered":"2024-03-24T06:40:00.653114+0000","last_clean":"2024-03-24T06:40:00.653114+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:40:00.653114+0000","last_undegraded":"2024-03-24T06:40:00.653114+0000","last_fullsized":"2024-03-24T06:40:00.653114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.290992+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:40:00.290992+0000","last_peered":"2024-03-24T06:40:00.290992+0000","last_clean":"2024-03-24T06:40:00.290992+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:40:00.290992+0000","last_undegraded":"2024-03-24T06:40:00.290992+0000","last_fullsized":"2024-03-24T06:40:00.290992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681423+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:40:00.681423+0000","last_peered":"2024-03-24T06:40:00.681423+0000","last_clean":"2024-03-24T06:40:00.681423+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:40:00.681423+0000","last_undegraded":"2024-03-24T06:40:00.681423+0000","last_fullsized":"2024-03-24T06:40:00.681423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653402+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:40:00.653402+0000","last_peered":"2024-03-24T06:40:00.653402+0000","last_clean":"2024-03-24T06:40:00.653402+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:40:00.653402+0000","last_undegraded":"2024-03-24T06:40:00.653402+0000","last_fullsized":"2024-03-24T06:40:00.653402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806949+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:40:00.806949+0000","last_peered":"2024-03-24T06:40:00.806949+0000","last_clean":"2024-03-24T06:40:00.806949+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:40:00.806949+0000","last_undegraded":"2024-03-24T06:40:00.806949+0000","last_fullsized":"2024-03-24T06:40:00.806949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124150+0000","last_change":"2024-03-24T06:37:56.832171+0000","last_active":"2024-03-24T06:40:00.124150+0000","last_peered":"2024-03-24T06:40:00.124150+0000","last_clean":"2024-03-24T06:40:00.124150+0000","last_became_active":"2024-03-24T06:37:56.831734+0000","last_became_peered":"2024-03-24T06:37:56.831734+0000","last_unstale":"2024-03-24T06:40:00.124150+0000","last_undegraded":"2024-03-24T06:40:00.124150+0000","last_fullsized":"2024-03-24T06:40:00.124150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:19.788831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123937+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:40:00.123937+0000","last_peered":"2024-03-24T06:40:00.123937+0000","last_clean":"2024-03-24T06:40:00.123937+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:40:00.123937+0000","last_undegraded":"2024-03-24T06:40:00.123937+0000","last_fullsized":"2024-03-24T06:40:00.123937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807833+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:40:00.807833+0000","last_peered":"2024-03-24T06:40:00.807833+0000","last_clean":"2024-03-24T06:40:00.807833+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:40:00.807833+0000","last_undegraded":"2024-03-24T06:40:00.807833+0000","last_fullsized":"2024-03-24T06:40:00.807833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680980+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:40:00.680980+0000","last_peered":"2024-03-24T06:40:00.680980+0000","last_clean":"2024-03-24T06:40:00.680980+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:40:00.680980+0000","last_undegraded":"2024-03-24T06:40:00.680980+0000","last_fullsized":"2024-03-24T06:40:00.680980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807380+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:40:00.807380+0000","last_peered":"2024-03-24T06:40:00.807380+0000","last_clean":"2024-03-24T06:40:00.807380+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:40:00.807380+0000","last_undegraded":"2024-03-24T06:40:00.807380+0000","last_fullsized":"2024-03-24T06:40:00.807380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124693+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:40:00.124693+0000","last_peered":"2024-03-24T06:40:00.124693+0000","last_clean":"2024-03-24T06:40:00.124693+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:40:00.124693+0000","last_undegraded":"2024-03-24T06:40:00.124693+0000","last_fullsized":"2024-03-24T06:40:00.124693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125145+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:40:00.125145+0000","last_peered":"2024-03-24T06:40:00.125145+0000","last_clean":"2024-03-24T06:40:00.125145+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:40:00.125145+0000","last_undegraded":"2024-03-24T06:40:00.125145+0000","last_fullsized":"2024-03-24T06:40:00.125145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291432+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:40:00.291432+0000","last_peered":"2024-03-24T06:40:00.291432+0000","last_clean":"2024-03-24T06:40:00.291432+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:40:00.291432+0000","last_undegraded":"2024-03-24T06:40:00.291432+0000","last_fullsized":"2024-03-24T06:40:00.291432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124080+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:40:00.124080+0000","last_peered":"2024-03-24T06:40:00.124080+0000","last_clean":"2024-03-24T06:40:00.124080+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:40:00.124080+0000","last_undegraded":"2024-03-24T06:40:00.124080+0000","last_fullsized":"2024-03-24T06:40:00.124080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687904+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:40:00.687904+0000","last_peered":"2024-03-24T06:40:00.687904+0000","last_clean":"2024-03-24T06:40:00.687904+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:40:00.687904+0000","last_undegraded":"2024-03-24T06:40:00.687904+0000","last_fullsized":"2024-03-24T06:40:00.687904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680146+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:40:00.680146+0000","last_peered":"2024-03-24T06:40:00.680146+0000","last_clean":"2024-03-24T06:40:00.680146+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:40:00.680146+0000","last_undegraded":"2024-03-24T06:40:00.680146+0000","last_fullsized":"2024-03-24T06:40:00.680146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807199+0000","last_change":"2024-03-24T06:37:56.828731+0000","last_active":"2024-03-24T06:40:00.807199+0000","last_peered":"2024-03-24T06:40:00.807199+0000","last_clean":"2024-03-24T06:40:00.807199+0000","last_became_active":"2024-03-24T06:37:56.828384+0000","last_became_peered":"2024-03-24T06:37:56.828384+0000","last_unstale":"2024-03-24T06:40:00.807199+0000","last_undegraded":"2024-03-24T06:40:00.807199+0000","last_fullsized":"2024-03-24T06:40:00.807199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:26.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600649,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1599,"num_read_kb":921,"num_write":364,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383730,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":520134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29060,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716092,"statfs":{"total":95995035648,"available":95965278208,"internally_reserved":0,"allocated":1908736,"data_stored":1310142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":524138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070532,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1116832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495695,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":578353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":587788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1556480,"data_stored":984363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757333,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37136,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708016,"statfs":{"total":95995035648,"available":95957008384,"internally_reserved":0,"allocated":1843200,"data_stored":1254242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":201268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:40:05.799 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-24T06:40:05.799 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3978919001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:40:05.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:05 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:40:06.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 -- 172.21.15.22:0/2016231385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c074170 msgr2=0x7fd71c0745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 --2- 172.21.15.22:0/2016231385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c074170 0x7fd71c0745f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd708009900 tx=0x7fd70802bc40 comp rx=0 tx=0).stop 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 -- 172.21.15.22:0/2016231385 shutdown_connections 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 --2- 172.21.15.22:0/2016231385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 --2- 172.21.15.22:0/2016231385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c074170 0x7fd71c0745f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 --2- 172.21.15.22:0/2016231385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c075750 0x7fd71c073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 -- 172.21.15.22:0/2016231385 >> 172.21.15.22:0/2016231385 conn(0x7fd71c0fcaf0 msgr2=0x7fd71c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.026+0000 7fd722e00640 1 -- 172.21.15.22:0/2016231385 shutdown_connections 2024-03-24T06:40:06.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.027+0000 7fd722e00640 1 -- 172.21.15.22:0/2016231385 wait complete. 2024-03-24T06:40:06.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.027+0000 7fd722e00640 1 Processor -- start 2024-03-24T06:40:06.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 -- start start 2024-03-24T06:40:06.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c074170 0x7fd71c19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c075750 0x7fd71c19e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c1a5700 con 0x7fd71c074170 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c1a5870 con 0x7fd71c10db50 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c1a5a30 con 0x7fd71c075750 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37050/0 (socket says 172.21.15.22:37050) 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd721000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c075750 0x7fd71c19e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 -- 172.21.15.22:0/4184550670 learned_addr learned my addr 172.21.15.22:0/4184550670 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd721a00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c074170 0x7fd71c19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 -- 172.21.15.22:0/4184550670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c075750 msgr2=0x7fd71c19e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c075750 0x7fd71c19e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.028+0000 7fd722400640 1 -- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c074170 msgr2=0x7fd71c19dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd722400640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c074170 0x7fd71c19dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd722400640 1 -- 172.21.15.22:0/4184550670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd708009560 con 0x7fd71c10db50 2024-03-24T06:40:06.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd722400640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eb10 tx=0x7fd71400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:06.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714004b20 con 0x7fd71c10db50 2024-03-24T06:40:06.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd714004c80 con 0x7fd71c10db50 2024-03-24T06:40:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c1a5d10 con 0x7fd71c10db50 2024-03-24T06:40:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714010650 con 0x7fd71c10db50 2024-03-24T06:40:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.029+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c1a61d0 con 0x7fd71c10db50 2024-03-24T06:40:06.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.030+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c103d20 con 0x7fd71c10db50 2024-03-24T06:40:06.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.032+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fd714005020 con 0x7fd71c10db50 2024-03-24T06:40:06.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.033+0000 7fd71a000640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 0x7fd7040a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.034+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd714014070 con 0x7fd71c10db50 2024-03-24T06:40:06.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.034+0000 7fd721a00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 0x7fd7040a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.034+0000 7fd721a00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 0x7fd7040a4780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd71c19f0f0 tx=0x7fd710005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:06.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.037+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd714097310 con 0x7fd71c10db50 2024-03-24T06:40:06.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.254+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd71c02d2e0 con 0x7fd7040a22c0 2024-03-24T06:40:06.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.276+0000 7fd71a000640 1 -- 172.21.15.22:0/4184550670 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339455 (secure 0 0 0) 0x7fd71c02d2e0 con 0x7fd7040a22c0 2024-03-24T06:40:06.280 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:06.282 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T06:40:06.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 msgr2=0x7fd7040a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 0x7fd7040a4780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd71c19f0f0 tx=0x7fd710005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 msgr2=0x7fd71c1a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eb10 tx=0x7fd71400c2d0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 shutdown_connections 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7040a22c0 0x7fd7040a4780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd71c10db50 0x7fd71c1a5190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd71c075750 0x7fd71c19e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 --2- 172.21.15.22:0/4184550670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd71c074170 0x7fd71c19dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 >> 172.21.15.22:0/4184550670 conn(0x7fd71c0fcaf0 msgr2=0x7fd71c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 shutdown_connections 2024-03-24T06:40:06.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.283+0000 7fd722e00640 1 -- 172.21.15.22:0/4184550670 wait complete. 2024-03-24T06:40:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-03-24T06:40:06.142727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191251,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1735,"num_read_kb":1040,"num_write":547,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181380,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":933,"ondisk_log_size":933,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389820,"kb_used_data":15424,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552004,"statfs":{"total":1151940427776,"available":1151541252096,"internally_reserved":0,"allocated":15794176,"data_stored":8969942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.977764"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807561+0000","last_change":"2024-03-24T06:36:56.773258+0000","last_active":"2024-03-24T06:40:00.807561+0000","last_peered":"2024-03-24T06:40:00.807561+0000","last_clean":"2024-03-24T06:40:00.807561+0000","last_became_active":"2024-03-24T06:36:56.772875+0000","last_became_peered":"2024-03-24T06:36:56.772875+0000","last_unstale":"2024-03-24T06:40:00.807561+0000","last_undegraded":"2024-03-24T06:40:00.807561+0000","last_fullsized":"2024-03-24T06:40:00.807561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:32.919412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808126+0000","last_change":"2024-03-24T06:36:56.772071+0000","last_active":"2024-03-24T06:40:00.808126+0000","last_peered":"2024-03-24T06:40:00.808126+0000","last_clean":"2024-03-24T06:40:00.808126+0000","last_became_active":"2024-03-24T06:36:56.771704+0000","last_became_peered":"2024-03-24T06:36:56.771704+0000","last_unstale":"2024-03-24T06:40:00.808126+0000","last_undegraded":"2024-03-24T06:40:00.808126+0000","last_fullsized":"2024-03-24T06:40:00.808126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:28:04.075842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807855+0000","last_change":"2024-03-24T06:36:56.776757+0000","last_active":"2024-03-24T06:40:00.807855+0000","last_peered":"2024-03-24T06:40:00.807855+0000","last_clean":"2024-03-24T06:40:00.807855+0000","last_became_active":"2024-03-24T06:36:56.776195+0000","last_became_peered":"2024-03-24T06:36:56.776195+0000","last_unstale":"2024-03-24T06:40:00.807855+0000","last_undegraded":"2024-03-24T06:40:00.807855+0000","last_fullsized":"2024-03-24T06:40:00.807855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:57:46.048721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697661+0000","last_change":"2024-03-24T06:36:56.757633+0000","last_active":"2024-03-24T06:40:00.697661+0000","last_peered":"2024-03-24T06:40:00.697661+0000","last_clean":"2024-03-24T06:40:00.697661+0000","last_became_active":"2024-03-24T06:36:56.757312+0000","last_became_peered":"2024-03-24T06:36:56.757312+0000","last_unstale":"2024-03-24T06:40:00.697661+0000","last_undegraded":"2024-03-24T06:40:00.697661+0000","last_fullsized":"2024-03-24T06:40:00.697661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:49.408773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281757+0000","last_change":"2024-03-24T06:36:56.762251+0000","last_active":"2024-03-24T06:40:00.281757+0000","last_peered":"2024-03-24T06:40:00.281757+0000","last_clean":"2024-03-24T06:40:00.281757+0000","last_became_active":"2024-03-24T06:36:56.761898+0000","last_became_peered":"2024-03-24T06:36:56.761898+0000","last_unstale":"2024-03-24T06:40:00.281757+0000","last_undegraded":"2024-03-24T06:40:00.281757+0000","last_fullsized":"2024-03-24T06:40:00.281757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:50.979374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123976+0000","last_change":"2024-03-24T06:36:56.749980+0000","last_active":"2024-03-24T06:40:00.123976+0000","last_peered":"2024-03-24T06:40:00.123976+0000","last_clean":"2024-03-24T06:40:00.123976+0000","last_became_active":"2024-03-24T06:36:56.749642+0000","last_became_peered":"2024-03-24T06:36:56.749642+0000","last_unstale":"2024-03-24T06:40:00.123976+0000","last_undegraded":"2024-03-24T06:40:00.123976+0000","last_fullsized":"2024-03-24T06:40:00.123976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:09:57.002257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652893+0000","last_change":"2024-03-24T06:36:56.762593+0000","last_active":"2024-03-24T06:40:00.652893+0000","last_peered":"2024-03-24T06:40:00.652893+0000","last_clean":"2024-03-24T06:40:00.652893+0000","last_became_active":"2024-03-24T06:36:56.762236+0000","last_became_peered":"2024-03-24T06:36:56.762236+0000","last_unstale":"2024-03-24T06:40:00.652893+0000","last_undegraded":"2024-03-24T06:40:00.652893+0000","last_fullsized":"2024-03-24T06:40:00.652893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:14.829814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123459+0000","last_change":"2024-03-24T06:36:56.764654+0000","last_active":"2024-03-24T06:40:00.123459+0000","last_peered":"2024-03-24T06:40:00.123459+0000","last_clean":"2024-03-24T06:40:00.123459+0000","last_became_active":"2024-03-24T06:36:56.764190+0000","last_became_peered":"2024-03-24T06:36:56.764190+0000","last_unstale":"2024-03-24T06:40:00.123459+0000","last_undegraded":"2024-03-24T06:40:00.123459+0000","last_fullsized":"2024-03-24T06:40:00.123459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:23:01.735180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688245+0000","last_change":"2024-03-24T06:36:56.747747+0000","last_active":"2024-03-24T06:40:00.688245+0000","last_peered":"2024-03-24T06:40:00.688245+0000","last_clean":"2024-03-24T06:40:00.688245+0000","last_became_active":"2024-03-24T06:36:56.747425+0000","last_became_peered":"2024-03-24T06:36:56.747425+0000","last_unstale":"2024-03-24T06:40:00.688245+0000","last_undegraded":"2024-03-24T06:40:00.688245+0000","last_fullsized":"2024-03-24T06:40:00.688245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:35.578444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124396+0000","last_change":"2024-03-24T06:36:56.750021+0000","last_active":"2024-03-24T06:40:00.124396+0000","last_peered":"2024-03-24T06:40:00.124396+0000","last_clean":"2024-03-24T06:40:00.124396+0000","last_became_active":"2024-03-24T06:36:56.749688+0000","last_became_peered":"2024-03-24T06:36:56.749688+0000","last_unstale":"2024-03-24T06:40:00.124396+0000","last_undegraded":"2024-03-24T06:40:00.124396+0000","last_fullsized":"2024-03-24T06:40:00.124396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:10:21.467889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653258+0000","last_change":"2024-03-24T06:36:56.753448+0000","last_active":"2024-03-24T06:40:00.653258+0000","last_peered":"2024-03-24T06:40:00.653258+0000","last_clean":"2024-03-24T06:40:00.653258+0000","last_became_active":"2024-03-24T06:36:56.753118+0000","last_became_peered":"2024-03-24T06:36:56.753118+0000","last_unstale":"2024-03-24T06:40:00.653258+0000","last_undegraded":"2024-03-24T06:40:00.653258+0000","last_fullsized":"2024-03-24T06:40:00.653258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:28.272047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697898+0000","last_change":"2024-03-24T06:36:56.757587+0000","last_active":"2024-03-24T06:40:00.697898+0000","last_peered":"2024-03-24T06:40:00.697898+0000","last_clean":"2024-03-24T06:40:00.697898+0000","last_became_active":"2024-03-24T06:36:56.757269+0000","last_became_peered":"2024-03-24T06:36:56.757269+0000","last_unstale":"2024-03-24T06:40:00.697898+0000","last_undegraded":"2024-03-24T06:40:00.697898+0000","last_fullsized":"2024-03-24T06:40:00.697898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:47.547680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680331+0000","last_change":"2024-03-24T06:36:56.761789+0000","last_active":"2024-03-24T06:40:00.680331+0000","last_peered":"2024-03-24T06:40:00.680331+0000","last_clean":"2024-03-24T06:40:00.680331+0000","last_became_active":"2024-03-24T06:36:56.761259+0000","last_became_peered":"2024-03-24T06:36:56.761259+0000","last_unstale":"2024-03-24T06:40:00.680331+0000","last_undegraded":"2024-03-24T06:40:00.680331+0000","last_fullsized":"2024-03-24T06:40:00.680331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:20:08.822480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807397+0000","last_change":"2024-03-24T06:36:56.778350+0000","last_active":"2024-03-24T06:40:00.807397+0000","last_peered":"2024-03-24T06:40:00.807397+0000","last_clean":"2024-03-24T06:40:00.807397+0000","last_became_active":"2024-03-24T06:36:56.777762+0000","last_became_peered":"2024-03-24T06:36:56.777762+0000","last_unstale":"2024-03-24T06:40:00.807397+0000","last_undegraded":"2024-03-24T06:40:00.807397+0000","last_fullsized":"2024-03-24T06:40:00.807397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:57.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807500+0000","last_change":"2024-03-24T06:36:56.777285+0000","last_active":"2024-03-24T06:40:00.807500+0000","last_peered":"2024-03-24T06:40:00.807500+0000","last_clean":"2024-03-24T06:40:00.807500+0000","last_became_active":"2024-03-24T06:36:56.774415+0000","last_became_peered":"2024-03-24T06:36:56.774415+0000","last_unstale":"2024-03-24T06:40:00.807500+0000","last_undegraded":"2024-03-24T06:40:00.807500+0000","last_fullsized":"2024-03-24T06:40:00.807500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.321106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687644+0000","last_change":"2024-03-24T06:36:56.770456+0000","last_active":"2024-03-24T06:40:00.687644+0000","last_peered":"2024-03-24T06:40:00.687644+0000","last_clean":"2024-03-24T06:40:00.687644+0000","last_became_active":"2024-03-24T06:36:56.769921+0000","last_became_peered":"2024-03-24T06:36:56.769921+0000","last_unstale":"2024-03-24T06:40:00.687644+0000","last_undegraded":"2024-03-24T06:40:00.687644+0000","last_fullsized":"2024-03-24T06:40:00.687644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:54.882212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807536+0000","last_change":"2024-03-24T06:36:56.769973+0000","last_active":"2024-03-24T06:40:00.807536+0000","last_peered":"2024-03-24T06:40:00.807536+0000","last_clean":"2024-03-24T06:40:00.807536+0000","last_became_active":"2024-03-24T06:36:56.769644+0000","last_became_peered":"2024-03-24T06:36:56.769644+0000","last_unstale":"2024-03-24T06:40:00.807536+0000","last_undegraded":"2024-03-24T06:40:00.807536+0000","last_fullsized":"2024-03-24T06:40:00.807536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:23:48.740564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.278167+0000","last_change":"2024-03-24T06:36:56.756072+0000","last_active":"2024-03-24T06:40:00.278167+0000","last_peered":"2024-03-24T06:40:00.278167+0000","last_clean":"2024-03-24T06:40:00.278167+0000","last_became_active":"2024-03-24T06:36:56.755543+0000","last_became_peered":"2024-03-24T06:36:56.755543+0000","last_unstale":"2024-03-24T06:40:00.278167+0000","last_undegraded":"2024-03-24T06:40:00.278167+0000","last_fullsized":"2024-03-24T06:40:00.278167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:06:59.258579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124626+0000","last_change":"2024-03-24T06:36:56.767190+0000","last_active":"2024-03-24T06:40:00.124626+0000","last_peered":"2024-03-24T06:40:00.124626+0000","last_clean":"2024-03-24T06:40:00.124626+0000","last_became_active":"2024-03-24T06:36:56.766879+0000","last_became_peered":"2024-03-24T06:36:56.766879+0000","last_unstale":"2024-03-24T06:40:00.124626+0000","last_undegraded":"2024-03-24T06:40:00.124626+0000","last_fullsized":"2024-03-24T06:40:00.124626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:55:11.203209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123859+0000","last_change":"2024-03-24T06:37:01.255420+0000","last_active":"2024-03-24T06:40:00.123859+0000","last_peered":"2024-03-24T06:40:00.123859+0000","last_clean":"2024-03-24T06:40:00.123859+0000","last_became_active":"2024-03-24T06:37:01.255134+0000","last_became_peered":"2024-03-24T06:37:01.255134+0000","last_unstale":"2024-03-24T06:40:00.123859+0000","last_undegraded":"2024-03-24T06:40:00.123859+0000","last_fullsized":"2024-03-24T06:40:00.123859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:11.039880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"94'14","reported_seq":197,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.651139+0000","last_change":"2024-03-24T06:36:55.748231+0000","last_active":"2024-03-24T06:40:00.651139+0000","last_peered":"2024-03-24T06:40:00.651139+0000","last_clean":"2024-03-24T06:40:00.651139+0000","last_became_active":"2024-03-24T06:36:55.747338+0000","last_became_peered":"2024-03-24T06:36:55.747338+0000","last_unstale":"2024-03-24T06:40:00.651139+0000","last_undegraded":"2024-03-24T06:40:00.651139+0000","last_fullsized":"2024-03-24T06:40:00.651139+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:57.629928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124742+0000","last_change":"2024-03-24T06:36:56.763664+0000","last_active":"2024-03-24T06:40:00.124742+0000","last_peered":"2024-03-24T06:40:00.124742+0000","last_clean":"2024-03-24T06:40:00.124742+0000","last_became_active":"2024-03-24T06:36:56.763310+0000","last_became_peered":"2024-03-24T06:36:56.763310+0000","last_unstale":"2024-03-24T06:40:00.124742+0000","last_undegraded":"2024-03-24T06:40:00.124742+0000","last_fullsized":"2024-03-24T06:40:00.124742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:46:45.221822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697277+0000","last_change":"2024-03-24T06:37:01.255264+0000","last_active":"2024-03-24T06:40:00.697277+0000","last_peered":"2024-03-24T06:40:00.697277+0000","last_clean":"2024-03-24T06:40:00.697277+0000","last_became_active":"2024-03-24T06:37:01.254873+0000","last_became_peered":"2024-03-24T06:37:01.254873+0000","last_unstale":"2024-03-24T06:40:00.697277+0000","last_undegraded":"2024-03-24T06:40:00.697277+0000","last_fullsized":"2024-03-24T06:40:00.697277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:17.060597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125229+0000","last_change":"2024-03-24T06:36:55.749074+0000","last_active":"2024-03-24T06:40:00.125229+0000","last_peered":"2024-03-24T06:40:00.125229+0000","last_clean":"2024-03-24T06:40:00.125229+0000","last_became_active":"2024-03-24T06:36:55.748631+0000","last_became_peered":"2024-03-24T06:36:55.748631+0000","last_unstale":"2024-03-24T06:40:00.125229+0000","last_undegraded":"2024-03-24T06:40:00.125229+0000","last_fullsized":"2024-03-24T06:40:00.125229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:40:50.968569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808502+0000","last_change":"2024-03-24T06:36:56.774299+0000","last_active":"2024-03-24T06:40:00.808502+0000","last_peered":"2024-03-24T06:40:00.808502+0000","last_clean":"2024-03-24T06:40:00.808502+0000","last_became_active":"2024-03-24T06:36:56.771328+0000","last_became_peered":"2024-03-24T06:36:56.771328+0000","last_unstale":"2024-03-24T06:40:00.808502+0000","last_undegraded":"2024-03-24T06:40:00.808502+0000","last_fullsized":"2024-03-24T06:40:00.808502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:38.446777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697506+0000","last_change":"2024-03-24T06:37:01.257287+0000","last_active":"2024-03-24T06:40:00.697506+0000","last_peered":"2024-03-24T06:40:00.697506+0000","last_clean":"2024-03-24T06:40:00.697506+0000","last_became_active":"2024-03-24T06:37:01.256211+0000","last_became_peered":"2024-03-24T06:37:01.256211+0000","last_unstale":"2024-03-24T06:40:00.697506+0000","last_undegraded":"2024-03-24T06:40:00.697506+0000","last_fullsized":"2024-03-24T06:40:00.697506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.296040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282154+0000","last_change":"2024-03-24T06:36:55.732853+0000","last_active":"2024-03-24T06:40:00.282154+0000","last_peered":"2024-03-24T06:40:00.282154+0000","last_clean":"2024-03-24T06:40:00.282154+0000","last_became_active":"2024-03-24T06:36:55.732514+0000","last_became_peered":"2024-03-24T06:36:55.732514+0000","last_unstale":"2024-03-24T06:40:00.282154+0000","last_undegraded":"2024-03-24T06:40:00.282154+0000","last_fullsized":"2024-03-24T06:40:00.282154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:48:38.989916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687945+0000","last_change":"2024-03-24T06:36:56.767305+0000","last_active":"2024-03-24T06:40:00.687945+0000","last_peered":"2024-03-24T06:40:00.687945+0000","last_clean":"2024-03-24T06:40:00.687945+0000","last_became_active":"2024-03-24T06:36:56.766657+0000","last_became_peered":"2024-03-24T06:36:56.766657+0000","last_unstale":"2024-03-24T06:40:00.687945+0000","last_undegraded":"2024-03-24T06:40:00.687945+0000","last_fullsized":"2024-03-24T06:40:00.687945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:34.015186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697300+0000","last_change":"2024-03-24T06:37:01.254266+0000","last_active":"2024-03-24T06:40:00.697300+0000","last_peered":"2024-03-24T06:40:00.697300+0000","last_clean":"2024-03-24T06:40:00.697300+0000","last_became_active":"2024-03-24T06:37:01.253795+0000","last_became_peered":"2024-03-24T06:37:01.253795+0000","last_unstale":"2024-03-24T06:40:00.697300+0000","last_undegraded":"2024-03-24T06:40:00.697300+0000","last_fullsized":"2024-03-24T06:40:00.697300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:31:51.085976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281817+0000","last_change":"2024-03-24T06:36:55.723707+0000","last_active":"2024-03-24T06:40:00.281817+0000","last_peered":"2024-03-24T06:40:00.281817+0000","last_clean":"2024-03-24T06:40:00.281817+0000","last_became_active":"2024-03-24T06:36:55.723394+0000","last_became_peered":"2024-03-24T06:36:55.723394+0000","last_unstale":"2024-03-24T06:40:00.281817+0000","last_undegraded":"2024-03-24T06:40:00.281817+0000","last_fullsized":"2024-03-24T06:40:00.281817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:10.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291734+0000","last_change":"2024-03-24T06:36:56.752322+0000","last_active":"2024-03-24T06:40:00.291734+0000","last_peered":"2024-03-24T06:40:00.291734+0000","last_clean":"2024-03-24T06:40:00.291734+0000","last_became_active":"2024-03-24T06:36:56.751979+0000","last_became_peered":"2024-03-24T06:36:56.751979+0000","last_unstale":"2024-03-24T06:40:00.291734+0000","last_undegraded":"2024-03-24T06:40:00.291734+0000","last_fullsized":"2024-03-24T06:40:00.291734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:44:47.601512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124183+0000","last_change":"2024-03-24T06:37:01.243448+0000","last_active":"2024-03-24T06:40:00.124183+0000","last_peered":"2024-03-24T06:40:00.124183+0000","last_clean":"2024-03-24T06:40:00.124183+0000","last_became_active":"2024-03-24T06:37:01.242984+0000","last_became_peered":"2024-03-24T06:37:01.242984+0000","last_unstale":"2024-03-24T06:40:00.124183+0000","last_undegraded":"2024-03-24T06:40:00.124183+0000","last_fullsized":"2024-03-24T06:40:00.124183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:37.645895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'7","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959636+0000","last_change":"2024-03-24T06:37:56.839629+0000","last_active":"2024-03-24T06:40:00.959636+0000","last_peered":"2024-03-24T06:40:00.959636+0000","last_clean":"2024-03-24T06:40:00.959636+0000","last_became_active":"2024-03-24T06:37:56.832648+0000","last_became_peered":"2024-03-24T06:37:56.832648+0000","last_unstale":"2024-03-24T06:40:00.959636+0000","last_undegraded":"2024-03-24T06:40:00.959636+0000","last_fullsized":"2024-03-24T06:40:00.959636+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:57.270140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281637+0000","last_change":"2024-03-24T06:36:56.747991+0000","last_active":"2024-03-24T06:40:00.281637+0000","last_peered":"2024-03-24T06:40:00.281637+0000","last_clean":"2024-03-24T06:40:00.281637+0000","last_became_active":"2024-03-24T06:36:56.747545+0000","last_became_peered":"2024-03-24T06:36:56.747545+0000","last_unstale":"2024-03-24T06:40:00.281637+0000","last_undegraded":"2024-03-24T06:40:00.281637+0000","last_fullsized":"2024-03-24T06:40:00.281637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:12.574869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124228+0000","last_change":"2024-03-24T06:37:01.261872+0000","last_active":"2024-03-24T06:40:00.124228+0000","last_peered":"2024-03-24T06:40:00.124228+0000","last_clean":"2024-03-24T06:40:00.124228+0000","last_became_active":"2024-03-24T06:37:01.261457+0000","last_became_peered":"2024-03-24T06:37:01.261457+0000","last_unstale":"2024-03-24T06:40:00.124228+0000","last_undegraded":"2024-03-24T06:40:00.124228+0000","last_fullsized":"2024-03-24T06:40:00.124228+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:37:17.029783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123925+0000","last_change":"2024-03-24T06:36:55.737308+0000","last_active":"2024-03-24T06:40:00.123925+0000","last_peered":"2024-03-24T06:40:00.123925+0000","last_clean":"2024-03-24T06:40:00.123925+0000","last_became_active":"2024-03-24T06:36:55.736984+0000","last_became_peered":"2024-03-24T06:36:55.736984+0000","last_unstale":"2024-03-24T06:40:00.123925+0000","last_undegraded":"2024-03-24T06:40:00.123925+0000","last_fullsized":"2024-03-24T06:40:00.123925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:41.010129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291895+0000","last_change":"2024-03-24T06:36:56.761030+0000","last_active":"2024-03-24T06:40:00.291895+0000","last_peered":"2024-03-24T06:40:00.291895+0000","last_clean":"2024-03-24T06:40:00.291895+0000","last_became_active":"2024-03-24T06:36:56.760598+0000","last_became_peered":"2024-03-24T06:36:56.760598+0000","last_unstale":"2024-03-24T06:40:00.291895+0000","last_undegraded":"2024-03-24T06:40:00.291895+0000","last_fullsized":"2024-03-24T06:40:00.291895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:57:34.813494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807182+0000","last_change":"2024-03-24T06:37:01.262633+0000","last_active":"2024-03-24T06:40:00.807182+0000","last_peered":"2024-03-24T06:40:00.807182+0000","last_clean":"2024-03-24T06:40:00.807182+0000","last_became_active":"2024-03-24T06:37:01.262376+0000","last_became_peered":"2024-03-24T06:37:01.262376+0000","last_unstale":"2024-03-24T06:40:00.807182+0000","last_undegraded":"2024-03-24T06:40:00.807182+0000","last_fullsized":"2024-03-24T06:40:00.807182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:02:57.853200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679807+0000","last_change":"2024-03-24T06:36:55.733426+0000","last_active":"2024-03-24T06:40:00.679807+0000","last_peered":"2024-03-24T06:40:00.679807+0000","last_clean":"2024-03-24T06:40:00.679807+0000","last_became_active":"2024-03-24T06:36:55.732728+0000","last_became_peered":"2024-03-24T06:36:55.732728+0000","last_unstale":"2024-03-24T06:40:00.679807+0000","last_undegraded":"2024-03-24T06:40:00.679807+0000","last_fullsized":"2024-03-24T06:40:00.679807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:33:23.635927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123726+0000","last_change":"2024-03-24T06:36:56.769587+0000","last_active":"2024-03-24T06:40:00.123726+0000","last_peered":"2024-03-24T06:40:00.123726+0000","last_clean":"2024-03-24T06:40:00.123726+0000","last_became_active":"2024-03-24T06:36:56.767105+0000","last_became_peered":"2024-03-24T06:36:56.767105+0000","last_unstale":"2024-03-24T06:40:00.123726+0000","last_undegraded":"2024-03-24T06:40:00.123726+0000","last_fullsized":"2024-03-24T06:40:00.123726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:07:14.701421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808088+0000","last_change":"2024-03-24T06:37:01.273004+0000","last_active":"2024-03-24T06:40:00.808088+0000","last_peered":"2024-03-24T06:40:00.808088+0000","last_clean":"2024-03-24T06:40:00.808088+0000","last_became_active":"2024-03-24T06:37:01.272132+0000","last_became_peered":"2024-03-24T06:37:01.272132+0000","last_unstale":"2024-03-24T06:40:00.808088+0000","last_undegraded":"2024-03-24T06:40:00.808088+0000","last_fullsized":"2024-03-24T06:40:00.808088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:35.465771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291185+0000","last_change":"2024-03-24T06:36:55.729117+0000","last_active":"2024-03-24T06:40:00.291185+0000","last_peered":"2024-03-24T06:40:00.291185+0000","last_clean":"2024-03-24T06:40:00.291185+0000","last_became_active":"2024-03-24T06:36:55.728071+0000","last_became_peered":"2024-03-24T06:36:55.728071+0000","last_unstale":"2024-03-24T06:40:00.291185+0000","last_undegraded":"2024-03-24T06:40:00.291185+0000","last_fullsized":"2024-03-24T06:40:00.291185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:01:53.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277299+0000","last_change":"2024-03-24T06:36:56.767950+0000","last_active":"2024-03-24T06:40:00.277299+0000","last_peered":"2024-03-24T06:40:00.277299+0000","last_clean":"2024-03-24T06:40:00.277299+0000","last_became_active":"2024-03-24T06:36:56.767628+0000","last_became_peered":"2024-03-24T06:36:56.767628+0000","last_unstale":"2024-03-24T06:40:00.277299+0000","last_undegraded":"2024-03-24T06:40:00.277299+0000","last_fullsized":"2024-03-24T06:40:00.277299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:01.026757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123735+0000","last_change":"2024-03-24T06:37:01.249674+0000","last_active":"2024-03-24T06:40:00.123735+0000","last_peered":"2024-03-24T06:40:00.123735+0000","last_clean":"2024-03-24T06:40:00.123735+0000","last_became_active":"2024-03-24T06:37:01.246895+0000","last_became_peered":"2024-03-24T06:37:01.246895+0000","last_unstale":"2024-03-24T06:40:00.123735+0000","last_undegraded":"2024-03-24T06:40:00.123735+0000","last_fullsized":"2024-03-24T06:40:00.123735+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:18:54.195755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":334,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.590647+0000","last_change":"2024-03-24T06:36:55.744398+0000","last_active":"2024-03-24T06:40:00.590647+0000","last_peered":"2024-03-24T06:40:00.590647+0000","last_clean":"2024-03-24T06:40:00.590647+0000","last_became_active":"2024-03-24T06:36:55.744095+0000","last_became_peered":"2024-03-24T06:36:55.744095+0000","last_unstale":"2024-03-24T06:40:00.590647+0000","last_undegraded":"2024-03-24T06:40:00.590647+0000","last_fullsized":"2024-03-24T06:40:00.590647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:10.699764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":296,"num_read_kb":296,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688179+0000","last_change":"2024-03-24T06:36:56.754662+0000","last_active":"2024-03-24T06:40:00.688179+0000","last_peered":"2024-03-24T06:40:00.688179+0000","last_clean":"2024-03-24T06:40:00.688179+0000","last_became_active":"2024-03-24T06:36:56.754355+0000","last_became_peered":"2024-03-24T06:36:56.754355+0000","last_unstale":"2024-03-24T06:40:00.688179+0000","last_undegraded":"2024-03-24T06:40:00.688179+0000","last_fullsized":"2024-03-24T06:40:00.688179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:49:15.169743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281341+0000","last_change":"2024-03-24T06:37:56.831960+0000","last_active":"2024-03-24T06:40:00.281341+0000","last_peered":"2024-03-24T06:40:00.281341+0000","last_clean":"2024-03-24T06:40:00.281341+0000","last_became_active":"2024-03-24T06:37:56.831524+0000","last_became_peered":"2024-03-24T06:37:56.831524+0000","last_unstale":"2024-03-24T06:40:00.281341+0000","last_undegraded":"2024-03-24T06:40:00.281341+0000","last_fullsized":"2024-03-24T06:40:00.281341+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:28.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653238+0000","last_change":"2024-03-24T06:36:55.730238+0000","last_active":"2024-03-24T06:40:00.653238+0000","last_peered":"2024-03-24T06:40:00.653238+0000","last_clean":"2024-03-24T06:40:00.653238+0000","last_became_active":"2024-03-24T06:36:55.729982+0000","last_became_peered":"2024-03-24T06:36:55.729982+0000","last_unstale":"2024-03-24T06:40:00.653238+0000","last_undegraded":"2024-03-24T06:40:00.653238+0000","last_fullsized":"2024-03-24T06:40:00.653238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:46.644635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808357+0000","last_change":"2024-03-24T06:36:56.775004+0000","last_active":"2024-03-24T06:40:00.808357+0000","last_peered":"2024-03-24T06:40:00.808357+0000","last_clean":"2024-03-24T06:40:00.808357+0000","last_became_active":"2024-03-24T06:36:56.773893+0000","last_became_peered":"2024-03-24T06:36:56.773893+0000","last_unstale":"2024-03-24T06:40:00.808357+0000","last_undegraded":"2024-03-24T06:40:00.808357+0000","last_fullsized":"2024-03-24T06:40:00.808357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:43.119302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697632+0000","last_change":"2024-03-24T06:37:57.056640+0000","last_active":"2024-03-24T06:40:00.697632+0000","last_peered":"2024-03-24T06:40:00.697632+0000","last_clean":"2024-03-24T06:40:00.697632+0000","last_became_active":"2024-03-24T06:37:57.056087+0000","last_became_peered":"2024-03-24T06:37:57.056087+0000","last_unstale":"2024-03-24T06:40:00.697632+0000","last_undegraded":"2024-03-24T06:40:00.697632+0000","last_fullsized":"2024-03-24T06:40:00.697632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:00:46.933123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291609+0000","last_change":"2024-03-24T06:36:55.744814+0000","last_active":"2024-03-24T06:40:00.291609+0000","last_peered":"2024-03-24T06:40:00.291609+0000","last_clean":"2024-03-24T06:40:00.291609+0000","last_became_active":"2024-03-24T06:36:55.744496+0000","last_became_peered":"2024-03-24T06:36:55.744496+0000","last_unstale":"2024-03-24T06:40:00.291609+0000","last_undegraded":"2024-03-24T06:40:00.291609+0000","last_fullsized":"2024-03-24T06:40:00.291609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:05.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.688042+0000","last_change":"2024-03-24T06:36:56.756478+0000","last_active":"2024-03-24T06:40:00.688042+0000","last_peered":"2024-03-24T06:40:00.688042+0000","last_clean":"2024-03-24T06:40:00.688042+0000","last_became_active":"2024-03-24T06:36:56.755901+0000","last_became_peered":"2024-03-24T06:36:56.755901+0000","last_unstale":"2024-03-24T06:40:00.688042+0000","last_undegraded":"2024-03-24T06:40:00.688042+0000","last_fullsized":"2024-03-24T06:40:00.688042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:17.058215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277348+0000","last_change":"2024-03-24T06:38:58.624331+0000","last_active":"2024-03-24T06:40:00.277348+0000","last_peered":"2024-03-24T06:40:00.277348+0000","last_clean":"2024-03-24T06:40:00.277348+0000","last_became_active":"2024-03-24T06:38:58.623562+0000","last_became_peered":"2024-03-24T06:38:58.623562+0000","last_unstale":"2024-03-24T06:40:00.277348+0000","last_undegraded":"2024-03-24T06:40:00.277348+0000","last_fullsized":"2024-03-24T06:40:00.277348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:19:32.355669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":377,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.959384+0000","last_change":"2024-03-24T06:36:55.729021+0000","last_active":"2024-03-24T06:40:00.959384+0000","last_peered":"2024-03-24T06:40:00.959384+0000","last_clean":"2024-03-24T06:40:00.959384+0000","last_became_active":"2024-03-24T06:36:55.728646+0000","last_became_peered":"2024-03-24T06:36:55.728646+0000","last_unstale":"2024-03-24T06:40:00.959384+0000","last_undegraded":"2024-03-24T06:40:00.959384+0000","last_fullsized":"2024-03-24T06:40:00.959384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:24.976883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67932,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":304,"num_read_kb":219,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281881+0000","last_change":"2024-03-24T06:36:56.750340+0000","last_active":"2024-03-24T06:40:00.281881+0000","last_peered":"2024-03-24T06:40:00.281881+0000","last_clean":"2024-03-24T06:40:00.281881+0000","last_became_active":"2024-03-24T06:36:56.749999+0000","last_became_peered":"2024-03-24T06:36:56.749999+0000","last_unstale":"2024-03-24T06:40:00.281881+0000","last_undegraded":"2024-03-24T06:40:00.281881+0000","last_fullsized":"2024-03-24T06:40:00.281881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:43.961671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291994+0000","last_change":"2024-03-24T06:37:01.259467+0000","last_active":"2024-03-24T06:40:00.291994+0000","last_peered":"2024-03-24T06:40:00.291994+0000","last_clean":"2024-03-24T06:40:00.291994+0000","last_became_active":"2024-03-24T06:37:01.259000+0000","last_became_peered":"2024-03-24T06:37:01.259000+0000","last_unstale":"2024-03-24T06:40:00.291994+0000","last_undegraded":"2024-03-24T06:40:00.291994+0000","last_fullsized":"2024-03-24T06:40:00.291994+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:08.069885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124117+0000","last_change":"2024-03-24T06:36:55.744144+0000","last_active":"2024-03-24T06:40:00.124117+0000","last_peered":"2024-03-24T06:40:00.124117+0000","last_clean":"2024-03-24T06:40:00.124117+0000","last_became_active":"2024-03-24T06:36:55.743701+0000","last_became_peered":"2024-03-24T06:36:55.743701+0000","last_unstale":"2024-03-24T06:40:00.124117+0000","last_undegraded":"2024-03-24T06:40:00.124117+0000","last_fullsized":"2024-03-24T06:40:00.124117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:58:07.843850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679797+0000","last_change":"2024-03-24T06:36:56.763056+0000","last_active":"2024-03-24T06:40:00.679797+0000","last_peered":"2024-03-24T06:40:00.679797+0000","last_clean":"2024-03-24T06:40:00.679797+0000","last_became_active":"2024-03-24T06:36:56.762736+0000","last_became_peered":"2024-03-24T06:36:56.762736+0000","last_unstale":"2024-03-24T06:40:00.679797+0000","last_undegraded":"2024-03-24T06:40:00.679797+0000","last_fullsized":"2024-03-24T06:40:00.679797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:50:15.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807095+0000","last_change":"2024-03-24T06:37:01.264632+0000","last_active":"2024-03-24T06:40:00.807095+0000","last_peered":"2024-03-24T06:40:00.807095+0000","last_clean":"2024-03-24T06:40:00.807095+0000","last_became_active":"2024-03-24T06:37:01.264269+0000","last_became_peered":"2024-03-24T06:37:01.264269+0000","last_unstale":"2024-03-24T06:40:00.807095+0000","last_undegraded":"2024-03-24T06:40:00.807095+0000","last_fullsized":"2024-03-24T06:40:00.807095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:29.167410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291586+0000","last_change":"2024-03-24T06:36:55.739624+0000","last_active":"2024-03-24T06:40:00.291586+0000","last_peered":"2024-03-24T06:40:00.291586+0000","last_clean":"2024-03-24T06:40:00.291586+0000","last_became_active":"2024-03-24T06:36:55.739374+0000","last_became_peered":"2024-03-24T06:36:55.739374+0000","last_unstale":"2024-03-24T06:40:00.291586+0000","last_undegraded":"2024-03-24T06:40:00.291586+0000","last_fullsized":"2024-03-24T06:40:00.291586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:27.552488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653539+0000","last_change":"2024-03-24T06:36:56.766568+0000","last_active":"2024-03-24T06:40:00.653539+0000","last_peered":"2024-03-24T06:40:00.653539+0000","last_clean":"2024-03-24T06:40:00.653539+0000","last_became_active":"2024-03-24T06:36:56.766190+0000","last_became_peered":"2024-03-24T06:36:56.766190+0000","last_unstale":"2024-03-24T06:40:00.653539+0000","last_undegraded":"2024-03-24T06:40:00.653539+0000","last_fullsized":"2024-03-24T06:40:00.653539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:07.355337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279380+0000","last_change":"2024-03-24T06:37:01.265421+0000","last_active":"2024-03-24T06:40:00.279380+0000","last_peered":"2024-03-24T06:40:00.279380+0000","last_clean":"2024-03-24T06:40:00.279380+0000","last_became_active":"2024-03-24T06:37:01.264987+0000","last_became_peered":"2024-03-24T06:37:01.264987+0000","last_unstale":"2024-03-24T06:40:00.279380+0000","last_undegraded":"2024-03-24T06:40:00.279380+0000","last_fullsized":"2024-03-24T06:40:00.279380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:33.638983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697530+0000","last_change":"2024-03-24T06:36:55.724726+0000","last_active":"2024-03-24T06:40:00.697530+0000","last_peered":"2024-03-24T06:40:00.697530+0000","last_clean":"2024-03-24T06:40:00.697530+0000","last_became_active":"2024-03-24T06:36:55.724387+0000","last_became_peered":"2024-03-24T06:36:55.724387+0000","last_unstale":"2024-03-24T06:40:00.697530+0000","last_undegraded":"2024-03-24T06:40:00.697530+0000","last_fullsized":"2024-03-24T06:40:00.697530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:22.546274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":151,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652957+0000","last_change":"2024-03-24T06:36:56.751377+0000","last_active":"2024-03-24T06:40:00.652957+0000","last_peered":"2024-03-24T06:40:00.652957+0000","last_clean":"2024-03-24T06:40:00.652957+0000","last_became_active":"2024-03-24T06:36:56.751056+0000","last_became_peered":"2024-03-24T06:36:56.751056+0000","last_unstale":"2024-03-24T06:40:00.652957+0000","last_undegraded":"2024-03-24T06:40:00.652957+0000","last_fullsized":"2024-03-24T06:40:00.652957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:39.243691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681557+0000","last_change":"2024-03-24T06:37:01.268017+0000","last_active":"2024-03-24T06:40:00.681557+0000","last_peered":"2024-03-24T06:40:00.681557+0000","last_clean":"2024-03-24T06:40:00.681557+0000","last_became_active":"2024-03-24T06:37:01.266766+0000","last_became_peered":"2024-03-24T06:37:01.266766+0000","last_unstale":"2024-03-24T06:40:00.681557+0000","last_undegraded":"2024-03-24T06:40:00.681557+0000","last_fullsized":"2024-03-24T06:40:00.681557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:44.718206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281346+0000","last_change":"2024-03-24T06:36:55.742999+0000","last_active":"2024-03-24T06:40:00.281346+0000","last_peered":"2024-03-24T06:40:00.281346+0000","last_clean":"2024-03-24T06:40:00.281346+0000","last_became_active":"2024-03-24T06:36:55.742718+0000","last_became_peered":"2024-03-24T06:36:55.742718+0000","last_unstale":"2024-03-24T06:40:00.281346+0000","last_undegraded":"2024-03-24T06:40:00.281346+0000","last_fullsized":"2024-03-24T06:40:00.281346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:11.819825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291241+0000","last_change":"2024-03-24T06:36:56.753064+0000","last_active":"2024-03-24T06:40:00.291241+0000","last_peered":"2024-03-24T06:40:00.291241+0000","last_clean":"2024-03-24T06:40:00.291241+0000","last_became_active":"2024-03-24T06:36:56.752753+0000","last_became_peered":"2024-03-24T06:36:56.752753+0000","last_unstale":"2024-03-24T06:40:00.291241+0000","last_undegraded":"2024-03-24T06:40:00.291241+0000","last_fullsized":"2024-03-24T06:40:00.291241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:46:27.939947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291201+0000","last_change":"2024-03-24T06:37:01.260345+0000","last_active":"2024-03-24T06:40:00.291201+0000","last_peered":"2024-03-24T06:40:00.291201+0000","last_clean":"2024-03-24T06:40:00.291201+0000","last_became_active":"2024-03-24T06:37:01.259967+0000","last_became_peered":"2024-03-24T06:37:01.259967+0000","last_unstale":"2024-03-24T06:40:00.291201+0000","last_undegraded":"2024-03-24T06:40:00.291201+0000","last_fullsized":"2024-03-24T06:40:00.291201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:13:38.873847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807189+0000","last_change":"2024-03-24T06:37:56.837415+0000","last_active":"2024-03-24T06:40:00.807189+0000","last_peered":"2024-03-24T06:40:00.807189+0000","last_clean":"2024-03-24T06:40:00.807189+0000","last_became_active":"2024-03-24T06:37:56.832240+0000","last_became_peered":"2024-03-24T06:37:56.832240+0000","last_unstale":"2024-03-24T06:40:00.807189+0000","last_undegraded":"2024-03-24T06:40:00.807189+0000","last_fullsized":"2024-03-24T06:40:00.807189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:44.867297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807274+0000","last_change":"2024-03-24T06:37:56.826934+0000","last_active":"2024-03-24T06:40:00.807274+0000","last_peered":"2024-03-24T06:40:00.807274+0000","last_clean":"2024-03-24T06:40:00.807274+0000","last_became_active":"2024-03-24T06:37:56.826574+0000","last_became_peered":"2024-03-24T06:37:56.826574+0000","last_unstale":"2024-03-24T06:40:00.807274+0000","last_undegraded":"2024-03-24T06:40:00.807274+0000","last_fullsized":"2024-03-24T06:40:00.807274+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:14:52.406936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281632+0000","last_change":"2024-03-24T06:37:01.254400+0000","last_active":"2024-03-24T06:40:00.281632+0000","last_peered":"2024-03-24T06:40:00.281632+0000","last_clean":"2024-03-24T06:40:00.281632+0000","last_became_active":"2024-03-24T06:37:01.253902+0000","last_became_peered":"2024-03-24T06:37:01.253902+0000","last_unstale":"2024-03-24T06:40:00.281632+0000","last_undegraded":"2024-03-24T06:40:00.281632+0000","last_fullsized":"2024-03-24T06:40:00.281632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:39:05.194771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.277730+0000","last_change":"2024-03-24T06:37:56.832165+0000","last_active":"2024-03-24T06:40:00.277730+0000","last_peered":"2024-03-24T06:40:00.277730+0000","last_clean":"2024-03-24T06:40:00.277730+0000","last_became_active":"2024-03-24T06:37:56.831670+0000","last_became_peered":"2024-03-24T06:37:56.831670+0000","last_unstale":"2024-03-24T06:40:00.277730+0000","last_undegraded":"2024-03-24T06:40:00.277730+0000","last_fullsized":"2024-03-24T06:40:00.277730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:22:56.201188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124107+0000","last_change":"2024-03-24T06:36:56.768066+0000","last_active":"2024-03-24T06:40:00.124107+0000","last_peered":"2024-03-24T06:40:00.124107+0000","last_clean":"2024-03-24T06:40:00.124107+0000","last_became_active":"2024-03-24T06:36:56.767839+0000","last_became_peered":"2024-03-24T06:36:56.767839+0000","last_unstale":"2024-03-24T06:40:00.124107+0000","last_undegraded":"2024-03-24T06:40:00.124107+0000","last_fullsized":"2024-03-24T06:40:00.124107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:07:03.287096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680650+0000","last_change":"2024-03-24T06:37:01.266086+0000","last_active":"2024-03-24T06:40:00.680650+0000","last_peered":"2024-03-24T06:40:00.680650+0000","last_clean":"2024-03-24T06:40:00.680650+0000","last_became_active":"2024-03-24T06:37:01.265602+0000","last_became_peered":"2024-03-24T06:37:01.265602+0000","last_unstale":"2024-03-24T06:40:00.680650+0000","last_undegraded":"2024-03-24T06:40:00.680650+0000","last_fullsized":"2024-03-24T06:40:00.680650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:52.071248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"94'7","reported_seq":189,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687830+0000","last_change":"2024-03-24T06:36:55.743287+0000","last_active":"2024-03-24T06:40:00.687830+0000","last_peered":"2024-03-24T06:40:00.687830+0000","last_clean":"2024-03-24T06:40:00.687830+0000","last_became_active":"2024-03-24T06:36:55.742944+0000","last_became_peered":"2024-03-24T06:36:55.742944+0000","last_unstale":"2024-03-24T06:40:00.687830+0000","last_undegraded":"2024-03-24T06:40:00.687830+0000","last_fullsized":"2024-03-24T06:40:00.687830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:51.313800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123953+0000","last_change":"2024-03-24T06:36:56.770322+0000","last_active":"2024-03-24T06:40:00.123953+0000","last_peered":"2024-03-24T06:40:00.123953+0000","last_clean":"2024-03-24T06:40:00.123953+0000","last_became_active":"2024-03-24T06:36:56.769938+0000","last_became_peered":"2024-03-24T06:36:56.769938+0000","last_unstale":"2024-03-24T06:40:00.123953+0000","last_undegraded":"2024-03-24T06:40:00.123953+0000","last_fullsized":"2024-03-24T06:40:00.123953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:42:59.252366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681686+0000","last_change":"2024-03-24T06:37:01.269982+0000","last_active":"2024-03-24T06:40:00.681686+0000","last_peered":"2024-03-24T06:40:00.681686+0000","last_clean":"2024-03-24T06:40:00.681686+0000","last_became_active":"2024-03-24T06:37:01.269561+0000","last_became_peered":"2024-03-24T06:37:01.269561+0000","last_unstale":"2024-03-24T06:40:00.681686+0000","last_undegraded":"2024-03-24T06:40:00.681686+0000","last_fullsized":"2024-03-24T06:40:00.681686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:20:26.497887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.292031+0000","last_change":"2024-03-24T06:36:56.751863+0000","last_active":"2024-03-24T06:40:00.292031+0000","last_peered":"2024-03-24T06:40:00.292031+0000","last_clean":"2024-03-24T06:40:00.292031+0000","last_became_active":"2024-03-24T06:36:56.751155+0000","last_became_peered":"2024-03-24T06:36:56.751155+0000","last_unstale":"2024-03-24T06:40:00.292031+0000","last_undegraded":"2024-03-24T06:40:00.292031+0000","last_fullsized":"2024-03-24T06:40:00.292031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:00:46.727390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":187,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679720+0000","last_change":"2024-03-24T06:36:55.750707+0000","last_active":"2024-03-24T06:40:00.679720+0000","last_peered":"2024-03-24T06:40:00.679720+0000","last_clean":"2024-03-24T06:40:00.679720+0000","last_became_active":"2024-03-24T06:36:55.750035+0000","last_became_peered":"2024-03-24T06:36:55.750035+0000","last_unstale":"2024-03-24T06:40:00.679720+0000","last_undegraded":"2024-03-24T06:40:00.679720+0000","last_fullsized":"2024-03-24T06:40:00.679720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:59:29.529033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653101+0000","last_change":"2024-03-24T06:37:01.256757+0000","last_active":"2024-03-24T06:40:00.653101+0000","last_peered":"2024-03-24T06:40:00.653101+0000","last_clean":"2024-03-24T06:40:00.653101+0000","last_became_active":"2024-03-24T06:37:01.256276+0000","last_became_peered":"2024-03-24T06:37:01.256276+0000","last_unstale":"2024-03-24T06:40:00.653101+0000","last_undegraded":"2024-03-24T06:40:00.653101+0000","last_fullsized":"2024-03-24T06:40:00.653101+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:52.878087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124031+0000","last_change":"2024-03-24T06:36:56.756637+0000","last_active":"2024-03-24T06:40:00.124031+0000","last_peered":"2024-03-24T06:40:00.124031+0000","last_clean":"2024-03-24T06:40:00.124031+0000","last_became_active":"2024-03-24T06:36:56.756142+0000","last_became_peered":"2024-03-24T06:36:56.756142+0000","last_unstale":"2024-03-24T06:40:00.124031+0000","last_undegraded":"2024-03-24T06:40:00.124031+0000","last_fullsized":"2024-03-24T06:40:00.124031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:57:47.707845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697561+0000","last_change":"2024-03-24T06:36:55.741745+0000","last_active":"2024-03-24T06:40:00.697561+0000","last_peered":"2024-03-24T06:40:00.697561+0000","last_clean":"2024-03-24T06:40:00.697561+0000","last_became_active":"2024-03-24T06:36:55.741343+0000","last_became_peered":"2024-03-24T06:36:55.741343+0000","last_unstale":"2024-03-24T06:40:00.697561+0000","last_undegraded":"2024-03-24T06:40:00.697561+0000","last_fullsized":"2024-03-24T06:40:00.697561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:32.311473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652830+0000","last_change":"2024-03-24T06:35:35.494271+0000","last_active":"2024-03-24T06:40:00.652830+0000","last_peered":"2024-03-24T06:40:00.652830+0000","last_clean":"2024-03-24T06:40:00.652830+0000","last_became_active":"2024-03-24T06:35:35.493773+0000","last_became_peered":"2024-03-24T06:35:35.493773+0000","last_unstale":"2024-03-24T06:40:00.652830+0000","last_undegraded":"2024-03-24T06:40:00.652830+0000","last_fullsized":"2024-03-24T06:40:00.652830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:31:39.794369+0000","last_clean_scrub_stamp":"2024-03-24T06:31:39.794369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:40:12.392546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291596+0000","last_change":"2024-03-24T06:37:01.259429+0000","last_active":"2024-03-24T06:40:00.291596+0000","last_peered":"2024-03-24T06:40:00.291596+0000","last_clean":"2024-03-24T06:40:00.291596+0000","last_became_active":"2024-03-24T06:37:01.258998+0000","last_became_peered":"2024-03-24T06:37:01.258998+0000","last_unstale":"2024-03-24T06:40:00.291596+0000","last_undegraded":"2024-03-24T06:40:00.291596+0000","last_fullsized":"2024-03-24T06:40:00.291596+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:03:29.891832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.679511+0000","last_change":"2024-03-24T06:37:56.831122+0000","last_active":"2024-03-24T06:40:00.679511+0000","last_peered":"2024-03-24T06:40:00.679511+0000","last_clean":"2024-03-24T06:40:00.679511+0000","last_became_active":"2024-03-24T06:37:56.830642+0000","last_became_peered":"2024-03-24T06:37:56.830642+0000","last_unstale":"2024-03-24T06:40:00.679511+0000","last_undegraded":"2024-03-24T06:40:00.679511+0000","last_fullsized":"2024-03-24T06:40:00.679511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:51.204542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.652901+0000","last_change":"2024-03-24T06:36:55.731312+0000","last_active":"2024-03-24T06:40:00.652901+0000","last_peered":"2024-03-24T06:40:00.652901+0000","last_clean":"2024-03-24T06:40:00.652901+0000","last_became_active":"2024-03-24T06:36:55.731024+0000","last_became_peered":"2024-03-24T06:36:55.731024+0000","last_unstale":"2024-03-24T06:40:00.652901+0000","last_undegraded":"2024-03-24T06:40:00.652901+0000","last_fullsized":"2024-03-24T06:40:00.652901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:16:05.939563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806953+0000","last_change":"2024-03-24T06:37:01.257104+0000","last_active":"2024-03-24T06:40:00.806953+0000","last_peered":"2024-03-24T06:40:00.806953+0000","last_clean":"2024-03-24T06:40:00.806953+0000","last_became_active":"2024-03-24T06:37:01.256642+0000","last_became_peered":"2024-03-24T06:37:01.256642+0000","last_unstale":"2024-03-24T06:40:00.806953+0000","last_undegraded":"2024-03-24T06:40:00.806953+0000","last_fullsized":"2024-03-24T06:40:00.806953+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:03.338547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.282064+0000","last_change":"2024-03-24T06:36:55.744367+0000","last_active":"2024-03-24T06:40:00.282064+0000","last_peered":"2024-03-24T06:40:00.282064+0000","last_clean":"2024-03-24T06:40:00.282064+0000","last_became_active":"2024-03-24T06:36:55.744034+0000","last_became_peered":"2024-03-24T06:36:55.744034+0000","last_unstale":"2024-03-24T06:40:00.282064+0000","last_undegraded":"2024-03-24T06:40:00.282064+0000","last_fullsized":"2024-03-24T06:40:00.282064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:15.328740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680823+0000","last_change":"2024-03-24T06:36:56.774057+0000","last_active":"2024-03-24T06:40:00.680823+0000","last_peered":"2024-03-24T06:40:00.680823+0000","last_clean":"2024-03-24T06:40:00.680823+0000","last_became_active":"2024-03-24T06:36:56.771687+0000","last_became_peered":"2024-03-24T06:36:56.771687+0000","last_unstale":"2024-03-24T06:40:00.680823+0000","last_undegraded":"2024-03-24T06:40:00.680823+0000","last_fullsized":"2024-03-24T06:40:00.680823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:41.201021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123597+0000","last_change":"2024-03-24T06:37:01.260477+0000","last_active":"2024-03-24T06:40:00.123597+0000","last_peered":"2024-03-24T06:40:00.123597+0000","last_clean":"2024-03-24T06:40:00.123597+0000","last_became_active":"2024-03-24T06:37:01.260049+0000","last_became_peered":"2024-03-24T06:37:01.260049+0000","last_unstale":"2024-03-24T06:40:00.123597+0000","last_undegraded":"2024-03-24T06:40:00.123597+0000","last_fullsized":"2024-03-24T06:40:00.123597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:22:17.165016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.457728+0000","last_change":"2024-03-24T06:36:55.746499+0000","last_active":"2024-03-24T06:40:00.457728+0000","last_peered":"2024-03-24T06:40:00.457728+0000","last_clean":"2024-03-24T06:40:00.457728+0000","last_became_active":"2024-03-24T06:36:55.746136+0000","last_became_peered":"2024-03-24T06:36:55.746136+0000","last_unstale":"2024-03-24T06:40:00.457728+0000","last_undegraded":"2024-03-24T06:40:00.457728+0000","last_fullsized":"2024-03-24T06:40:00.457728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:04.735329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":85,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124952+0000","last_change":"2024-03-24T06:36:56.762261+0000","last_active":"2024-03-24T06:40:00.124952+0000","last_peered":"2024-03-24T06:40:00.124952+0000","last_clean":"2024-03-24T06:40:00.124952+0000","last_became_active":"2024-03-24T06:36:56.761870+0000","last_became_peered":"2024-03-24T06:36:56.761870+0000","last_unstale":"2024-03-24T06:40:00.124952+0000","last_undegraded":"2024-03-24T06:40:00.124952+0000","last_fullsized":"2024-03-24T06:40:00.124952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:58:46.904450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123851+0000","last_change":"2024-03-24T06:37:01.243719+0000","last_active":"2024-03-24T06:40:00.123851+0000","last_peered":"2024-03-24T06:40:00.123851+0000","last_clean":"2024-03-24T06:40:00.123851+0000","last_became_active":"2024-03-24T06:37:01.243310+0000","last_became_peered":"2024-03-24T06:37:01.243310+0000","last_unstale":"2024-03-24T06:40:00.123851+0000","last_undegraded":"2024-03-24T06:40:00.123851+0000","last_fullsized":"2024-03-24T06:40:00.123851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:44.827197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697902+0000","last_change":"2024-03-24T06:36:55.733705+0000","last_active":"2024-03-24T06:40:00.697902+0000","last_peered":"2024-03-24T06:40:00.697902+0000","last_clean":"2024-03-24T06:40:00.697902+0000","last_became_active":"2024-03-24T06:36:55.733376+0000","last_became_peered":"2024-03-24T06:36:55.733376+0000","last_unstale":"2024-03-24T06:40:00.697902+0000","last_undegraded":"2024-03-24T06:40:00.697902+0000","last_fullsized":"2024-03-24T06:40:00.697902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:09.063530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.808204+0000","last_change":"2024-03-24T06:36:56.775610+0000","last_active":"2024-03-24T06:40:00.808204+0000","last_peered":"2024-03-24T06:40:00.808204+0000","last_clean":"2024-03-24T06:40:00.808204+0000","last_became_active":"2024-03-24T06:36:56.774841+0000","last_became_peered":"2024-03-24T06:36:56.774841+0000","last_unstale":"2024-03-24T06:40:00.808204+0000","last_undegraded":"2024-03-24T06:40:00.808204+0000","last_fullsized":"2024-03-24T06:40:00.808204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:21:33.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697694+0000","last_change":"2024-03-24T06:37:57.056479+0000","last_active":"2024-03-24T06:40:00.697694+0000","last_peered":"2024-03-24T06:40:00.697694+0000","last_clean":"2024-03-24T06:40:00.697694+0000","last_became_active":"2024-03-24T06:37:57.055993+0000","last_became_peered":"2024-03-24T06:37:57.055993+0000","last_unstale":"2024-03-24T06:40:00.697694+0000","last_undegraded":"2024-03-24T06:40:00.697694+0000","last_fullsized":"2024-03-24T06:40:00.697694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:06:12.473494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687639+0000","last_change":"2024-03-24T06:36:55.734671+0000","last_active":"2024-03-24T06:40:00.687639+0000","last_peered":"2024-03-24T06:40:00.687639+0000","last_clean":"2024-03-24T06:40:00.687639+0000","last_became_active":"2024-03-24T06:36:55.734383+0000","last_became_peered":"2024-03-24T06:36:55.734383+0000","last_unstale":"2024-03-24T06:40:00.687639+0000","last_undegraded":"2024-03-24T06:40:00.687639+0000","last_fullsized":"2024-03-24T06:40:00.687639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:40.116433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281408+0000","last_change":"2024-03-24T06:36:56.751259+0000","last_active":"2024-03-24T06:40:00.281408+0000","last_peered":"2024-03-24T06:40:00.281408+0000","last_clean":"2024-03-24T06:40:00.281408+0000","last_became_active":"2024-03-24T06:36:56.749462+0000","last_became_peered":"2024-03-24T06:36:56.749462+0000","last_unstale":"2024-03-24T06:40:00.281408+0000","last_undegraded":"2024-03-24T06:40:00.281408+0000","last_fullsized":"2024-03-24T06:40:00.281408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:40:17.142774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.279502+0000","last_change":"2024-03-24T06:37:01.249026+0000","last_active":"2024-03-24T06:40:00.279502+0000","last_peered":"2024-03-24T06:40:00.279502+0000","last_clean":"2024-03-24T06:40:00.279502+0000","last_became_active":"2024-03-24T06:37:01.248574+0000","last_became_peered":"2024-03-24T06:37:01.248574+0000","last_unstale":"2024-03-24T06:40:00.279502+0000","last_undegraded":"2024-03-24T06:40:00.279502+0000","last_fullsized":"2024-03-24T06:40:00.279502+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:08.969648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123941+0000","last_change":"2024-03-24T06:36:55.749969+0000","last_active":"2024-03-24T06:40:00.123941+0000","last_peered":"2024-03-24T06:40:00.123941+0000","last_clean":"2024-03-24T06:40:00.123941+0000","last_became_active":"2024-03-24T06:36:55.748796+0000","last_became_peered":"2024-03-24T06:36:55.748796+0000","last_unstale":"2024-03-24T06:40:00.123941+0000","last_undegraded":"2024-03-24T06:40:00.123941+0000","last_fullsized":"2024-03-24T06:40:00.123941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:27:01.212330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806870+0000","last_change":"2024-03-24T06:36:56.758527+0000","last_active":"2024-03-24T06:40:00.806870+0000","last_peered":"2024-03-24T06:40:00.806870+0000","last_clean":"2024-03-24T06:40:00.806870+0000","last_became_active":"2024-03-24T06:36:56.758224+0000","last_became_peered":"2024-03-24T06:36:56.758224+0000","last_unstale":"2024-03-24T06:40:00.806870+0000","last_undegraded":"2024-03-24T06:40:00.806870+0000","last_fullsized":"2024-03-24T06:40:00.806870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:53:59.538964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697505+0000","last_change":"2024-03-24T06:37:01.256705+0000","last_active":"2024-03-24T06:40:00.697505+0000","last_peered":"2024-03-24T06:40:00.697505+0000","last_clean":"2024-03-24T06:40:00.697505+0000","last_became_active":"2024-03-24T06:37:01.256159+0000","last_became_peered":"2024-03-24T06:37:01.256159+0000","last_unstale":"2024-03-24T06:40:00.697505+0000","last_undegraded":"2024-03-24T06:40:00.697505+0000","last_fullsized":"2024-03-24T06:40:00.697505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:30:00.680169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123999+0000","last_change":"2024-03-24T06:36:55.742002+0000","last_active":"2024-03-24T06:40:00.123999+0000","last_peered":"2024-03-24T06:40:00.123999+0000","last_clean":"2024-03-24T06:40:00.123999+0000","last_became_active":"2024-03-24T06:36:55.741618+0000","last_became_peered":"2024-03-24T06:36:55.741618+0000","last_unstale":"2024-03-24T06:40:00.123999+0000","last_undegraded":"2024-03-24T06:40:00.123999+0000","last_fullsized":"2024-03-24T06:40:00.123999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:25.234683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125506+0000","last_change":"2024-03-24T06:36:56.771974+0000","last_active":"2024-03-24T06:40:00.125506+0000","last_peered":"2024-03-24T06:40:00.125506+0000","last_clean":"2024-03-24T06:40:00.125506+0000","last_became_active":"2024-03-24T06:36:56.771619+0000","last_became_peered":"2024-03-24T06:36:56.771619+0000","last_unstale":"2024-03-24T06:40:00.125506+0000","last_undegraded":"2024-03-24T06:40:00.125506+0000","last_fullsized":"2024-03-24T06:40:00.125506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:36.571250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.697583+0000","last_change":"2024-03-24T06:37:01.249151+0000","last_active":"2024-03-24T06:40:00.697583+0000","last_peered":"2024-03-24T06:40:00.697583+0000","last_clean":"2024-03-24T06:40:00.697583+0000","last_became_active":"2024-03-24T06:37:01.248593+0000","last_became_peered":"2024-03-24T06:37:01.248593+0000","last_unstale":"2024-03-24T06:40:00.697583+0000","last_undegraded":"2024-03-24T06:40:00.697583+0000","last_fullsized":"2024-03-24T06:40:00.697583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:21:58.983888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.281619+0000","last_change":"2024-03-24T06:36:55.733021+0000","last_active":"2024-03-24T06:40:00.281619+0000","last_peered":"2024-03-24T06:40:00.281619+0000","last_clean":"2024-03-24T06:40:00.281619+0000","last_became_active":"2024-03-24T06:36:55.732693+0000","last_became_peered":"2024-03-24T06:36:55.732693+0000","last_unstale":"2024-03-24T06:40:00.281619+0000","last_undegraded":"2024-03-24T06:40:00.281619+0000","last_fullsized":"2024-03-24T06:40:00.281619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:31.367091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653331+0000","last_change":"2024-03-24T06:36:56.766620+0000","last_active":"2024-03-24T06:40:00.653331+0000","last_peered":"2024-03-24T06:40:00.653331+0000","last_clean":"2024-03-24T06:40:00.653331+0000","last_became_active":"2024-03-24T06:36:56.766271+0000","last_became_peered":"2024-03-24T06:36:56.766271+0000","last_unstale":"2024-03-24T06:40:00.653331+0000","last_undegraded":"2024-03-24T06:40:00.653331+0000","last_fullsized":"2024-03-24T06:40:00.653331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:11:02.588386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687653+0000","last_change":"2024-03-24T06:37:01.264725+0000","last_active":"2024-03-24T06:40:00.687653+0000","last_peered":"2024-03-24T06:40:00.687653+0000","last_clean":"2024-03-24T06:40:00.687653+0000","last_became_active":"2024-03-24T06:37:01.264220+0000","last_became_peered":"2024-03-24T06:37:01.264220+0000","last_unstale":"2024-03-24T06:40:00.687653+0000","last_undegraded":"2024-03-24T06:40:00.687653+0000","last_fullsized":"2024-03-24T06:40:00.687653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:36:13.232479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124472+0000","last_change":"2024-03-24T06:36:55.737344+0000","last_active":"2024-03-24T06:40:00.124472+0000","last_peered":"2024-03-24T06:40:00.124472+0000","last_clean":"2024-03-24T06:40:00.124472+0000","last_became_active":"2024-03-24T06:36:55.736986+0000","last_became_peered":"2024-03-24T06:36:55.736986+0000","last_unstale":"2024-03-24T06:40:00.124472+0000","last_undegraded":"2024-03-24T06:40:00.124472+0000","last_fullsized":"2024-03-24T06:40:00.124472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:35.177273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":150,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653414+0000","last_change":"2024-03-24T06:36:56.753452+0000","last_active":"2024-03-24T06:40:00.653414+0000","last_peered":"2024-03-24T06:40:00.653414+0000","last_clean":"2024-03-24T06:40:00.653414+0000","last_became_active":"2024-03-24T06:36:56.753076+0000","last_became_peered":"2024-03-24T06:36:56.753076+0000","last_unstale":"2024-03-24T06:40:00.653414+0000","last_undegraded":"2024-03-24T06:40:00.653414+0000","last_fullsized":"2024-03-24T06:40:00.653414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:59:24.304463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807599+0000","last_change":"2024-03-24T06:37:01.263675+0000","last_active":"2024-03-24T06:40:00.807599+0000","last_peered":"2024-03-24T06:40:00.807599+0000","last_clean":"2024-03-24T06:40:00.807599+0000","last_became_active":"2024-03-24T06:37:01.263249+0000","last_became_peered":"2024-03-24T06:37:01.263249+0000","last_unstale":"2024-03-24T06:40:00.807599+0000","last_undegraded":"2024-03-24T06:40:00.807599+0000","last_fullsized":"2024-03-24T06:40:00.807599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.931085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.454352+0000","last_change":"2024-03-24T06:36:55.738122+0000","last_active":"2024-03-24T06:40:00.454352+0000","last_peered":"2024-03-24T06:40:00.454352+0000","last_clean":"2024-03-24T06:40:00.454352+0000","last_became_active":"2024-03-24T06:36:55.737724+0000","last_became_peered":"2024-03-24T06:36:55.737724+0000","last_unstale":"2024-03-24T06:40:00.454352+0000","last_undegraded":"2024-03-24T06:40:00.454352+0000","last_fullsized":"2024-03-24T06:40:00.454352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:51:19.063982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653114+0000","last_change":"2024-03-24T06:36:56.761599+0000","last_active":"2024-03-24T06:40:00.653114+0000","last_peered":"2024-03-24T06:40:00.653114+0000","last_clean":"2024-03-24T06:40:00.653114+0000","last_became_active":"2024-03-24T06:36:56.761264+0000","last_became_peered":"2024-03-24T06:36:56.761264+0000","last_unstale":"2024-03-24T06:40:00.653114+0000","last_undegraded":"2024-03-24T06:40:00.653114+0000","last_fullsized":"2024-03-24T06:40:00.653114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:40.887372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.290992+0000","last_change":"2024-03-24T06:37:01.244572+0000","last_active":"2024-03-24T06:40:00.290992+0000","last_peered":"2024-03-24T06:40:00.290992+0000","last_clean":"2024-03-24T06:40:00.290992+0000","last_became_active":"2024-03-24T06:37:01.244082+0000","last_became_peered":"2024-03-24T06:37:01.244082+0000","last_unstale":"2024-03-24T06:40:00.290992+0000","last_undegraded":"2024-03-24T06:40:00.290992+0000","last_fullsized":"2024-03-24T06:40:00.290992+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:37:00.193084+0000","last_clean_scrub_stamp":"2024-03-24T06:37:00.193084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:31:12.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.681423+0000","last_change":"2024-03-24T06:36:55.744769+0000","last_active":"2024-03-24T06:40:00.681423+0000","last_peered":"2024-03-24T06:40:00.681423+0000","last_clean":"2024-03-24T06:40:00.681423+0000","last_became_active":"2024-03-24T06:36:55.743119+0000","last_became_peered":"2024-03-24T06:36:55.743119+0000","last_unstale":"2024-03-24T06:40:00.681423+0000","last_undegraded":"2024-03-24T06:40:00.681423+0000","last_fullsized":"2024-03-24T06:40:00.681423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:54.693565+0000","last_clean_scrub_stamp":"2024-03-24T06:36:54.693565+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:29:50.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.653402+0000","last_change":"2024-03-24T06:36:56.757330+0000","last_active":"2024-03-24T06:40:00.653402+0000","last_peered":"2024-03-24T06:40:00.653402+0000","last_clean":"2024-03-24T06:40:00.653402+0000","last_became_active":"2024-03-24T06:36:56.757008+0000","last_became_peered":"2024-03-24T06:36:56.757008+0000","last_unstale":"2024-03-24T06:40:00.653402+0000","last_undegraded":"2024-03-24T06:40:00.653402+0000","last_fullsized":"2024-03-24T06:40:00.653402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:50.791718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.806949+0000","last_change":"2024-03-24T06:36:56.749983+0000","last_active":"2024-03-24T06:40:00.806949+0000","last_peered":"2024-03-24T06:40:00.806949+0000","last_clean":"2024-03-24T06:40:00.806949+0000","last_became_active":"2024-03-24T06:36:56.749183+0000","last_became_peered":"2024-03-24T06:36:56.749183+0000","last_unstale":"2024-03-24T06:40:00.806949+0000","last_undegraded":"2024-03-24T06:40:00.806949+0000","last_fullsized":"2024-03-24T06:40:00.806949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:01:52.659623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124150+0000","last_change":"2024-03-24T06:37:56.832171+0000","last_active":"2024-03-24T06:40:00.124150+0000","last_peered":"2024-03-24T06:40:00.124150+0000","last_clean":"2024-03-24T06:40:00.124150+0000","last_became_active":"2024-03-24T06:37:56.831734+0000","last_became_peered":"2024-03-24T06:37:56.831734+0000","last_unstale":"2024-03-24T06:40:00.124150+0000","last_undegraded":"2024-03-24T06:40:00.124150+0000","last_fullsized":"2024-03-24T06:40:00.124150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:19.788831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.123937+0000","last_change":"2024-03-24T06:36:56.769293+0000","last_active":"2024-03-24T06:40:00.123937+0000","last_peered":"2024-03-24T06:40:00.123937+0000","last_clean":"2024-03-24T06:40:00.123937+0000","last_became_active":"2024-03-24T06:36:56.768996+0000","last_became_peered":"2024-03-24T06:36:56.768996+0000","last_unstale":"2024-03-24T06:40:00.123937+0000","last_undegraded":"2024-03-24T06:40:00.123937+0000","last_fullsized":"2024-03-24T06:40:00.123937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:55:14.119315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807833+0000","last_change":"2024-03-24T06:36:56.773456+0000","last_active":"2024-03-24T06:40:00.807833+0000","last_peered":"2024-03-24T06:40:00.807833+0000","last_clean":"2024-03-24T06:40:00.807833+0000","last_became_active":"2024-03-24T06:36:56.772873+0000","last_became_peered":"2024-03-24T06:36:56.772873+0000","last_unstale":"2024-03-24T06:40:00.807833+0000","last_undegraded":"2024-03-24T06:40:00.807833+0000","last_fullsized":"2024-03-24T06:40:00.807833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:49:14.001014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680980+0000","last_change":"2024-03-24T06:36:56.778059+0000","last_active":"2024-03-24T06:40:00.680980+0000","last_peered":"2024-03-24T06:40:00.680980+0000","last_clean":"2024-03-24T06:40:00.680980+0000","last_became_active":"2024-03-24T06:36:56.777615+0000","last_became_peered":"2024-03-24T06:36:56.777615+0000","last_unstale":"2024-03-24T06:40:00.680980+0000","last_undegraded":"2024-03-24T06:40:00.680980+0000","last_fullsized":"2024-03-24T06:40:00.680980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:14.972194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807380+0000","last_change":"2024-03-24T06:36:56.751064+0000","last_active":"2024-03-24T06:40:00.807380+0000","last_peered":"2024-03-24T06:40:00.807380+0000","last_clean":"2024-03-24T06:40:00.807380+0000","last_became_active":"2024-03-24T06:36:56.749640+0000","last_became_peered":"2024-03-24T06:36:56.749640+0000","last_unstale":"2024-03-24T06:40:00.807380+0000","last_undegraded":"2024-03-24T06:40:00.807380+0000","last_fullsized":"2024-03-24T06:40:00.807380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:27:27.160219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124693+0000","last_change":"2024-03-24T06:36:56.757371+0000","last_active":"2024-03-24T06:40:00.124693+0000","last_peered":"2024-03-24T06:40:00.124693+0000","last_clean":"2024-03-24T06:40:00.124693+0000","last_became_active":"2024-03-24T06:36:56.757001+0000","last_became_peered":"2024-03-24T06:36:56.757001+0000","last_unstale":"2024-03-24T06:40:00.124693+0000","last_undegraded":"2024-03-24T06:40:00.124693+0000","last_fullsized":"2024-03-24T06:40:00.124693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:33.989276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.125145+0000","last_change":"2024-03-24T06:36:56.759884+0000","last_active":"2024-03-24T06:40:00.125145+0000","last_peered":"2024-03-24T06:40:00.125145+0000","last_clean":"2024-03-24T06:40:00.125145+0000","last_became_active":"2024-03-24T06:36:56.759485+0000","last_became_peered":"2024-03-24T06:36:56.759485+0000","last_unstale":"2024-03-24T06:40:00.125145+0000","last_undegraded":"2024-03-24T06:40:00.125145+0000","last_fullsized":"2024-03-24T06:40:00.125145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:16.599871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.291432+0000","last_change":"2024-03-24T06:36:56.755742+0000","last_active":"2024-03-24T06:40:00.291432+0000","last_peered":"2024-03-24T06:40:00.291432+0000","last_clean":"2024-03-24T06:40:00.291432+0000","last_became_active":"2024-03-24T06:36:56.755449+0000","last_became_peered":"2024-03-24T06:36:56.755449+0000","last_unstale":"2024-03-24T06:40:00.291432+0000","last_undegraded":"2024-03-24T06:40:00.291432+0000","last_fullsized":"2024-03-24T06:40:00.291432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:48.717640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.124080+0000","last_change":"2024-03-24T06:36:56.763286+0000","last_active":"2024-03-24T06:40:00.124080+0000","last_peered":"2024-03-24T06:40:00.124080+0000","last_clean":"2024-03-24T06:40:00.124080+0000","last_became_active":"2024-03-24T06:36:56.761842+0000","last_became_peered":"2024-03-24T06:36:56.761842+0000","last_unstale":"2024-03-24T06:40:00.124080+0000","last_undegraded":"2024-03-24T06:40:00.124080+0000","last_fullsized":"2024-03-24T06:40:00.124080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:15:20.795294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.687904+0000","last_change":"2024-03-24T06:36:56.753627+0000","last_active":"2024-03-24T06:40:00.687904+0000","last_peered":"2024-03-24T06:40:00.687904+0000","last_clean":"2024-03-24T06:40:00.687904+0000","last_became_active":"2024-03-24T06:36:56.753291+0000","last_became_peered":"2024-03-24T06:36:56.753291+0000","last_unstale":"2024-03-24T06:40:00.687904+0000","last_undegraded":"2024-03-24T06:40:00.687904+0000","last_fullsized":"2024-03-24T06:40:00.687904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:05.965282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.680146+0000","last_change":"2024-03-24T06:36:56.776437+0000","last_active":"2024-03-24T06:40:00.680146+0000","last_peered":"2024-03-24T06:40:00.680146+0000","last_clean":"2024-03-24T06:40:00.680146+0000","last_became_active":"2024-03-24T06:36:56.776029+0000","last_became_peered":"2024-03-24T06:36:56.776029+0000","last_unstale":"2024-03-24T06:40:00.680146+0000","last_undegraded":"2024-03-24T06:40:00.680146+0000","last_fullsized":"2024-03-24T06:40:00.680146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:06.693085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T06:40:00.807199+0000","last_change":"2024-03-24T06:37:56.828731+0000","last_active":"2024-03-24T06:40:00.807199+0000","last_peered":"2024-03-24T06:40:00.807199+0000","last_clean":"2024-03-24T06:40:00.807199+0000","last_became_active":"2024-03-24T06:37:56.828384+0000","last_became_peered":"2024-03-24T06:37:56.828384+0000","last_unstale":"2024-03-24T06:40:00.807199+0000","last_undegraded":"2024-03-24T06:40:00.807199+0000","last_fullsized":"2024-03-24T06:40:00.807199+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T06:36:55.707253+0000","last_clean_scrub_stamp":"2024-03-24T06:36:55.707253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:26.897702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":600649,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1599,"num_read_kb":921,"num_write":364,"num_write_kb":1462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":644,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1801947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383730,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":520134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29060,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716092,"statfs":{"total":95995035648,"available":95965278208,"internally_reserved":0,"allocated":1908736,"data_stored":1310142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808892,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":524138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070532,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332168,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1116832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495695,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":526165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":578353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1105920,"data_stored":526839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":587788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182494,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1556480,"data_stored":984363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":520548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757333,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37136,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708016,"statfs":{"total":95995035648,"available":95957008384,"internally_reserved":0,"allocated":1843200,"data_stored":1254242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":394694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":201268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T06:40:06.308 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 -- 172.21.15.22:0/1337148935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c100650 msgr2=0x7f815c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 --2- 172.21.15.22:0/1337148935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c100650 0x7f815c100a70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f815000b9f0 tx=0x7f815002d8f0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 -- 172.21.15.22:0/1337148935 shutdown_connections 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 --2- 172.21.15.22:0/1337148935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c102200 0x7f815c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 --2- 172.21.15.22:0/1337148935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c101840 0x7f815c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 --2- 172.21.15.22:0/1337148935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c100650 0x7f815c100a70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 -- 172.21.15.22:0/1337148935 >> 172.21.15.22:0/1337148935 conn(0x7f815c0fbbc0 msgr2=0x7f815c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:06.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 -- 172.21.15.22:0/1337148935 shutdown_connections 2024-03-24T06:40:06.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.536+0000 7f8164000640 1 -- 172.21.15.22:0/1337148935 wait complete. 2024-03-24T06:40:06.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.537+0000 7f8164000640 1 Processor -- start 2024-03-24T06:40:06.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.537+0000 7f8164000640 1 -- start start 2024-03-24T06:40:06.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c101840 0x7f815c195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c102200 0x7f815c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f815c19d130 con 0x7f815c101840 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f815c19d2a0 con 0x7f815c102200 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8164000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f815c19d460 con 0x7f815c100650 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44886/0 (socket says 172.21.15.22:44886) 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 -- 172.21.15.22:0/703408288 learned_addr learned my addr 172.21.15.22:0/703408288 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8162000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c102200 0x7f815c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8160c00640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c101840 0x7f815c195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 -- 172.21.15.22:0/703408288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c102200 msgr2=0x7f815c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c102200 0x7f815c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 -- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c101840 msgr2=0x7f815c195b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c101840 0x7f815c195b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.538+0000 7f8161600640 1 -- 172.21.15.22:0/703408288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814c009560 con 0x7f815c100650 2024-03-24T06:40:06.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f8161600640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f815002de30 tx=0x7f81500052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:06.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8150004250 con 0x7f815c100650 2024-03-24T06:40:06.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8150009740 con 0x7f815c100650 2024-03-24T06:40:06.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815000b6a0 con 0x7f815c100650 2024-03-24T06:40:06.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8150004bc0 con 0x7f815c100650 2024-03-24T06:40:06.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.539+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815c19da50 con 0x7f815c100650 2024-03-24T06:40:06.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.540+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815c1079f0 con 0x7f815c100650 2024-03-24T06:40:06.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.542+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f8150031050 con 0x7f815c100650 2024-03-24T06:40:06.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.543+0000 7f815a000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 0x7f81440a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:06.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.543+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f81500f27b0 con 0x7f815c100650 2024-03-24T06:40:06.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.544+0000 7f8160c00640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 0x7f81440a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:06.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.544+0000 7f8160c00640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 0x7f81440a4800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f815c196b50 tx=0x7f814c009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:06.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.552+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f81500baf10 con 0x7f815c100650 2024-03-24T06:40:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:06 smithi022 ceph-mon[32003]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:06 smithi022 ceph-mon[32003]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:06 smithi022 ceph-mon[32003]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:06 smithi155 ceph-mon[36262]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:06 smithi155 ceph-mon[36262]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:06 smithi155 ceph-mon[36262]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:06.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.844+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f815c10af60 con 0x7f815c100650 2024-03-24T06:40:06.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.846+0000 7f815a000640 1 -- 172.21.15.22:0/703408288 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25085 (secure 0 0 0) 0x7f81500be5d0 con 0x7f815c100650 2024-03-24T06:40:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:06.847 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":94,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:40:00.112797+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:40:06.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 msgr2=0x7f81440a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 0x7f81440a4800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f815c196b50 tx=0x7f814c009260 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 msgr2=0x7f815c195640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f815002de30 tx=0x7f81500052f0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 shutdown_connections 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f81440a2340 0x7f81440a4800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f815c102200 0x7f815c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f815c101840 0x7f815c195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 --2- 172.21.15.22:0/703408288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f815c100650 0x7f815c195640 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.848+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 >> 172.21.15.22:0/703408288 conn(0x7f815c0fbbc0 msgr2=0x7f815c104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.849+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 shutdown_connections 2024-03-24T06:40:06.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:06.849+0000 7f8164000640 1 -- 172.21.15.22:0/703408288 wait complete. 2024-03-24T06:40:06.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:40:06.873 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:40:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:06 smithi143 ceph-mon[36520]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:06 smithi143 ceph-mon[36520]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:06 smithi143 ceph-mon[36520]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:40:07.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.061+0000 7fb639400640 1 -- 172.21.15.22:0/3560391066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb634102800 msgr2=0x7fb634102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.061+0000 7fb639400640 1 --2- 172.21.15.22:0/3560391066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb634102800 0x7fb634102c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb620009860 tx=0x7fb62002bb70 comp rx=0 tx=0).stop 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 -- 172.21.15.22:0/3560391066 shutdown_connections 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 --2- 172.21.15.22:0/3560391066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb6341031c0 0x7fb634109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 --2- 172.21.15.22:0/3560391066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb634102800 0x7fb634102c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 --2- 172.21.15.22:0/3560391066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 -- 172.21.15.22:0/3560391066 >> 172.21.15.22:0/3560391066 conn(0x7fb6340fcb80 msgr2=0x7fb6340fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 -- 172.21.15.22:0/3560391066 shutdown_connections 2024-03-24T06:40:07.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.062+0000 7fb639400640 1 -- 172.21.15.22:0/3560391066 wait complete. 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 Processor -- start 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 -- start start 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 0x7fb634199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 0x7fb6341a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb6341a1420 con 0x7fb6341031c0 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb6341a1590 con 0x7fb634101610 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.063+0000 7fb639400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb6341a1750 con 0x7fb634102800 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 0x7fb634199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37066/0 (socket says 172.21.15.22:37066) 2024-03-24T06:40:07.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633e00640 1 -- 172.21.15.22:0/2692062462 learned_addr learned my addr 172.21.15.22:0/2692062462 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb638a00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 0x7fb6341a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 -- 172.21.15.22:0/2692062462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 msgr2=0x7fb634199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 -- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 msgr2=0x7fb6341a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 0x7fb6341a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 -- 172.21.15.22:0/2692062462 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb624009560 con 0x7fb634102800 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633e00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:07.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb638a00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 0x7fb6341a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:07.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.064+0000 7fb633400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 0x7fb634199e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb620037040 tx=0x7fb620005cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:07.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.065+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62002c510 con 0x7fb634102800 2024-03-24T06:40:07.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.065+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb62002cb50 con 0x7fb634102800 2024-03-24T06:40:07.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.065+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620032600 con 0x7fb634102800 2024-03-24T06:40:07.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.065+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb620009490 con 0x7fb634102800 2024-03-24T06:40:07.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.065+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6341a1c00 con 0x7fb634102800 2024-03-24T06:40:07.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.066+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb63410d020 con 0x7fb634102800 2024-03-24T06:40:07.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.068+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fb62003b040 con 0x7fb634102800 2024-03-24T06:40:07.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.069+0000 7fb630c00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 0x7fb6180a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.069+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb6200effe0 con 0x7fb634102800 2024-03-24T06:40:07.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.070+0000 7fb633e00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 0x7fb6180a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.070+0000 7fb633e00640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 0x7fb6180a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6240058c0 tx=0x7fb624009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:07.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.073+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb6200b8940 con 0x7fb634102800 2024-03-24T06:40:07.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.292+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6341a2030 con 0x7fb634102800 2024-03-24T06:40:07.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.293+0000 7fb630c00640 1 -- 172.21.15.22:0/2692062462 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb6200bc000 con 0x7fb634102800 2024-03-24T06:40:07.293 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:40:07.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 msgr2=0x7fb6180a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 0x7fb6180a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6240058c0 tx=0x7fb624009260 comp rx=0 tx=0).stop 2024-03-24T06:40:07.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 msgr2=0x7fb634199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 0x7fb634199e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb620037040 tx=0x7fb620005cb0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 shutdown_connections 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb6180a2340 0x7fb6180a4800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6341031c0 0x7fb6341a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb634102800 0x7fb634199e70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 --2- 172.21.15.22:0/2692062462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb634101610 0x7fb634199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 >> 172.21.15.22:0/2692062462 conn(0x7fb6340fcb80 msgr2=0x7fb634073940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 shutdown_connections 2024-03-24T06:40:07.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.296+0000 7fb639400640 1 -- 172.21.15.22:0/2692062462 wait complete. 2024-03-24T06:40:07.314 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 -- 172.21.15.22:0/3206683041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c104b20 msgr2=0x7ff48c104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 --2- 172.21.15.22:0/3206683041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c104b20 0x7ff48c104fa0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff478009900 tx=0x7ff47802bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 -- 172.21.15.22:0/3206683041 shutdown_connections 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 --2- 172.21.15.22:0/3206683041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c1054e0 0x7ff48c10bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 --2- 172.21.15.22:0/3206683041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c104b20 0x7ff48c104fa0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 --2- 172.21.15.22:0/3206683041 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 0x7ff48c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 -- 172.21.15.22:0/3206683041 >> 172.21.15.22:0/3206683041 conn(0x7ff48c0feec0 msgr2=0x7ff48c1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:07.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 -- 172.21.15.22:0/3206683041 shutdown_connections 2024-03-24T06:40:07.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.514+0000 7ff493c00640 1 -- 172.21.15.22:0/3206683041 wait complete. 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 Processor -- start 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 -- start start 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 0x7ff48c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 0x7ff48c199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 0x7ff48c1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1a14c0 con 0x7ff48c103930 2024-03-24T06:40:07.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1a1630 con 0x7ff48c104b20 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.515+0000 7ff493c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1a17f0 con 0x7ff48c1054e0 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff491c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 0x7ff48c1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff491200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 0x7ff48c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 0x7ff48c199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff491c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 0x7ff48c1a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44936/0 (socket says 172.21.15.22:44936) 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff491c00640 1 -- 172.21.15.22:0/2277396536 learned_addr learned my addr 172.21.15.22:0/2277396536 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 -- 172.21.15.22:0/2277396536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 msgr2=0x7ff48c1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 0x7ff48c1a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 -- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 msgr2=0x7ff48c1999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 0x7ff48c1999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 -- 172.21.15.22:0/2277396536 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff480009490 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.516+0000 7ff48be00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 0x7ff48c199f10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff478000c00 tx=0x7ff478005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.517+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4780043d0 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.517+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4780052d0 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.517+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478003d40 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.517+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff484009840 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.517+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff478009560 con 0x7ff48c104b20 2024-03-24T06:40:07.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.518+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48c10ac70 con 0x7ff48c104b20 2024-03-24T06:40:07.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.521+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7ff478005440 con 0x7ff48c104b20 2024-03-24T06:40:07.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.523+0000 7ff489600640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 0x7ff4740a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:07.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.524+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff4780f05b0 con 0x7ff48c104b20 2024-03-24T06:40:07.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.524+0000 7ff491200640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 0x7ff4740a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:07.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.525+0000 7ff491200640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 0x7ff4740a4780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff48c104980 tx=0x7ff480005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:07.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.526+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff4780b8e20 con 0x7ff48c104b20 2024-03-24T06:40:07.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.814+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff48c10f520 con 0x7ff48c104b20 2024-03-24T06:40:07.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.820+0000 7ff489600640 1 -- 172.21.15.22:0/2277396536 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7ff4780bc4e0 con 0x7ff48c104b20 2024-03-24T06:40:07.820 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:07.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 msgr2=0x7ff4740a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 0x7ff4740a4780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff48c104980 tx=0x7ff480005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:07.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 msgr2=0x7ff48c199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:07.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 0x7ff48c199f10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff478000c00 tx=0x7ff478005e50 comp rx=0 tx=0).stop 2024-03-24T06:40:07.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 shutdown_connections 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff4740a22c0 0x7ff4740a4780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff48c1054e0 0x7ff48c1a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff48c104b20 0x7ff48c199f10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 --2- 172.21.15.22:0/2277396536 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff48c103930 0x7ff48c1999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 >> 172.21.15.22:0/2277396536 conn(0x7ff48c0feec0 msgr2=0x7ff48c107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 shutdown_connections 2024-03-24T06:40:07.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:07.822+0000 7ff493c00640 1 -- 172.21.15.22:0/2277396536 wait complete. 2024-03-24T06:40:07.846 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:40:07.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:07.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:40:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/703408288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:07.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:07 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2692062462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:40:07.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/703408288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:07.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:07.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:07.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:07 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2692062462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 -- 172.21.15.22:0/156290968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f00756f0 msgr2=0x7fe5f0073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/156290968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f00756f0 0x7fe5f0073b40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009900 tx=0x7fe5e002bcc0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 -- 172.21.15.22:0/156290968 shutdown_connections 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/156290968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f01095b0 0x7fe5f010ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/156290968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f0074110 0x7fe5f0074570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/156290968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f00756f0 0x7fe5f0073b40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 -- 172.21.15.22:0/156290968 >> 172.21.15.22:0/156290968 conn(0x7fe5f00fcc90 msgr2=0x7fe5f00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 -- 172.21.15.22:0/156290968 shutdown_connections 2024-03-24T06:40:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.026+0000 7fe5f6a00640 1 -- 172.21.15.22:0/156290968 wait complete. 2024-03-24T06:40:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.027+0000 7fe5f6a00640 1 Processor -- start 2024-03-24T06:40:08.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.027+0000 7fe5f6a00640 1 -- start start 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:40:08.029 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/703408288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:08.030 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:07 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2692062462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:40:08.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f00756f0 0x7fe5f0199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 0x7fe5f01a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01a1500 con 0x7fe5f00756f0 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01a1670 con 0x7fe5f0074110 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f6a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01a1830 con 0x7fe5f01095b0 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5efe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5f4a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 0x7fe5f01a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5efe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37104/0 (socket says 172.21.15.22:37104) 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5efe00640 1 -- 172.21.15.22:0/1451184432 learned_addr learned my addr 172.21.15.22:0/1451184432 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.028+0000 7fe5ef400640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f00756f0 0x7fe5f0199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 -- 172.21.15.22:0/1451184432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 msgr2=0x7fe5f01a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 0x7fe5f01a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 -- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f00756f0 msgr2=0x7fe5f0199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f00756f0 0x7fe5f0199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 -- 172.21.15.22:0/1451184432 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e8009840 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5f4a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 0x7fe5f01a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5efe00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0000c00 tx=0x7fe5e0005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e002dd10 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5e002de70 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e0009560 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.029+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0003c20 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.030+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f01a1e30 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.031+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f0108370 con 0x7fe5f0074110 2024-03-24T06:40:08.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.032+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fe5e0004df0 con 0x7fe5f0074110 2024-03-24T06:40:08.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.034+0000 7fe5ecc00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 0x7fe5d40a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.034+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe5e00f0b80 con 0x7fe5f0074110 2024-03-24T06:40:08.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.034+0000 7fe5ef400640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 0x7fe5d40a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.035+0000 7fe5ef400640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 0x7fe5d40a4780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5f019af20 tx=0x7fe5dc005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.038+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe5e00b9450 con 0x7fe5f0074110 2024-03-24T06:40:08.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.245+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe5f002d130 con 0x7fe5f0074110 2024-03-24T06:40:08.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.245+0000 7fe5ecc00640 1 -- 172.21.15.22:0/1451184432 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fe5e00430a0 con 0x7fe5f0074110 2024-03-24T06:40:08.245 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:40:08.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.248+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 msgr2=0x7fe5d40a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.248+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 0x7fe5d40a4780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe5f019af20 tx=0x7fe5dc005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.248+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 msgr2=0x7fe5f0199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.248+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0000c00 tx=0x7fe5e0005e30 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.248+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 shutdown_connections 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5d40a22c0 0x7fe5d40a4780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f01095b0 0x7fe5f01a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe5f00756f0 0x7fe5f0199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 --2- 172.21.15.22:0/1451184432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5f0074110 0x7fe5f0199a10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 >> 172.21.15.22:0/1451184432 conn(0x7fe5f00fcc90 msgr2=0x7fe5f0105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 shutdown_connections 2024-03-24T06:40:08.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.249+0000 7fe5f6a00640 1 -- 172.21.15.22:0/1451184432 wait complete. 2024-03-24T06:40:08.265 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 -- 172.21.15.22:0/367165762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb550107660 msgr2=0x7fb550107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 --2- 172.21.15.22:0/367165762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb550107660 0x7fb550107a40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb540009930 tx=0x7fb54002bc20 comp rx=0 tx=0).stop 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 -- 172.21.15.22:0/367165762 shutdown_connections 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 --2- 172.21.15.22:0/367165762 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 0x7fb55010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 --2- 172.21.15.22:0/367165762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5501015a0 0x7fb550101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 --2- 172.21.15.22:0/367165762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb550107660 0x7fb550107a40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 -- 172.21.15.22:0/367165762 >> 172.21.15.22:0/367165762 conn(0x7fb5500fd0b0 msgr2=0x7fb5500ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:08.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 -- 172.21.15.22:0/367165762 shutdown_connections 2024-03-24T06:40:08.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.458+0000 7fb555800640 1 -- 172.21.15.22:0/367165762 wait complete. 2024-03-24T06:40:08.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.460+0000 7fb555800640 1 Processor -- start 2024-03-24T06:40:08.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.460+0000 7fb555800640 1 -- start start 2024-03-24T06:40:08.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.460+0000 7fb555800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.460+0000 7fb555800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 0x7fb55019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb555800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb550107660 0x7fb55019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb555800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb55019ec10 con 0x7fb550101f60 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb555800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb55019ed80 con 0x7fb550107660 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb555800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb55019ef40 con 0x7fb5501015a0 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44962/0 (socket says 172.21.15.22:44962) 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 -- 172.21.15.22:0/2923504698 learned_addr learned my addr 172.21.15.22:0/2923504698 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb547e00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 0x7fb55019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb554e00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb550107660 0x7fb55019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 -- 172.21.15.22:0/2923504698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb550107660 msgr2=0x7fb55019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb550107660 0x7fb55019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 -- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 msgr2=0x7fb55019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 0x7fb55019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.461+0000 7fb54fe00640 1 -- 172.21.15.22:0/2923504698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb540009560 con 0x7fb5501015a0 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb54fe00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb540000c00 tx=0x7fb54002c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54002dea0 con 0x7fb5501015a0 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5400042d0 con 0x7fb5501015a0 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb55019f1c0 con 0x7fb5501015a0 2024-03-24T06:40:08.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540003b50 con 0x7fb5501015a0 2024-03-24T06:40:08.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.462+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5501a6aa0 con 0x7fb5501015a0 2024-03-24T06:40:08.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.463+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb550068420 con 0x7fb5501015a0 2024-03-24T06:40:08.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.467+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fb540003cb0 con 0x7fb5501015a0 2024-03-24T06:40:08.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.468+0000 7fb54d600640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 0x7fb5340a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.468+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb5400f0d60 con 0x7fb5501015a0 2024-03-24T06:40:08.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.468+0000 7fb547e00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 0x7fb5340a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.469+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb5400b95d0 con 0x7fb5501015a0 2024-03-24T06:40:08.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.469+0000 7fb547e00640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 0x7fb5340a4800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb55010e080 tx=0x7fb538005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.679+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb5501116f0 con 0x7fb5501015a0 2024-03-24T06:40:08.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.680+0000 7fb54d600640 1 -- 172.21.15.22:0/2923504698 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb5400430a0 con 0x7fb5501015a0 2024-03-24T06:40:08.680 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.682+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 msgr2=0x7fb5340a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.682+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 0x7fb5340a4800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb55010e080 tx=0x7fb538005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.682+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 msgr2=0x7fb550199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.682+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb540000c00 tx=0x7fb54002c5e0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 shutdown_connections 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5340a2340 0x7fb5340a4800 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb550107660 0x7fb55019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb550101f60 0x7fb55019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 --2- 172.21.15.22:0/2923504698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5501015a0 0x7fb550199d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 >> 172.21.15.22:0/2923504698 conn(0x7fb5500fd0b0 msgr2=0x7fb5500fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:08.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 shutdown_connections 2024-03-24T06:40:08.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.683+0000 7fb555800640 1 -- 172.21.15.22:0/2923504698 wait complete. 2024-03-24T06:40:08.704 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:40:08.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 -- 172.21.15.22:0/665764064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881016a0 msgr2=0x7fcd88101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 --2- 172.21.15.22:0/665764064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881016a0 0x7fcd88101b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009860 tx=0x7fcd7402bb70 comp rx=0 tx=0).stop 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 -- 172.21.15.22:0/665764064 shutdown_connections 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 --2- 172.21.15.22:0/665764064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd88102060 0x7fcd8810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 --2- 172.21.15.22:0/665764064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881016a0 0x7fcd88101b20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 --2- 172.21.15.22:0/665764064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd881076b0 0x7fcd88107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 -- 172.21.15.22:0/665764064 >> 172.21.15.22:0/665764064 conn(0x7fcd880fd1b0 msgr2=0x7fcd880ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 -- 172.21.15.22:0/665764064 shutdown_connections 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.897+0000 7fcd90600640 1 -- 172.21.15.22:0/665764064 wait complete. 2024-03-24T06:40:08.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.898+0000 7fcd90600640 1 Processor -- start 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.898+0000 7fcd90600640 1 -- start start 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 0x7fcd88073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 0x7fcd88074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcd881a6110 con 0x7fcd881016a0 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcd881a6280 con 0x7fcd881076b0 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd90600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd881a6440 con 0x7fcd88102060 2024-03-24T06:40:08.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44972/0 (socket says 172.21.15.22:44972) 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8e600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 0x7fcd88074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 -- 172.21.15.22:0/706499363 learned_addr learned my addr 172.21.15.22:0/706499363 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8dc00640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 0x7fcd88073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 -- 172.21.15.22:0/706499363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 msgr2=0x7fcd88074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 0x7fcd88074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 -- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 msgr2=0x7fcd88073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 0x7fcd88073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8d200640 1 -- 172.21.15.22:0/706499363 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd74009490 con 0x7fcd88102060 2024-03-24T06:40:08.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8dc00640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 0x7fcd88073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.899+0000 7fcd8e600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 0x7fcd88074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd8d200640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd74037040 tx=0x7fcd740045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd740349d0 con 0x7fcd88102060 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7402c9f0 con 0x7fcd88102060 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74032600 con 0x7fcd88102060 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd881a66c0 con 0x7fcd88102060 2024-03-24T06:40:08.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.900+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd881a6a30 con 0x7fcd88102060 2024-03-24T06:40:08.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.902+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd88068950 con 0x7fcd88102060 2024-03-24T06:40:08.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.903+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fcd7403b040 con 0x7fcd88102060 2024-03-24T06:40:08.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.904+0000 7fcd86000640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 0x7fcd700a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:08.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.905+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fcd740efff0 con 0x7fcd88102060 2024-03-24T06:40:08.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.905+0000 7fcd8dc00640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 0x7fcd700a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:08.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.906+0000 7fcd8dc00640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 0x7fcd700a4800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c005fd0 tx=0x7fcd7c0095c0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:08.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:08.908+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd740b88a0 con 0x7fcd88102060 2024-03-24T06:40:09.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2277396536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 146 B/s wr, 24 op/s 2024-03-24T06:40:09.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1451184432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:08 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2923504698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2277396536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 146 B/s wr, 24 op/s 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1451184432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:08 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2923504698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.180+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd88074f40 con 0x7fcd88102060 2024-03-24T06:40:09.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.182+0000 7fcd86000640 1 -- 172.21.15.22:0/706499363 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11939 (secure 0 0 0) 0x7fcd740b8020 con 0x7fcd88102060 2024-03-24T06:40:09.182 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:09.182 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:39:58.997320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:40:09.182 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 29 2024-03-24T06:40:09.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 msgr2=0x7fcd700a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 0x7fcd700a4800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c005fd0 tx=0x7fcd7c0095c0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 msgr2=0x7fcd88073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd74037040 tx=0x7fcd740045b0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 shutdown_connections 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcd700a2340 0x7fcd700a4800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd881076b0 0x7fcd88074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd88102060 0x7fcd88073f50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 --2- 172.21.15.22:0/706499363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcd881016a0 0x7fcd88073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 >> 172.21.15.22:0/706499363 conn(0x7fcd880fd1b0 msgr2=0x7fcd880fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 shutdown_connections 2024-03-24T06:40:09.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.185+0000 7fcd90600640 1 -- 172.21.15.22:0/706499363 wait complete. 2024-03-24T06:40:09.203 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:40:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2277396536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T06:40:09.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 146 B/s wr, 24 op/s 2024-03-24T06:40:09.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1451184432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:08 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2923504698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:40:09.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 -- 172.21.15.22:0/555799262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 msgr2=0x7f7424073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 --2- 172.21.15.22:0/555799262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424073f20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7414009900 tx=0x7f741402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 -- 172.21.15.22:0/555799262 shutdown_connections 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 --2- 172.21.15.22:0/555799262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 0x7f7424110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 --2- 172.21.15.22:0/555799262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742410d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 --2- 172.21.15.22:0/555799262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424073f20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 -- 172.21.15.22:0/555799262 >> 172.21.15.22:0/555799262 conn(0x7f74240fcff0 msgr2=0x7f74240ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 -- 172.21.15.22:0/555799262 shutdown_connections 2024-03-24T06:40:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.410+0000 7f7429a00640 1 -- 172.21.15.22:0/555799262 wait complete. 2024-03-24T06:40:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.411+0000 7f7429a00640 1 Processor -- start 2024-03-24T06:40:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 -- start start 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 0x7f742419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f742419ec40 con 0x7f742410dd40 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f742419edb0 con 0x7f7424074460 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f742419ef70 con 0x7f7424073b40 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7423e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7423e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44990/0 (socket says 172.21.15.22:44990) 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7423e00640 1 -- 172.21.15.22:0/1848547863 learned_addr learned my addr 172.21.15.22:0/1848547863 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f741be00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.412+0000 7f7429000640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 0x7f742419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 -- 172.21.15.22:0/1848547863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 msgr2=0x7f7424199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 -- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 msgr2=0x7f742419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 0x7f742419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 -- 172.21.15.22:0/1848547863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7414009560 con 0x7f7424074460 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7423e00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f741be00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742419a310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f740c00e9f0 tx=0x7f740c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c00ccc0 con 0x7f7424074460 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c00ce20 con 0x7f7424074460 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742419f250 con 0x7f7424074460 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c004a00 con 0x7f7424074460 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.413+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74241a6ad0 con 0x7f7424074460 2024-03-24T06:40:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.415+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7424068420 con 0x7f7424074460 2024-03-24T06:40:09.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.418+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f740c0050b0 con 0x7f7424074460 2024-03-24T06:40:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.419+0000 7f7421600640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 0x7f74080a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.419+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f740c014070 con 0x7f7424074460 2024-03-24T06:40:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.420+0000 7f7423e00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 0x7f74080a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.420+0000 7f7423e00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 0x7f74080a4780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7414005ec0 tx=0x7f7414005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.423+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f740c096cd0 con 0x7f7424074460 2024-03-24T06:40:09.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.630+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7424108f20 con 0x7f7424074460 2024-03-24T06:40:09.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.632+0000 7f7421600640 1 -- 172.21.15.22:0/1848547863 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25085 (secure 0 0 0) 0x7f740c09a390 con 0x7f7424074460 2024-03-24T06:40:09.633 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:09.633 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":94,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:40:00.112797+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:40:09.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 msgr2=0x7f74080a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 0x7f74080a4780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7414005ec0 tx=0x7f7414005a20 comp rx=0 tx=0).stop 2024-03-24T06:40:09.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 msgr2=0x7f742419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742419a310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f740c00e9f0 tx=0x7f740c00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 shutdown_connections 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f74080a22c0 0x7f74080a4780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f742410dd40 0x7f742419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7424074460 0x7f742419a310 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 --2- 172.21.15.22:0/1848547863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7424073b40 0x7f7424199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.635+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 >> 172.21.15.22:0/1848547863 conn(0x7f74240fcff0 msgr2=0x7f74240fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.636+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 shutdown_connections 2024-03-24T06:40:09.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.636+0000 7f7429a00640 1 -- 172.21.15.22:0/1848547863 wait complete. 2024-03-24T06:40:09.654 DEBUG:tasks.fwd_scrub:Yielding 2024-03-24T06:40:09.655 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T06:40:09.664 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T06:40:09.664 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:40:09.664 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-24T06:40:09.665 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-24T06:40:09.665 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:40:09.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:40:09.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:40:09.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 -- 172.21.15.22:0/2932207013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 msgr2=0x7f8f08073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 --2- 172.21.15.22:0/2932207013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08073f20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009900 tx=0x7f8efc02bcc0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 -- 172.21.15.22:0/2932207013 shutdown_connections 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 --2- 172.21.15.22:0/2932207013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f0810dd30 0x7f8f08110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 --2- 172.21.15.22:0/2932207013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f08074460 0x7f8f0810d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 --2- 172.21.15.22:0/2932207013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08073f20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.864+0000 7f8f0f200640 1 -- 172.21.15.22:0/2932207013 >> 172.21.15.22:0/2932207013 conn(0x7f8f080fd060 msgr2=0x7f8f080ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.865+0000 7f8f0f200640 1 -- 172.21.15.22:0/2932207013 shutdown_connections 2024-03-24T06:40:09.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.865+0000 7f8f0f200640 1 -- 172.21.15.22:0/2932207013 wait complete. 2024-03-24T06:40:09.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.866+0000 7f8f0f200640 1 Processor -- start 2024-03-24T06:40:09.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.866+0000 7f8f0f200640 1 -- start start 2024-03-24T06:40:09.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 0x7f8f0819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 0x7f8f0819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0819ec40 con 0x7f8f0810dd30 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0819edb0 con 0x7f8f08074460 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0819ef70 con 0x7f8f08073b40 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 0x7f8f0819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45030/0 (socket says 172.21.15.22:45030) 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0de00640 1 -- 172.21.15.22:0/1622549333 learned_addr learned my addr 172.21.15.22:0/1622549333 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.867+0000 7f8f0e800640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 0x7f8f0819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 -- 172.21.15.22:0/1622549333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 msgr2=0x7f8f0819a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 0x7f8f0819a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 -- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 msgr2=0x7f8f0819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 0x7f8f0819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 -- 172.21.15.22:0/1622549333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc009560 con 0x7f8f08073b40 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0d400640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 0x7f8f0819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0e800640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 0x7f8f0819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0de00640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8efc000c00 tx=0x7f8efc02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc02dbd0 con 0x7f8f08073b40 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8efc02dd30 con 0x7f8f08073b40 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc003b20 con 0x7f8f08073b40 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.868+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0819f1f0 con 0x7f8f08073b40 2024-03-24T06:40:09.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f081a6ad0 con 0x7f8f08073b40 2024-03-24T06:40:09.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 -- 172.21.15.22:0/3372845458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81038a0 msgr2=0x7f07a8103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 --2- 172.21.15.22:0/3372845458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81038a0 0x7f07a8103d20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0794009890 tx=0x7f079402bb70 comp rx=0 tx=0).stop 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 -- 172.21.15.22:0/3372845458 shutdown_connections 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 --2- 172.21.15.22:0/3372845458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a8110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 --2- 172.21.15.22:0/3372845458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81038a0 0x7f07a8103d20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 --2- 172.21.15.22:0/3372845458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81098b0 0x7f07a8109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 -- 172.21.15.22:0/3372845458 >> 172.21.15.22:0/3372845458 conn(0x7f07a80ff350 msgr2=0x7f07a8101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 -- 172.21.15.22:0/3372845458 shutdown_connections 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.869+0000 7f07b0000640 1 -- 172.21.15.22:0/3372845458 wait complete. 2024-03-24T06:40:09.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.870+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f080751f0 con 0x7f8f08073b40 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 Processor -- start 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 -- start start 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81038a0 0x7f07a810f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81098b0 0x7f07a81a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f07a81a32c0 con 0x7f07a81038a0 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f07a81a3430 con 0x7f07a8104260 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.871+0000 7f07b0000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f07a81a35f0 con 0x7f07a81098b0 2024-03-24T06:40:09.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07ae000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81098b0 0x7f07a81a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37196/0 (socket says 172.21.15.22:37196) 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 -- 172.21.15.22:0/3162196479 learned_addr learned my addr 172.21.15.22:0/3162196479 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07ad600640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81038a0 0x7f07a810f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 -- 172.21.15.22:0/3162196479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81098b0 msgr2=0x7f07a81a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81098b0 0x7f07a81a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 -- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81038a0 msgr2=0x7f07a810f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81038a0 0x7f07a810f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 -- 172.21.15.22:0/3162196479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0794009490 con 0x7f07a8104260 2024-03-24T06:40:09.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f8efc003c80 con 0x7f8f08073b40 2024-03-24T06:40:09.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.872+0000 7f07acc00640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0794037040 tx=0x7f07940045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.873+0000 7f07a6000640 1 -- 172.21.15.22:0/3162196479 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940349d0 con 0x7f07a8104260 2024-03-24T06:40:09.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.873+0000 7f07a6000640 1 -- 172.21.15.22:0/3162196479 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079402c9f0 con 0x7f07a8104260 2024-03-24T06:40:09.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.873+0000 7f07a6000640 1 -- 172.21.15.22:0/3162196479 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794032eb0 con 0x7f07a8104260 2024-03-24T06:40:09.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.873+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a81a3870 con 0x7f07a8104260 2024-03-24T06:40:09.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.873+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a81aaef0 con 0x7f07a8104260 2024-03-24T06:40:09.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.874+0000 7f8f02a00640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 0x7f8ef00a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.874+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8efc0f0ac0 con 0x7f8f08073b40 2024-03-24T06:40:09.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.874+0000 7f8f0d400640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 0x7f8ef00a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.875+0000 7f8f0d400640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 0x7f8ef00a47f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f0810f040 tx=0x7f8ef8005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 Processor -- start 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 -- start start 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579eddeb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07a6000640 1 -- 172.21.15.22:0/3162196479 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f079403b040 con 0x7f07a8104260 2024-03-24T06:40:09.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579edd4e2d0 con 0x5579edde9330 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579edd4b800 con 0x5579edde0750 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.877+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579edd46fe0 con 0x5579edddfe30 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ae000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ae000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45058/0 (socket says 172.21.15.22:45058) 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ae000640 1 -- 172.21.15.22:0/370979180 learned_addr learned my addr 172.21.15.22:0/370979180 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 -- 172.21.15.22:0/370979180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 msgr2=0x5579edde0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 -- 172.21.15.22:0/370979180 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 msgr2=0x5579eddeb720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579eddeb720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.878+0000 7f07ad600640 1 -- 172.21.15.22:0/370979180 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579edd08e90 con 0x5579edde0750 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f07ad600640 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde8df0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f079c009c30 tx=0x7f079c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56c64d0e8ec2114c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f079a600640 1 -- 172.21.15.22:0/370979180 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c009de0 con 0x5579edde0750 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f079a600640 1 -- 172.21.15.22:0/370979180 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c004510 con 0x5579edde0750 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f079a600640 1 -- 172.21.15.22:0/370979180 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c019670 con 0x5579edde0750 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f07a6000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 0x7f07900a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.879+0000 7f07a6000640 1 -- 172.21.15.22:0/3162196479 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f07940effe0 con 0x7f07a8104260 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07ae000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 0x7f07900a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 -- 172.21.15.22:0/370979180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 msgr2=0x5579edde8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde8df0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f079c009c30 tx=0x7f079c011810 comp rx=0 tx=0).stop 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 -- 172.21.15.22:0/370979180 shutdown_connections 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579eddeb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde8df0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/370979180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 -- 172.21.15.22:0/370979180 >> 172.21.15.22:0/370979180 conn(0x5579eddda560 msgr2=0x5579eddda940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 -- 172.21.15.22:0/370979180 shutdown_connections 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07ae000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 0x7f07900a4800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07a0012640 tx=0x7f07a0009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.880+0000 7f07b2c5d740 1 -- 172.21.15.22:0/370979180 wait complete. 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.881+0000 7f07b2c5d740 1 -- start start 2024-03-24T06:40:09.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.881+0000 7f07b2c5d740 1 Processor -- start 2024-03-24T06:40:09.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579edde6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579edcebfc0 con 0x5579edde9330 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579edcebc60 con 0x5579edde0750 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579eda8f0c0 con 0x5579edddfe30 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07b2c5d740 10 client.0 _ll_register_callbacks cb 0x5579edd03e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37216/0 (socket says 172.21.15.22:37216) 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ad600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ae000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579edde6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8efc0b9420 con 0x7f8f08073b40 2024-03-24T06:40:09.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07acc00640 1 -- 172.21.15.22:0/2214273252 learned_addr learned my addr 172.21.15.22:0/2214273252 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ad600640 1 -- 172.21.15.22:0/2214273252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 msgr2=0x5579edde68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ad600640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ad600640 1 -- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 msgr2=0x5579edde6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.882+0000 7f07ad600640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579edde6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f07ad600640 1 -- 172.21.15.22:0/2214273252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c009890 con 0x5579edddfe30 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f07acc00640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f07ae000640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579edde6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f07ad600640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde63b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f079c006010 tx=0x7f079c003ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ef4857dd3258d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f078ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c004130 con 0x5579edddfe30 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579edd6be90 con 0x5579edddfe30 2024-03-24T06:40:09.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.883+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f079c004290 con 0x5579edddfe30 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f079c005040 con 0x5579edddfe30 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 10 client.25702 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579edd45660 con 0x5579edddfe30 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5579edd6be90 con 0x5579edddfe30 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f079c02e4b0 con 0x5579edddfe30 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 10 client.25702 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 10 client.25702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 20 client.25702 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579eddf9760 0x5579eddfbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 4 client.25702 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.884+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579edd3d210 con 0x5579eddf9760 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.885+0000 7f07acc00640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579eddf9760 0x5579eddfbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.885+0000 7f07acc00640 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579eddf9760 0x5579eddfbb70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=642e61d6317a78dd server_cookie=fc5b8818491db593 in_seq=0 out_seq=0 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.885+0000 7f078ea00640 10 client.25702 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.886+0000 7f078ea00640 1 -- 172.21.15.22:0/2214273252 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579edd3d210 con 0x5579eddf9760 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.886+0000 7f078ea00640 10 client.25702 handle_command_reply: tid=0 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.886+0000 7f07b2c5d740 1 client.25702 shutdown 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.886+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 msgr2=0x5579edde63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.886+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde63b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f079c006010 tx=0x7f079c003ac0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 shutdown_connections 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde9330 0x5579edde6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579eddf9760 0x5579eddfbb70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/2214273252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579edde63b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 >> 172.21.15.22:0/2214273252 conn(0x5579eddda560 msgr2=0x5579edddc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f078ea00640 0 client.25702 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:09.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 shutdown_connections 2024-03-24T06:40:09.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 1 -- 172.21.15.22:0/2214273252 wait complete. 2024-03-24T06:40:09.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.887+0000 7f07b2c5d740 20 client.25702 trim_cache size 0 max 16384 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 Processor -- start 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 -- start start 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 0x5579ede01450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 0x5579edddbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579eddd57d0 con 0x5579edde0750 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579eddd89b0 con 0x5579edde9330 2024-03-24T06:40:09.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579edde4900 con 0x5579edddfe30 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.907+0000 7f07ad600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ae000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 0x5579edddbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45074/0 (socket says 172.21.15.22:45074) 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 -- 172.21.15.22:0/1900434306 learned_addr learned my addr 172.21.15.22:0/1900434306 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07acc00640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 0x5579ede01450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 -- 172.21.15.22:0/1900434306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 msgr2=0x5579edddbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 0x5579edddbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 -- 172.21.15.22:0/1900434306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 msgr2=0x5579ede01450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 0x5579ede01450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 -- 172.21.15.22:0/1900434306 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579edce1ae0 con 0x5579edddfe30 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ae000640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 0x5579edddbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:09.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07acc00640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 0x5579ede01450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:09.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.908+0000 7f07ad600640 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f079c005ec0 tx=0x7f079c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbf7ed31d40e882a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.909+0000 7f0799c00640 1 -- 172.21.15.22:0/1900434306 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c011dc0 con 0x5579edddfe30 2024-03-24T06:40:09.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.909+0000 7f0799c00640 1 -- 172.21.15.22:0/1900434306 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c003d70 con 0x5579edddfe30 2024-03-24T06:40:09.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.909+0000 7f0799c00640 1 -- 172.21.15.22:0/1900434306 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c0052a0 con 0x5579edddfe30 2024-03-24T06:40:09.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- 172.21.15.22:0/1900434306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 msgr2=0x5579eddf4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f079c005ec0 tx=0x7f079c009b30 comp rx=0 tx=0).stop 2024-03-24T06:40:09.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- 172.21.15.22:0/1900434306 shutdown_connections 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde9330 0x5579edddbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edde0750 0x5579ede01450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/1900434306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edddfe30 0x5579eddf4bc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- 172.21.15.22:0/1900434306 >> 172.21.15.22:0/1900434306 conn(0x5579eddda560 msgr2=0x5579edddb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- 172.21.15.22:0/1900434306 shutdown_connections 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- 172.21.15.22:0/1900434306 wait complete. 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.910+0000 7f07b2c5d740 1 -- start start 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 Processor -- start 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edddfe30 0x5579eddf14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edde9330 0x5579edde31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579edde38c0 con 0x5579edddfe30 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579edde3a30 con 0x5579edde0750 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.911+0000 7f07b2c5d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579ede5ea80 con 0x5579edde9330 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07b2c5d740 10 client.0 _ll_register_callbacks cb 0x7f079c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07ae000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edde9330 0x5579edde31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37238/0 (socket says 172.21.15.22:37238) 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 -- 172.21.15.22:0/3831037447 learned_addr learned my addr 172.21.15.22:0/3831037447 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07ad600640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edddfe30 0x5579eddf14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 -- 172.21.15.22:0/3831037447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edde9330 msgr2=0x5579edde31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edde9330 0x5579edde31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 -- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edddfe30 msgr2=0x5579eddf14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edddfe30 0x5579eddf14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 -- 172.21.15.22:0/3831037447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a000f960 con 0x5579edde0750 2024-03-24T06:40:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f07acc00640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07940bc310 tx=0x7f07940dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d229dd982c83c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.912+0000 7f078ea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940bbf70 con 0x5579edde0750 2024-03-24T06:40:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f07940bcd10 con 0x5579edde0750 2024-03-24T06:40:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f079c004510 con 0x5579edde0750 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07940bc610 con 0x5579edde0750 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f07b2c5d740 10 client.25661 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579edd45660 con 0x5579edde0750 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c004510 con 0x5579edde0750 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.913+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f07940ed5c0 con 0x5579edde0750 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 10 client.25661 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 10 client.25661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 20 client.25661 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579ede5fe30 0x5579ede62220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:09.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 4 client.25661 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5579eda46860 con 0x5579ede5fe30 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.914+0000 7f07ad600640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579ede5fe30 0x5579ede62220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.915+0000 7f078ea00640 10 client.25661 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.915+0000 7f07ad600640 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579ede5fe30 0x5579ede62220 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dfed86bf46b29d2 server_cookie=752b4f260807136a in_seq=0 out_seq=0 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.916+0000 7f078ea00640 1 -- 172.21.15.22:0/3831037447 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5579eda46860 con 0x5579ede5fe30 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.916+0000 7f078ea00640 10 client.25661 handle_command_reply: tid=0 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.916+0000 7f07b2c5d740 1 client.25661 shutdown 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 msgr2=0x5579edde2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f07940bc310 tx=0x7f07940dc040 comp rx=0 tx=0).stop 2024-03-24T06:40:09.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 shutdown_connections 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579edde9330 0x5579edde31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579ede5fe30 0x5579ede62220 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579edde0750 0x5579edde2c80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 --2- 172.21.15.22:0/3831037447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579edddfe30 0x5579eddf14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 >> 172.21.15.22:0/3831037447 conn(0x5579eddda560 msgr2=0x5579eddd95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.917+0000 7f078ea00640 0 client.25661 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.918+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 shutdown_connections 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.918+0000 7f07b2c5d740 1 -- 172.21.15.22:0/3831037447 wait complete. 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.918+0000 7f07b2c5d740 20 client.25661 trim_cache size 0 max 16384 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "82146505-7638-470d-9d0d-dacc5cffd4f7", 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:40:09.920 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:40:09.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 msgr2=0x7f07900a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 0x7f07900a4800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07a0012640 tx=0x7f07a0009500 comp rx=0 tx=0).stop 2024-03-24T06:40:09.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 msgr2=0x7f07a810f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:09.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0794037040 tx=0x7f07940045b0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 shutdown_connections 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07900a2340 0x7f07900a4800 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07a81098b0 0x7f07a81a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a8104260 0x7f07a810f8a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 --2- 172.21.15.22:0/3162196479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07a81038a0 0x7f07a810f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 >> 172.21.15.22:0/3162196479 conn(0x7f07a80ff350 msgr2=0x7f07a80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.919+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 shutdown_connections 2024-03-24T06:40:09.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:09.920+0000 7f07b0000640 1 -- 172.21.15.22:0/3162196479 wait complete. 2024-03-24T06:40:09.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:40:09.938 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:40:10.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.120+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8f0819f3e0 con 0x7f8ef00a2330 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 -- 172.21.15.22:0/3784529769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4076560 msgr2=0x7f18b4112010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 --2- 172.21.15.22:0/3784529769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4076560 0x7f18b4112010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009930 tx=0x7f18a002bc20 comp rx=0 tx=0).stop 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 -- 172.21.15.22:0/3784529769 shutdown_connections 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 --2- 172.21.15.22:0/3784529769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4112550 0x7f18b4114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 --2- 172.21.15.22:0/3784529769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4076560 0x7f18b4112010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 --2- 172.21.15.22:0/3784529769 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4075c40 0x7f18b4076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 -- 172.21.15.22:0/3784529769 >> 172.21.15.22:0/3784529769 conn(0x7f18b40ff380 msgr2=0x7f18b41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 -- 172.21.15.22:0/3784529769 shutdown_connections 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.148+0000 7f18bc600640 1 -- 172.21.15.22:0/3784529769 wait complete. 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.150+0000 7f18bc600640 1 Processor -- start 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.150+0000 7f18bc600640 1 -- start start 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.150+0000 7f18bc600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18bc600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4076560 0x7f18b419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18bc600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4112550 0x7f18b41a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18bc600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41a3230 con 0x7f18b4076560 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18bc600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41a33a0 con 0x7f18b4112550 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18bc600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41a3560 con 0x7f18b4075c40 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4076560 0x7f18b419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45102/0 (socket says 172.21.15.22:45102) 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 -- 172.21.15.22:0/419024972 learned_addr learned my addr 172.21.15.22:0/419024972 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18ba600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4112550 0x7f18b41a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 -- 172.21.15.22:0/419024972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4112550 msgr2=0x7f18b41a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4112550 0x7f18b41a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 -- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4076560 msgr2=0x7f18b419e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4076560 0x7f18b419e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.151+0000 7f18b9c00640 1 -- 172.21.15.22:0/419024972 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a0009560 con 0x7f18b4075c40 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18b9c00640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00d950 tx=0x7f18ac00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18b2000640 1 -- 172.21.15.22:0/419024972 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac009c30 con 0x7f18b4075c40 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18b2000640 1 -- 172.21.15.22:0/419024972 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18ac009d90 con 0x7f18b4075c40 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b41a3840 con 0x7f18b4075c40 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18b2000640 1 -- 172.21.15.22:0/419024972 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac00f7e0 con 0x7f18b4075c40 2024-03-24T06:40:10.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.152+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b41ab0c0 con 0x7f18b4075c40 2024-03-24T06:40:10.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 Processor -- start 2024-03-24T06:40:10.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18b2000640 1 -- 172.21.15.22:0/419024972 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f18ac010ad0 con 0x7f18b4075c40 2024-03-24T06:40:10.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 -- start start 2024-03-24T06:40:10.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e5c5e790 con 0x5636e5cf0a60 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e5c5bcc0 con 0x5636e5cefd80 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.157+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e5c574a0 con 0x5636e5cf90f0 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45110/0 (socket says 172.21.15.22:45110) 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 -- 172.21.15.22:0/843135992 learned_addr learned my addr 172.21.15.22:0/843135992 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18ba600640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9c00640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 -- 172.21.15.22:0/843135992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 msgr2=0x5636e5cf0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 -- 172.21.15.22:0/843135992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 msgr2=0x5636e5cef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cef7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.158+0000 7f18b9200640 1 -- 172.21.15.22:0/843135992 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e5c18c20 con 0x5636e5cf90f0 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.159+0000 7f18ba600640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.159+0000 7f18b9c00640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.159+0000 7f18b9200640 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005ab0 tx=0x7f18a00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee785f7087a5eec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.159+0000 7f18aaa00640 1 -- 172.21.15.22:0/843135992 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a002dea0 con 0x5636e5cf90f0 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.159+0000 7f18aaa00640 1 -- 172.21.15.22:0/843135992 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a00048b0 con 0x5636e5cf90f0 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.160+0000 7f18aaa00640 1 -- 172.21.15.22:0/843135992 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0005050 con 0x5636e5cf90f0 2024-03-24T06:40:10.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.160+0000 7f18b2000640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 0x7f18980a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.160+0000 7f18b2000640 1 -- 172.21.15.22:0/419024972 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18ac0ce1b0 con 0x7f18b4075c40 2024-03-24T06:40:10.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.160+0000 7f18ba600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 0x7f18980a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 -- 172.21.15.22:0/843135992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 msgr2=0x5636e5cfb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005ab0 tx=0x7f18a00046c0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 -- 172.21.15.22:0/843135992 shutdown_connections 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cfb4e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/843135992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 -- 172.21.15.22:0/843135992 >> 172.21.15.22:0/843135992 conn(0x5636e5cb6b30 msgr2=0x5636e5cea7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 -- 172.21.15.22:0/843135992 shutdown_connections 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18ba600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 0x7f18980a4780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18b406abc0 tx=0x7f18a4009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.161+0000 7f18bf3fc740 1 -- 172.21.15.22:0/843135992 wait complete. 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.162+0000 7f18bf3fc740 1 -- start start 2024-03-24T06:40:10.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.162+0000 7f18bf3fc740 1 Processor -- start 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cf6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e5bfbf20 con 0x5636e5cf0a60 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e5bfbbc0 con 0x5636e5cefd80 2024-03-24T06:40:10.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e59c51c0 con 0x5636e5cf90f0 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18bf3fc740 10 client.0 _ll_register_callbacks cb 0x5636e5967be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18ba600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18b9200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cf6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18ba600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45124/0 (socket says 172.21.15.22:45124) 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.163+0000 7f18ba600640 1 -- 172.21.15.22:0/1336965319 learned_addr learned my addr 172.21.15.22:0/1336965319 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 -- 172.21.15.22:0/1336965319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 msgr2=0x5636e5cf60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 -- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 msgr2=0x5636e5cf6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cf6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 -- 172.21.15.22:0/1336965319 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b41153d0 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f18ba600640 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f18a4003f40 tx=0x7f18a4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a772650770c6c39e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f189f400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4017070 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.164+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18a400f890 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.165+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5636e5c4f160 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.165+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18a400fa50 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.165+0000 7f18bf3fc740 10 client.25720 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.165+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5636e5c55b20 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.165+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5636e5c4f160 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f18a4029ca0 con 0x5636e5cf90f0 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 10 client.25720 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:10.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 10 client.25720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 20 client.25720 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d09610 0x5636e5d0ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 4 client.25720 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.166+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e59567e0 con 0x5636e5d09610 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.167+0000 7f18b9c00640 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d09610 0x5636e5d0ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.167+0000 7f18b9c00640 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d09610 0x5636e5d0ba00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2bc2cbc957812d server_cookie=89616cd310dbc674 in_seq=0 out_seq=0 2024-03-24T06:40:10.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.167+0000 7f189f400640 10 client.25720 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f189f400640 1 -- 172.21.15.22:0/1336965319 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636e59567e0 con 0x5636e5d09610 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f189f400640 10 client.25720 handle_command_reply: tid=0 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 client.25720 shutdown 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 msgr2=0x5636e5cf6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f18a4003f40 tx=0x7f18a4004ab0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 shutdown_connections 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf6b50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5cf6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d09610 0x5636e5d0ba00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/1336965319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cefd80 0x5636e5cf60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.169+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 >> 172.21.15.22:0/1336965319 conn(0x5636e5cb6b30 msgr2=0x5636e5cec520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.168+0000 7f189f400640 0 client.25720 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:10.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.169+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 shutdown_connections 2024-03-24T06:40:10.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.169+0000 7f18bf3fc740 1 -- 172.21.15.22:0/1336965319 wait complete. 2024-03-24T06:40:10.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.169+0000 7f18bf3fc740 20 client.25720 trim_cache size 0 max 16384 2024-03-24T06:40:10.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.198+0000 7f18bf3fc740 1 Processor -- start 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 -- start start 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5d049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e5cf1a70 con 0x5636e5cf0a60 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e59567e0 con 0x5636e5cf90f0 2024-03-24T06:40:10.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e5ce4c90 con 0x5636e5cefd80 2024-03-24T06:40:10.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18ba600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18b9200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5d049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18ba600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37278/0 (socket says 172.21.15.22:37278) 2024-03-24T06:40:10.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18ba600640 1 -- 172.21.15.22:0/3122387721 learned_addr learned my addr 172.21.15.22:0/3122387721 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.199+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45136/0 (socket says 172.21.15.22:45136) 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 -- 172.21.15.22:0/3122387721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 msgr2=0x5636e5cf8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 -- 172.21.15.22:0/3122387721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 msgr2=0x5636e5d049e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5d049e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 -- 172.21.15.22:0/3122387721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e5bf19a0 con 0x5636e5cf90f0 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18b9c00640 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.200+0000 7f18ba600640 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18a401f6a0 tx=0x7f18a4009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b9a783f0a2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.201+0000 7f18aa000640 1 -- 172.21.15.22:0/3122387721 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a401f920 con 0x5636e5cf90f0 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.201+0000 7f18aa000640 1 -- 172.21.15.22:0/3122387721 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a4004920 con 0x5636e5cf90f0 2024-03-24T06:40:10.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.201+0000 7f18aa000640 1 -- 172.21.15.22:0/3122387721 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a400fe00 con 0x5636e5cf90f0 2024-03-24T06:40:10.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.201+0000 7f18bf3fc740 1 -- 172.21.15.22:0/3122387721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 msgr2=0x5636e5d112e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.201+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18a401f6a0 tx=0x7f18a4009ea0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 -- 172.21.15.22:0/3122387721 shutdown_connections 2024-03-24T06:40:10.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf90f0 0x5636e5d112e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cf0a60 0x5636e5d049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/3122387721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cefd80 0x5636e5cf8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 -- 172.21.15.22:0/3122387721 >> 172.21.15.22:0/3122387721 conn(0x5636e5cb6b30 msgr2=0x5636e5cea300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 -- 172.21.15.22:0/3122387721 shutdown_connections 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 -- 172.21.15.22:0/3122387721 wait complete. 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.202+0000 7f18bf3fc740 1 -- start start 2024-03-24T06:40:10.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 Processor -- start 2024-03-24T06:40:10.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 0x5636e5cf3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf0a60 0x5636e5cf3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e5d01490 con 0x5636e5cefd80 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e5d01600 con 0x5636e5cf0a60 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.203+0000 7f18bf3fc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e5d017c0 con 0x5636e5cf90f0 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18bf3fc740 10 client.0 _ll_register_callbacks cb 0x7f18a4009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18ba600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18b9c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 0x5636e5cf3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18ba600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45152/0 (socket says 172.21.15.22:45152) 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18ba600640 1 -- 172.21.15.22:0/2980637081 learned_addr learned my addr 172.21.15.22:0/2980637081 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18b9c00640 1 -- 172.21.15.22:0/2980637081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 msgr2=0x5636e5cf9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.204+0000 7f18b9c00640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18b9c00640 1 -- 172.21.15.22:0/2980637081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf0a60 msgr2=0x5636e5cf3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18b9200640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf0a60 0x5636e5cf3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18b9c00640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf0a60 0x5636e5cf3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18b9c00640 1 -- 172.21.15.22:0/2980637081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b41ab0c0 con 0x5636e5cefd80 2024-03-24T06:40:10.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18ba600640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:10.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f8f02a00640 1 -- 172.21.15.22:0/1622549333 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8f0819f3e0 con 0x7f8ef00a2330 2024-03-24T06:40:10.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18b9c00640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 0x5636e5cf3510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0cb480 tx=0x7f18ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937b13de9cc0fca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f189f400640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:40:10.207 INFO:teuthology.orchestra.run.smithi022.stdout:Schedule set for path / 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac0a2e10 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18ac098a20 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.205+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a00048b0 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.206+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18ac096d90 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.206+0000 7f18bf3fc740 10 client.15549 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.206+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7f18a402e040 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.206+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a00048b0 con 0x5636e5cefd80 2024-03-24T06:40:10.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.206+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f18ac0cb7c0 con 0x5636e5cefd80 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 10 client.15549 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 20 client.15549 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d70380 0x5636e5d727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 4 client.15549 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.207+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636e5993720 con 0x5636e5d70380 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.208+0000 7f18b9200640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d70380 0x5636e5d727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.208+0000 7f18b9200640 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d70380 0x5636e5d727a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26079697e041b899 server_cookie=8dd9143d161d6dad in_seq=0 out_seq=0 2024-03-24T06:40:10.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.208+0000 7f189f400640 10 client.15549 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 msgr2=0x7f8ef00a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 0x7f8ef00a47f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f0810f040 tx=0x7f8ef8005f90 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 msgr2=0x7f8f08199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8efc000c00 tx=0x7f8efc02c6c0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 shutdown_connections 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ef00a2330 0x7f8ef00a47f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f0810dd30 0x7f8f0819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f08074460 0x7f8f0819a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 --2- 172.21.15.22:0/1622549333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f08073b40 0x7f8f08199d80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 >> 172.21.15.22:0/1622549333 conn(0x7f8f080fd060 msgr2=0x7f8f080fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 shutdown_connections 2024-03-24T06:40:10.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f8f0f200640 1 -- 172.21.15.22:0/1622549333 wait complete. 2024-03-24T06:40:10.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f189f400640 1 -- 172.21.15.22:0/2980637081 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5636e5993720 con 0x5636e5d70380 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f189f400640 10 client.15549 handle_command_reply: tid=0 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f18bf3fc740 1 client.15549 shutdown 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 msgr2=0x5636e5cf3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.209+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 0x5636e5cf3510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f18ac0cb480 tx=0x7f18ac000bc0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 shutdown_connections 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e5cf90f0 0x5636e5cf9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e5d70380 0x5636e5d727a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e5cf0a60 0x5636e5cf3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 --2- 172.21.15.22:0/2980637081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e5cefd80 0x5636e5cf3510 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 >> 172.21.15.22:0/2980637081 conn(0x5636e5cb6b30 msgr2=0x5636e5ce8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f189f400640 0 client.15549 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 shutdown_connections 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 1 -- 172.21.15.22:0/2980637081 wait complete. 2024-03-24T06:40:10.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.210+0000 7f18bf3fc740 20 client.15549 trim_cache size 0 max 16384 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "82146505-7638-470d-9d0d-dacc5cffd4f7": { 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "82146505-7638-470d-9d0d-dacc5cffd4f7", 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:40:10.213 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:40:10.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 msgr2=0x7f18980a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 0x7f18980a4780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18b406abc0 tx=0x7f18a4009500 comp rx=0 tx=0).stop 2024-03-24T06:40:10.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 msgr2=0x7f18b419e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00d950 tx=0x7f18ac00de10 comp rx=0 tx=0).stop 2024-03-24T06:40:10.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 shutdown_connections 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18980a22c0 0x7f18980a4780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b4112550 0x7f18b41a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18b4076560 0x7f18b419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 --2- 172.21.15.22:0/419024972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18b4075c40 0x7f18b419e3c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 >> 172.21.15.22:0/419024972 conn(0x7f18b40ff380 msgr2=0x7f18b4113c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 shutdown_connections 2024-03-24T06:40:10.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.211+0000 7f18bc600640 1 -- 172.21.15.22:0/419024972 wait complete. 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/706499363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.225 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1848547863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.227 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-24T06:40:10.232 INFO:tasks.cephfs.filesystem:scrub status for tag:82146505-7638-470d-9d0d-dacc5cffd4f7 - {'path': '/', 'tag': '82146505-7638-470d-9d0d-dacc5cffd4f7', 'options': 'recursive,force'} 2024-03-24T06:40:10.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 -- 172.21.15.22:0/2471841478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c10dba0 msgr2=0x7f520c10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 --2- 172.21.15.22:0/2471841478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c10dba0 0x7f520c10df80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009900 tx=0x7f51fc02bc50 comp rx=0 tx=0).stop 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 -- 172.21.15.22:0/2471841478 shutdown_connections 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 --2- 172.21.15.22:0/2471841478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c108610 0x7f520c1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 --2- 172.21.15.22:0/2471841478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c107c50 0x7f520c1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 --2- 172.21.15.22:0/2471841478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c10dba0 0x7f520c10df80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.414+0000 7f520b200640 1 -- 172.21.15.22:0/2471841478 >> 172.21.15.22:0/2471841478 conn(0x7f520c0774f0 msgr2=0x7f520c106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.415+0000 7f520b200640 1 -- 172.21.15.22:0/2471841478 shutdown_connections 2024-03-24T06:40:10.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.415+0000 7f520b200640 1 -- 172.21.15.22:0/2471841478 wait complete. 2024-03-24T06:40:10.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.415+0000 7f520b200640 1 Processor -- start 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 -- start start 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c108610 0x7f520c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 0x7f520c1a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1a3010 con 0x7f520c108610 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1a3180 con 0x7f520c107c50 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1a3340 con 0x7f520c10dba0 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f5209e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f5209400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c108610 0x7f520c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f5209e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37314/0 (socket says 172.21.15.22:37314) 2024-03-24T06:40:10.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f5209e00640 1 -- 172.21.15.22:0/2812241480 learned_addr learned my addr 172.21.15.22:0/2812241480 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.416+0000 7f520a800640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 0x7f520c1a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 -- 172.21.15.22:0/2812241480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 msgr2=0x7f520c1a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 0x7f520c1a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 -- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c108610 msgr2=0x7f520c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c108610 0x7f520c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 -- 172.21.15.22:0/2812241480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc009560 con 0x7f520c107c50 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f520a800640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 0x7f520c1a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5209e00640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f51fc000c00 tx=0x7f51fc02c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc004490 con 0x7f520c107c50 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc0052d0 con 0x7f520c107c50 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c1a35c0 con 0x7f520c107c50 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.417+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc003dc0 con 0x7f520c107c50 2024-03-24T06:40:10.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.418+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1aaea0 con 0x7f520c107c50 2024-03-24T06:40:10.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.419+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c1091e0 con 0x7f520c107c50 2024-03-24T06:40:10.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.420+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f51fc0035f0 con 0x7f520c107c50 2024-03-24T06:40:10.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.422+0000 7f5202a00640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 0x7f51ec0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.422+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f51fc0f0c20 con 0x7f520c107c50 2024-03-24T06:40:10.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.422+0000 7f5209400640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 0x7f51ec0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.423+0000 7f5209400640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 0x7f51ec0a4770 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520c10d1d0 tx=0x7f51f4009760 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.426+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f51fc0b9490 con 0x7f520c107c50 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/706499363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1848547863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: Updating smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/706499363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:40:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1848547863' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:40:10.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.653+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f520c117cd0 con 0x7f51ec0a22b0 2024-03-24T06:40:10.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.758+0000 7f5202a00640 1 -- 172.21.15.22:0/2812241480 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f520c117cd0 con 0x7f51ec0a22b0 2024-03-24T06:40:10.761 INFO:teuthology.orchestra.run.smithi022.stdout:Retention added to path / 2024-03-24T06:40:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 msgr2=0x7f51ec0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 0x7f51ec0a4770 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f520c10d1d0 tx=0x7f51f4009760 comp rx=0 tx=0).stop 2024-03-24T06:40:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 msgr2=0x7f520c19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f51fc000c00 tx=0x7f51fc02c740 comp rx=0 tx=0).stop 2024-03-24T06:40:10.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 shutdown_connections 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f51ec0a22b0 0x7f51ec0a4770 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f520c10dba0 0x7f520c1a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f520c108610 0x7f520c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 --2- 172.21.15.22:0/2812241480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520c107c50 0x7f520c19e150 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 >> 172.21.15.22:0/2812241480 conn(0x7f520c0774f0 msgr2=0x7f520c0784a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 shutdown_connections 2024-03-24T06:40:10.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.762+0000 7f520b200640 1 -- 172.21.15.22:0/2812241480 wait complete. 2024-03-24T06:40:10.781 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T06:40:10.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 -- 172.21.15.22:0/1898448197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 msgr2=0x7f186c10d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 --2- 172.21.15.22:0/1898448197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c10d7f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1854009900 tx=0x7f185402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 -- 172.21.15.22:0/1898448197 shutdown_connections 2024-03-24T06:40:10.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 --2- 172.21.15.22:0/1898448197 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 0x7f186c110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 --2- 172.21.15.22:0/1898448197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c10d7f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 --2- 172.21.15.22:0/1898448197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c073f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 -- 172.21.15.22:0/1898448197 >> 172.21.15.22:0/1898448197 conn(0x7f186c0fd060 msgr2=0x7f186c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:10.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 -- 172.21.15.22:0/1898448197 shutdown_connections 2024-03-24T06:40:10.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.974+0000 7f186b200640 1 -- 172.21.15.22:0/1898448197 wait complete. 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.975+0000 7f186b200640 1 Processor -- start 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.975+0000 7f186b200640 1 -- start start 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 0x7f186c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f186c19ec20 con 0x7f186c10dd30 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f186c19ed90 con 0x7f186c073b40 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f186c19ef50 con 0x7f186c074460 2024-03-24T06:40:10.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37328/0 (socket says 172.21.15.22:37328) 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 -- 172.21.15.22:0/2691025444 learned_addr learned my addr 172.21.15.22:0/2691025444 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1861400640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f186a800640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 0x7f186c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 -- 172.21.15.22:0/2691025444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 msgr2=0x7f186c19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 -- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 msgr2=0x7f186c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 0x7f186c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.976+0000 7f1869e00640 1 -- 172.21.15.22:0/2691025444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1854009560 con 0x7f186c073b40 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f1869e00640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f185c00ebc0 tx=0x7f185c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c0099b0 con 0x7f186c073b40 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186c19f230 con 0x7f186c073b40 2024-03-24T06:40:10.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f185c00eec0 con 0x7f186c073b40 2024-03-24T06:40:10.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c018b60 con 0x7f186c073b40 2024-03-24T06:40:10.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.977+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186c1a6ab0 con 0x7f186c073b40 2024-03-24T06:40:10.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.978+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186c0751f0 con 0x7f186c073b40 2024-03-24T06:40:10.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.980+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f185c018cc0 con 0x7f186c073b40 2024-03-24T06:40:10.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.981+0000 7f1863400640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 0x7f18500a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:10.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.981+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f185c014070 con 0x7f186c073b40 2024-03-24T06:40:10.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.982+0000 7f1861400640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 0x7f18500a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:10.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.982+0000 7f1861400640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 0x7f18500a47f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1854005ab0 tx=0x7f1854005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:10.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:10.983+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f185c0973e0 con 0x7f186c073b40 2024-03-24T06:40:11.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.219+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f186c108f50 con 0x7f18500a2330 2024-03-24T06:40:11.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.231+0000 7f1863400640 1 -- 172.21.15.22:0/2691025444 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f186c108f50 con 0x7f18500a2330 2024-03-24T06:40:11.231 INFO:teuthology.orchestra.run.smithi022.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T06:40:10", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 msgr2=0x7f18500a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 0x7f18500a47f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1854005ab0 tx=0x7f1854005a20 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 msgr2=0x7f186c199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f185c00ebc0 tx=0x7f185c00c6a0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 shutdown_connections 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18500a2330 0x7f18500a47f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f186c10dd30 0x7f186c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c074460 0x7f186c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 --2- 172.21.15.22:0/2691025444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186c073b40 0x7f186c199d60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 >> 172.21.15.22:0/2691025444 conn(0x7f186c0fd060 msgr2=0x7f186c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:11.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 shutdown_connections 2024-03-24T06:40:11.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.235+0000 7f186b200640 1 -- 172.21.15.22:0/2691025444 wait complete. 2024-03-24T06:40:11.254 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:11.323 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:11 smithi022 ceph-mon[32003]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 114 B/s wr, 19 op/s 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:11.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:11 smithi143 ceph-mon[36520]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 114 B/s wr, 19 op/s 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.453+0000 7ff7c1000640 1 -- 172.21.15.22:0/724590478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc075c40 msgr2=0x7ff7bc076020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.453+0000 7ff7c1000640 1 --2- 172.21.15.22:0/724590478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc075c40 0x7ff7bc076020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009930 tx=0x7ff7ac02bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 -- 172.21.15.22:0/724590478 shutdown_connections 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 --2- 172.21.15.22:0/724590478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1147c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 --2- 172.21.15.22:0/724590478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc076560 0x7ff7bc111e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 --2- 172.21.15.22:0/724590478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc075c40 0x7ff7bc076020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 -- 172.21.15.22:0/724590478 >> 172.21.15.22:0/724590478 conn(0x7ff7bc0ff200 msgr2=0x7ff7bc101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 -- 172.21.15.22:0/724590478 shutdown_connections 2024-03-24T06:40:11.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.454+0000 7ff7c1000640 1 -- 172.21.15.22:0/724590478 wait complete. 2024-03-24T06:40:11.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.455+0000 7ff7c1000640 1 Processor -- start 2024-03-24T06:40:11.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.455+0000 7ff7c1000640 1 -- start start 2024-03-24T06:40:11.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc075c40 0x7ff7bc19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc076560 0x7ff7bc19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc1a33e0 con 0x7ff7bc075c40 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc1a3550 con 0x7ff7bc1123d0 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7c1000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc1a3710 con 0x7ff7bc076560 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7bb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc075c40 0x7ff7bc19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7bbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7baa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc076560 0x7ff7bc19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7bbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:37352/0 (socket says 172.21.15.22:37352) 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.456+0000 7ff7bbe00640 1 -- 172.21.15.22:0/118135663 learned_addr learned my addr 172.21.15.22:0/118135663 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 -- 172.21.15.22:0/118135663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc076560 msgr2=0x7ff7bc19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc076560 0x7ff7bc19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 -- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc075c40 msgr2=0x7ff7bc19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc075c40 0x7ff7bc19e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 -- 172.21.15.22:0/118135663 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac009560 con 0x7ff7bc1123d0 2024-03-24T06:40:11.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7bbe00640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400eb40 tx=0x7ff7b400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:11.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4004b20 con 0x7ff7bc1123d0 2024-03-24T06:40:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc1a39f0 con 0x7ff7bc1123d0 2024-03-24T06:40:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b4004c80 con 0x7ff7bc1123d0 2024-03-24T06:40:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.457+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b40107b0 con 0x7ff7bc1123d0 2024-03-24T06:40:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.458+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc1ab270 con 0x7ff7bc1123d0 2024-03-24T06:40:11.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.459+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7bc04fcc0 con 0x7ff7bc1123d0 2024-03-24T06:40:11.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.460+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7ff7b4005020 con 0x7ff7bc1123d0 2024-03-24T06:40:11.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.462+0000 7ff7b3e00640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 0x7ff7a00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:11.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.462+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff7b4014070 con 0x7ff7bc1123d0 2024-03-24T06:40:11.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.462+0000 7ff7bb400640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 0x7ff7a00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:11.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.463+0000 7ff7bb400640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 0x7ff7a00a4770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac005ab0 tx=0x7ff7ac005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:11.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.466+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff7b40973e0 con 0x7ff7bc1123d0 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:11.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:11 smithi155 ceph-mon[36262]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 114 B/s wr, 19 op/s 2024-03-24T06:40:11.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.689+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff7bc06adf0 con 0x7ff7a00a22b0 2024-03-24T06:40:11.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.700+0000 7ff7b3e00640 1 -- 172.21.15.22:0/118135663 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff7bc06adf0 con 0x7ff7a00a22b0 2024-03-24T06:40:11.701 INFO:teuthology.orchestra.run.smithi022.stdout:/ 1m 6m3h 2024-03-24T06:40:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 msgr2=0x7ff7a00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 0x7ff7a00a4770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac005ab0 tx=0x7ff7ac005fd0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 msgr2=0x7ff7bc1a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:11.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400eb40 tx=0x7ff7b400c2d0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 shutdown_connections 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff7a00a22b0 0x7ff7a00a4770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7bc1123d0 0x7ff7bc1a2de0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7bc076560 0x7ff7bc19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 --2- 172.21.15.22:0/118135663 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff7bc075c40 0x7ff7bc19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.703+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 >> 172.21.15.22:0/118135663 conn(0x7ff7bc0ff200 msgr2=0x7ff7bc1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.704+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 shutdown_connections 2024-03-24T06:40:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:11.704+0000 7ff7c1000640 1 -- 172.21.15.22:0/118135663 wait complete. 2024-03-24T06:40:11.724 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-24T06:40:11.820 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-24T06:40:11.831 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T06:40:11.864 INFO:tasks.workunit:Pulling workunits from ref d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T06:40:11.864 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T06:40:11.865 INFO:tasks.workunit:timeout=3h 2024-03-24T06:40:11.865 INFO:tasks.workunit:cleanup=True 2024-03-24T06:40:11.865 DEBUG:teuthology.orchestra.run.smithi022:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 06:37:33.339865996 +0000 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 06:37:33.339865996 +0000 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 06:39:57.934452780 +0000 2024-03-24T06:40:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 06:37:33.339865996 +0000 2024-03-24T06:40:11.921 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T06:40:11.921 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T06:40:12.028 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T06:40:12.078 INFO:tasks.workunit.client.0.smithi022.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T06:40:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:12 smithi022 ceph-mon[32003]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:12 smithi143 ceph-mon[36520]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:12 smithi155 ceph-mon[36262]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:13 smithi022 ceph-mon[32003]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:13 smithi022 ceph-mon[32003]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:13 smithi022 ceph-mon[32003]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:13 smithi143 ceph-mon[36520]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:13 smithi143 ceph-mon[36520]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:13 smithi143 ceph-mon[36520]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:13 smithi155 ceph-mon[36262]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:13 smithi155 ceph-mon[36262]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:40:13.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:13 smithi155 ceph-mon[36262]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-03-24T06:40:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:14 smithi155 ceph-mon[36262]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:14.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:14 smithi022 ceph-mon[32003]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:14 smithi143 ceph-mon[36520]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:16.479 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:16 smithi143 ceph-mon[36520]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:16 smithi155 ceph-mon[36262]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:16.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:16 smithi022 ceph-mon[32003]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-03-24T06:40:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:18 smithi155 ceph-mon[36262]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-24T06:40:18.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:18 smithi022 ceph-mon[32003]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-24T06:40:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:18 smithi143 ceph-mon[36520]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-24T06:40:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:20 smithi155 ceph-mon[36262]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T06:40:20.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:20 smithi022 ceph-mon[32003]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T06:40:20.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:20 smithi143 ceph-mon[36520]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T06:40:23.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:23 smithi155 ceph-mon[36262]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T06:40:23.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:23 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:40:23.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:23 smithi022 ceph-mon[32003]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T06:40:23.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:23 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:40:23.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:23 smithi143 ceph-mon[36520]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T06:40:23.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:23 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:40:25.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:25 smithi155 ceph-mon[36262]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:40:25.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:25 smithi022 ceph-mon[32003]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:40:25.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:25 smithi143 ceph-mon[36520]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:40:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:27 smithi155 ceph-mon[36262]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-24T06:40:27.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:27 smithi022 ceph-mon[32003]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-24T06:40:27.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:27 smithi143 ceph-mon[36520]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-24T06:40:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:29 smithi155 ceph-mon[36262]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 767 B/s wr, 15 op/s 2024-03-24T06:40:29.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:29 smithi022 ceph-mon[32003]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 767 B/s wr, 15 op/s 2024-03-24T06:40:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:29 smithi143 ceph-mon[36520]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 767 B/s wr, 15 op/s 2024-03-24T06:40:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:31 smithi155 ceph-mon[36262]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:31.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:31 smithi022 ceph-mon[32003]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:31.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:31 smithi143 ceph-mon[36520]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:33 smithi155 ceph-mon[36262]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:33.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:33 smithi022 ceph-mon[32003]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:33 smithi143 ceph-mon[36520]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:35.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:35 smithi155 ceph-mon[36262]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:35.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:35 smithi022 ceph-mon[32003]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:35.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:35 smithi143 ceph-mon[36520]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:37.378 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:37 smithi022 ceph-mon[32003]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:40:37.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:37 smithi155 ceph-mon[36262]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:40:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:37 smithi143 ceph-mon[36520]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T06:40:39.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:39 smithi155 ceph-mon[36262]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T06:40:39.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:39 smithi022 ceph-mon[32003]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T06:40:39.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:39 smithi143 ceph-mon[36520]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T06:40:40.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:40:40.234 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:40:40.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 -- 172.21.15.22:0/3711384384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff010bb60 msgr2=0x7f6ff010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 --2- 172.21.15.22:0/3711384384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff010bb60 0x7f6ff010e000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009be0 tx=0x7f6fe802bf80 comp rx=0 tx=0).stop 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 -- 172.21.15.22:0/3711384384 shutdown_connections 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 --2- 172.21.15.22:0/3711384384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff010bb60 0x7f6ff010e000 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 --2- 172.21.15.22:0/3711384384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 0x7f6ff010b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 --2- 172.21.15.22:0/3711384384 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff0102740 0x7f6ff0102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.430+0000 7f6ff6600640 1 -- 172.21.15.22:0/3711384384 >> 172.21.15.22:0/3711384384 conn(0x7f6ff00fbf60 msgr2=0x7f6ff00fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.431+0000 7f6ff6600640 1 -- 172.21.15.22:0/3711384384 shutdown_connections 2024-03-24T06:40:40.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.431+0000 7f6ff6600640 1 -- 172.21.15.22:0/3711384384 wait complete. 2024-03-24T06:40:40.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 Processor -- start 2024-03-24T06:40:40.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 -- start start 2024-03-24T06:40:40.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 0x7f6ff019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 0x7f6ff019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff019ecc0 con 0x7f6ff010bb60 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41488/0 (socket says 172.21.15.22:41488) 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff019ee30 con 0x7f6ff0103060 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff5200640 1 -- 172.21.15.22:0/2137001686 learned_addr learned my addr 172.21.15.22:0/2137001686 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.432+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff019eff0 con 0x7f6ff0102740 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6fefe00640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 0x7f6ff019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 -- 172.21.15.22:0/2137001686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 msgr2=0x7f6ff019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 0x7f6ff019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 0x7f6ff019a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 -- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 msgr2=0x7f6ff019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 0x7f6ff019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 -- 172.21.15.22:0/2137001686 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc009560 con 0x7f6ff0102740 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 0x7f6ff019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400d950 tx=0x7f6fe400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.433+0000 7f6fed600640 1 -- 172.21.15.22:0/2137001686 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4009a30 con 0x7f6ff0102740 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.434+0000 7f6fed600640 1 -- 172.21.15.22:0/2137001686 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe4009b90 con 0x7f6ff0102740 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.434+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe8009840 con 0x7f6ff0102740 2024-03-24T06:40:40.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.434+0000 7f6fed600640 1 -- 172.21.15.22:0/2137001686 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004a00 con 0x7f6ff0102740 2024-03-24T06:40:40.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.434+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff01a6970 con 0x7f6ff0102740 2024-03-24T06:40:40.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.438+0000 7f6ffa884740 1 Processor -- start 2024-03-24T06:40:40.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.439+0000 7f6fed600640 1 -- 172.21.15.22:0/2137001686 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f6fe4009d00 con 0x7f6ff0102740 2024-03-24T06:40:40.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 -- start start 2024-03-24T06:40:40.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6fed600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 0x7f6fd80a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e45a790 con 0x55ca9e4ebf60 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e457cc0 con 0x55ca9e4ecc40 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6ffa884740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e4534a0 con 0x55ca9e4f52d0 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.440+0000 7f6fed600640 1 -- 172.21.15.22:0/2137001686 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fe40cdaa0 con 0x7f6ff0102740 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35082/0 (socket says 172.21.15.22:35082) 2024-03-24T06:40:40.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 -- 172.21.15.22:0/2672208347 learned_addr learned my addr 172.21.15.22:0/2672208347 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 -- 172.21.15.22:0/2672208347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 msgr2=0x55ca9e4f76c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f76c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 -- 172.21.15.22:0/2672208347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 msgr2=0x55ca9e4ec3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4ec3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 -- 172.21.15.22:0/2672208347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9e414c20 con 0x55ca9e4ecc40 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 0x7f6fd80a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.441+0000 7f6ff5200640 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40a21d0 tx=0x7f6fe4005680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65a0d917af9777c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6fe2000640 1 -- 172.21.15.22:0/2672208347 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4099f60 con 0x55ca9e4ecc40 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6fe2000640 1 -- 172.21.15.22:0/2672208347 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe409e0b0 con 0x55ca9e4ecc40 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6fe2000640 1 -- 172.21.15.22:0/2672208347 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40993b0 con 0x55ca9e4ecc40 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6ffa884740 1 -- 172.21.15.22:0/2672208347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 msgr2=0x55ca9e4eb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 0x7f6fd80a4780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8036040 tx=0x7f6fe8005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.442+0000 7f6ffa884740 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40a21d0 tx=0x7f6fe4005680 comp rx=0 tx=0).stop 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 -- 172.21.15.22:0/2672208347 shutdown_connections 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 --2- 172.21.15.22:0/2672208347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4eb990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 -- 172.21.15.22:0/2672208347 >> 172.21.15.22:0/2672208347 conn(0x55ca9e4b2b30 msgr2=0x55ca9e4e69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 -- 172.21.15.22:0/2672208347 shutdown_connections 2024-03-24T06:40:40.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 -- 172.21.15.22:0/2672208347 wait complete. 2024-03-24T06:40:40.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.443+0000 7f6ffa884740 1 -- start start 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6ffa884740 1 Processor -- start 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4f2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6fefe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.444+0000 7f6fefe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41500/0 (socket says 172.21.15.22:41500) 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6fefe00640 1 -- 172.21.15.22:0/920664138 learned_addr learned my addr 172.21.15.22:0/920664138 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e3f7f20 con 0x55ca9e4ebf60 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e3f7bc0 con 0x55ca9e4f52d0 2024-03-24T06:40:40.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e1c11c0 con 0x55ca9e4ecc40 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ffa884740 10 client.0 _ll_register_callbacks cb 0x55ca9e163be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5200640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 -- 172.21.15.22:0/920664138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 msgr2=0x55ca9e4f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 -- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 msgr2=0x55ca9e4f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff01a6970 con 0x55ca9e4f52d0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.445+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f2d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6fe802c4c0 tx=0x7f6fe802bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa3a532b0daacf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6fd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:40.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8004450 con 0x55ca9e4f52d0 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fe8033870 con 0x55ca9e4f52d0 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca9e44b160 con 0x55ca9e4f52d0 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6fe80045b0 con 0x55ca9e4f52d0 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6ffa884740 10 client.25703 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.446+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ca9e451b20 con 0x55ca9e4f52d0 2024-03-24T06:40:40.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca9e44b160 con 0x55ca9e4f52d0 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe803c040 con 0x55ca9e4f52d0 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 10 client.25703 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 20 client.25703 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e50e050 0x55ca9e510470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 4 client.25703 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.447+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca9e1527e0 con 0x55ca9e50e050 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.448+0000 7f6ff5200640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e50e050 0x55ca9e510470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.448+0000 7f6ff5200640 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e50e050 0x55ca9e510470 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2366610b9f41976f server_cookie=2be75bc394d64158 in_seq=0 out_seq=0 2024-03-24T06:40:40.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.448+0000 7f6fd6a00640 10 client.25703 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.449+0000 7f6fd6a00640 1 -- 172.21.15.22:0/920664138 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca9e1527e0 con 0x55ca9e50e050 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.449+0000 7f6fd6a00640 10 client.25703 handle_command_reply: tid=0 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 client.25703 shutdown 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 msgr2=0x55ca9e4f2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f2d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6fe802c4c0 tx=0x7f6fe802bf80 comp rx=0 tx=0).stop 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 shutdown_connections 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e50e050 0x55ca9e510470 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f2d80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ecc40 0x55ca9e4f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 --2- 172.21.15.22:0/920664138 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 >> 172.21.15.22:0/920664138 conn(0x55ca9e4b2b30 msgr2=0x55ca9e4e8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6fd6a00640 0 client.25703 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:40.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 shutdown_connections 2024-03-24T06:40:40.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 1 -- 172.21.15.22:0/920664138 wait complete. 2024-03-24T06:40:40.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.450+0000 7f6ffa884740 20 client.25703 trim_cache size 0 max 16384 2024-03-24T06:40:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 Processor -- start 2024-03-24T06:40:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 -- start start 2024-03-24T06:40:40.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e4f3790 con 0x55ca9e4ecc40 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e1527e0 con 0x55ca9e4f52d0 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ffa884740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e44b160 con 0x55ca9e4ebf60 2024-03-24T06:40:40.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.469+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41504/0 (socket says 172.21.15.22:41504) 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 -- 172.21.15.22:0/628831969 learned_addr learned my addr 172.21.15.22:0/628831969 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6fefe00640 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 -- 172.21.15.22:0/628831969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 msgr2=0x55ca9e4f7490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f7490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 -- 172.21.15.22:0/628831969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 msgr2=0x55ca9e5093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 -- 172.21.15.22:0/628831969 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9e3ed9a0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6fefe00640 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6ff5200640 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4098210 tx=0x7f6fe4099e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82abc148a9904a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6fe1600640 1 -- 172.21.15.22:0/628831969 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4099fe0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6fe1600640 1 -- 172.21.15.22:0/628831969 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe4096200 con 0x55ca9e4ebf60 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.470+0000 7f6fe1600640 1 -- 172.21.15.22:0/628831969 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4094ad0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 -- 172.21.15.22:0/628831969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 msgr2=0x55ca9e4f4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4098210 tx=0x7f6fe4099e60 comp rx=0 tx=0).stop 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 -- 172.21.15.22:0/628831969 shutdown_connections 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4f52d0 0x55ca9e4f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 --2- 172.21.15.22:0/628831969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4ebf60 0x55ca9e4f4a30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 -- 172.21.15.22:0/628831969 >> 172.21.15.22:0/628831969 conn(0x55ca9e4b2b30 msgr2=0x55ca9e4e7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 -- 172.21.15.22:0/628831969 shutdown_connections 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.471+0000 7f6ffa884740 1 -- 172.21.15.22:0/628831969 wait complete. 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 -- start start 2024-03-24T06:40:40.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 Processor -- start 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5009e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e500f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e505ff0 con 0x55ca9e4ecc40 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e506160 con 0x55ca9e4ebf60 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ca9e501460 con 0x55ca9e4f52d0 2024-03-24T06:40:40.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.472+0000 7f6ffa884740 10 client.0 _ll_register_callbacks cb 0x7f6fdc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35098/0 (socket says 172.21.15.22:35098) 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 -- 172.21.15.22:0/1421172651 learned_addr learned my addr 172.21.15.22:0/1421172651 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5c00640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e500f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6fefe00640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 -- 172.21.15.22:0/1421172651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 msgr2=0x55ca9e500f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e500f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 -- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 msgr2=0x55ca9e5009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 -- 172.21.15.22:0/1421172651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca9e414c20 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6fefe00640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6ff5200640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005fd0 tx=0x7f6fe4098260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d2e4d0c00d0bf3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.473+0000 7f6fd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40996b0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6fe40b8280 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe409e0b0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6fe40b8480 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 10 client.25706 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ca9e413480 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe409e0b0 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe4098e20 con 0x55ca9e4ebf60 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 10 client.25706 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:40.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 10 client.25706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 20 client.25706 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.474+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e575040 0x55ca9e577430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.475+0000 7f6ffa884740 4 client.25706 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.475+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca9e18f720 con 0x55ca9e575040 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.475+0000 7f6fefe00640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e575040 0x55ca9e577430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6fefe00640 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e575040 0x55ca9e577430 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2e5148cefc9a1b server_cookie=1368d471c1041484 in_seq=0 out_seq=0 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6fd6a00640 10 client.25706 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6fd6a00640 1 -- 172.21.15.22:0/1421172651 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ca9e18f720 con 0x55ca9e575040 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6fd6a00640 10 client.25706 handle_command_reply: tid=0 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 client.25706 shutdown 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 msgr2=0x55ca9e505a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005fd0 tx=0x7f6fe4098260 comp rx=0 tx=0).stop 2024-03-24T06:40:40.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 shutdown_connections 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ca9e4f52d0 0x55ca9e500f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ca9e4ecc40 0x55ca9e5009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ca9e575040 0x55ca9e577430 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 --2- 172.21.15.22:0/1421172651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ca9e4ebf60 0x55ca9e505a30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 >> 172.21.15.22:0/1421172651 conn(0x55ca9e4b2b30 msgr2=0x55ca9e4e5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.476+0000 7f6fd6a00640 0 client.25706 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.477+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 shutdown_connections 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.477+0000 7f6ffa884740 1 -- 172.21.15.22:0/1421172651 wait complete. 2024-03-24T06:40:40.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.477+0000 7f6ffa884740 20 client.25706 trim_cache size 0 max 16384 2024-03-24T06:40:40.482 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:40:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:40:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:40:40.482 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:40:40.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 msgr2=0x7f6fd80a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 0x7f6fd80a4780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8036040 tx=0x7f6fe8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 msgr2=0x7f6ff0199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400d950 tx=0x7f6fe400de10 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 shutdown_connections 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6fd80a22c0 0x7f6fd80a4780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6ff010bb60 0x7f6ff019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ff0103060 0x7f6ff019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 --2- 172.21.15.22:0/2137001686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ff0102740 0x7f6ff0199e00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 >> 172.21.15.22:0/2137001686 conn(0x7f6ff00fbf60 msgr2=0x7f6ff00fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 shutdown_connections 2024-03-24T06:40:40.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.478+0000 7f6ff6600640 1 -- 172.21.15.22:0/2137001686 wait complete. 2024-03-24T06:40:40.499 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:40:40.499 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 -- 172.21.15.22:0/714300945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc1075e0 msgr2=0x7f32cc1079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 --2- 172.21.15.22:0/714300945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc1075e0 0x7f32cc1079c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009930 tx=0x7f32c002bc20 comp rx=0 tx=0).stop 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 -- 172.21.15.22:0/714300945 shutdown_connections 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 --2- 172.21.15.22:0/714300945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32cc101ee0 0x7f32cc10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 --2- 172.21.15.22:0/714300945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101520 0x7f32cc1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 --2- 172.21.15.22:0/714300945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc1075e0 0x7f32cc1079c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 -- 172.21.15.22:0/714300945 >> 172.21.15.22:0/714300945 conn(0x7f32cc0fcff0 msgr2=0x7f32cc0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 -- 172.21.15.22:0/714300945 shutdown_connections 2024-03-24T06:40:40.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.678+0000 7f32d2c00640 1 -- 172.21.15.22:0/714300945 wait complete. 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.679+0000 7f32d2c00640 1 Processor -- start 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 -- start start 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101ee0 0x7f32cc19ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32cc1075e0 0x7f32cc199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc19a2c0 con 0x7f32cc1075e0 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc19a430 con 0x7f32cc101520 2024-03-24T06:40:40.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d2c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc19a5f0 con 0x7f32cc101ee0 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35116/0 (socket says 172.21.15.22:35116) 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d1800640 1 -- 172.21.15.22:0/1864171555 learned_addr learned my addr 172.21.15.22:0/1864171555 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.680+0000 7f32d0e00640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101ee0 0x7f32cc19ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 -- 172.21.15.22:0/1864171555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101ee0 msgr2=0x7f32cc19ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101ee0 0x7f32cc19ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 -- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32cc1075e0 msgr2=0x7f32cc199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32cc1075e0 0x7f32cc199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 -- 172.21.15.22:0/1864171555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c0009560 con 0x7f32cc101520 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32d1800640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005ec0 tx=0x7f32c00040e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c002dea0 con 0x7f32cc101520 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c00044e0 con 0x7f32cc101520 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.681+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0003b80 con 0x7f32cc101520 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.682+0000 7f32d2c00640 1 -- 172.21.15.22:0/1864171555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc19a870 con 0x7f32cc101520 2024-03-24T06:40:40.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.682+0000 7f32d2c00640 1 -- 172.21.15.22:0/1864171555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc1a69f0 con 0x7f32cc101520 2024-03-24T06:40:40.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.684+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32cc19aa60 con 0x7f32cc101520 2024-03-24T06:40:40.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.685+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f32c00054b0 con 0x7f32cc101520 2024-03-24T06:40:40.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.686+0000 7f32ca000640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 0x7f32b40a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:40.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.687+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f32c00f0650 con 0x7f32cc101520 2024-03-24T06:40:40.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.687+0000 7f32d0e00640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 0x7f32b40a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:40.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.688+0000 7f32d0e00640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 0x7f32b40a4800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32cc19b310 tx=0x7f32b800a2d0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:40.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.691+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f32c00b8ec0 con 0x7f32cc101520 2024-03-24T06:40:40.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.967+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32cc068420 con 0x7f32cc101520 2024-03-24T06:40:40.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.968+0000 7f32ca000640 1 -- 172.21.15.22:0/1864171555 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11939 (secure 0 0 0) 0x7f32c00bc580 con 0x7f32cc101520 2024-03-24T06:40:40.968 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:40:40.968 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:39:58.997320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:40:40.968 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 29 2024-03-24T06:40:40.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.971+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 msgr2=0x7f32b40a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.971+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 0x7f32b40a4800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32cc19b310 tx=0x7f32b800a2d0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.971+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 msgr2=0x7f32cc19fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:40.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.971+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f32c0005ec0 tx=0x7f32c00040e0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 shutdown_connections 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32b40a2340 0x7f32b40a4800 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32cc1075e0 0x7f32cc199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32cc101ee0 0x7f32cc19ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 --2- 172.21.15.22:0/1864171555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101520 0x7f32cc19fa70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 >> 172.21.15.22:0/1864171555 conn(0x7f32cc0fcff0 msgr2=0x7f32cc0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 shutdown_connections 2024-03-24T06:40:40.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:40.973+0000 7f32bf400640 1 -- 172.21.15.22:0/1864171555 wait complete. 2024-03-24T06:40:40.998 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.201+0000 7f18efe00640 1 -- 172.21.15.22:0/998407037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 msgr2=0x7f18f0107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.201+0000 7f18efe00640 1 --2- 172.21.15.22:0/998407037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f0107a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009930 tx=0x7f18e002bc50 comp rx=0 tx=0).stop 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 -- 172.21.15.22:0/998407037 shutdown_connections 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 --2- 172.21.15.22:0/998407037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18f0101f60 0x7f18f010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 --2- 172.21.15.22:0/998407037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 0x7f18f0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 --2- 172.21.15.22:0/998407037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f0107a40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 -- 172.21.15.22:0/998407037 >> 172.21.15.22:0/998407037 conn(0x7f18f00fd0b0 msgr2=0x7f18f00ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 -- 172.21.15.22:0/998407037 shutdown_connections 2024-03-24T06:40:41.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.202+0000 7f18efe00640 1 -- 172.21.15.22:0/998407037 wait complete. 2024-03-24T06:40:41.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.203+0000 7f18efe00640 1 Processor -- start 2024-03-24T06:40:41.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.203+0000 7f18efe00640 1 -- start start 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 0x7f18f019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18f0101f60 0x7f18f019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0196030 con 0x7f18f01015a0 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f18f01961a0 con 0x7f18f0107660 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18efe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0196360 con 0x7f18f0101f60 2024-03-24T06:40:41.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18eea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 0x7f18f019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35132/0 (socket says 172.21.15.22:35132) 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 -- 172.21.15.22:0/206473792 learned_addr learned my addr 172.21.15.22:0/206473792 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 -- 172.21.15.22:0/206473792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18f0101f60 msgr2=0x7f18f019cc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18f0101f60 0x7f18f019cc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 -- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 msgr2=0x7f18f019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 0x7f18f019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.204+0000 7f18ef400640 1 -- 172.21.15.22:0/206473792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e0009560 con 0x7f18f0107660 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.205+0000 7f18ef400640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f18e800eb10 tx=0x7f18e800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.205+0000 7f18f4a00640 1 -- 172.21.15.22:0/206473792 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e8004b20 con 0x7f18f0107660 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.205+0000 7f18f4a00640 1 -- 172.21.15.22:0/206473792 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18e8004c80 con 0x7f18f0107660 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.205+0000 7f18f4a00640 1 -- 172.21.15.22:0/206473792 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e8010650 con 0x7f18f0107660 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.206+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f0196640 con 0x7f18f0107660 2024-03-24T06:40:41.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.206+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f019a540 con 0x7f18f0107660 2024-03-24T06:40:41.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.209+0000 7f18f8bbb740 1 Processor -- start 2024-03-24T06:40:41.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.210+0000 7f18f4a00640 1 -- 172.21.15.22:0/206473792 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f18e8005020 con 0x7f18f0107660 2024-03-24T06:40:41.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 -- start start 2024-03-24T06:40:41.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f12830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f1b950 0x563954f1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563954e806e0 con 0x563954f1b950 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563954e7dc10 con 0x563954f12450 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.211+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563954e793f0 con 0x563954f12d70 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41570/0 (socket says 172.21.15.22:41570) 2024-03-24T06:40:41.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 -- 172.21.15.22:0/3520371995 learned_addr learned my addr 172.21.15.22:0/3520371995 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 -- 172.21.15.22:0/3520371995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 msgr2=0x563954f12830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f12830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 -- 172.21.15.22:0/3520371995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f1b950 msgr2=0x563954f1dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f1b950 0x563954f1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.212+0000 7f18eea00640 1 -- 172.21.15.22:0/3520371995 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563954e3ae40 con 0x563954f12d70 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18eea00640 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009900 tx=0x7f18e0004c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d89fdb986b993a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18dbe00640 1 -- 172.21.15.22:0/3520371995 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00046c0 con 0x563954f12d70 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18f4a00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 0x7f18d40a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18dbe00640 1 -- 172.21.15.22:0/3520371995 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18e00054e0 con 0x563954f12d70 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18dbe00640 1 -- 172.21.15.22:0/3520371995 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e002cb30 con 0x563954f12d70 2024-03-24T06:40:41.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.213+0000 7f18f4a00640 1 -- 172.21.15.22:0/206473792 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18e8014070 con 0x7f18f0107660 2024-03-24T06:40:41.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 -- 172.21.15.22:0/3520371995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 msgr2=0x563954f1b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009900 tx=0x7f18e0004c90 comp rx=0 tx=0).stop 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 -- 172.21.15.22:0/3520371995 shutdown_connections 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f1b950 0x563954f1dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f1b410 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/3520371995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f12830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 -- 172.21.15.22:0/3520371995 >> 172.21.15.22:0/3520371995 conn(0x563954f0cb80 msgr2=0x563954f0cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18ee000640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 0x7f18d40a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.214+0000 7f18f8bbb740 1 -- 172.21.15.22:0/3520371995 shutdown_connections 2024-03-24T06:40:41.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.215+0000 7f18f8bbb740 1 -- 172.21.15.22:0/3520371995 wait complete. 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.215+0000 7f18f8bbb740 1 -- start start 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.215+0000 7f18ee000640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 0x7f18d40a47f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x563954f0dd10 tx=0x7f18dc005760 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.215+0000 7f18f8bbb740 1 Processor -- start 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 0x563954f18ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f19400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563954e1e060 con 0x563954f12d70 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563954e1dd00 con 0x563954f12450 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563954be71c0 con 0x563954f1b950 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18f8bbb740 10 client.0 _ll_register_callbacks cb 0x563954b89be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35140/0 (socket says 172.21.15.22:35140) 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18eea00640 1 -- 172.21.15.22:0/961660876 learned_addr learned my addr 172.21.15.22:0/961660876 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.216+0000 7f18ef400640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f19400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18ee000640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 0x563954f18ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 -- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 msgr2=0x563954f19400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f19400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 -- 172.21.15.22:0/961660876 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 msgr2=0x563954f18ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 0x563954f18ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 -- 172.21.15.22:0/961660876 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e0038040 con 0x563954f12450 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18ee000640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 0x563954f18ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18eea00640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f18e002c190 tx=0x7f18e0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b769219fd09b6193 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.217+0000 7f18d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00048d0 con 0x563954f12450 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563954b787e0 con 0x563954f12450 2024-03-24T06:40:41.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18e002c8b0 con 0x563954f12450 2024-03-24T06:40:41.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18e0004a30 con 0x563954f12450 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18f8bbb740 10 client.25721 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563954e77a70 con 0x563954f12450 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.218+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563954b787e0 con 0x563954f12450 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f18e003fc30 con 0x563954f12450 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 10 client.25721 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 10 client.25721 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 20 client.25721 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f2b960 0x563954f2ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 4 client.25721 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.219+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563954bb5720 con 0x563954f2b960 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.220+0000 7f18ee000640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f2b960 0x563954f2ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.220+0000 7f18ee000640 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f2b960 0x563954f2ddc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=726a0064d0d989cd server_cookie=d33f874b929c2780 in_seq=0 out_seq=0 2024-03-24T06:40:41.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.220+0000 7f18d8c00640 10 client.25721 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:40:41.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.221+0000 7f18d8c00640 1 -- 172.21.15.22:0/961660876 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563954bb5720 con 0x563954f2b960 2024-03-24T06:40:41.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.221+0000 7f18d8c00640 10 client.25721 handle_command_reply: tid=0 2024-03-24T06:40:41.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.221+0000 7f18f8bbb740 1 client.25721 shutdown 2024-03-24T06:40:41.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.221+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 msgr2=0x563954f18980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.221+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f18e002c190 tx=0x7f18e0005850 comp rx=0 tx=0).stop 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 shutdown_connections 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f2b960 0x563954f2ddc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f19400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12d70 0x563954f18ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/961660876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12450 0x563954f18980 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 >> 172.21.15.22:0/961660876 conn(0x563954f0cb80 msgr2=0x563954f0e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18d8c00640 0 client.25721 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 shutdown_connections 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 1 -- 172.21.15.22:0/961660876 wait complete. 2024-03-24T06:40:41.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.222+0000 7f18f8bbb740 20 client.25721 trim_cache size 0 max 16384 2024-03-24T06:40:41.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.236+0000 7f18f8bbb740 1 Processor -- start 2024-03-24T06:40:41.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 -- start start 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f27180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f336d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563954f11880 con 0x563954f12450 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563954f0dbb0 con 0x563954f12d70 2024-03-24T06:40:41.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563954f0c6b0 con 0x563954f1b950 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18ee000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18eea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f27180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18ee000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35142/0 (socket says 172.21.15.22:35142) 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18eea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f27180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33308/0 (socket says 172.21.15.22:33308) 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.237+0000 7f18ee000640 1 -- 172.21.15.22:0/1017613885 learned_addr learned my addr 172.21.15.22:0/1017613885 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 -- 172.21.15.22:0/1017613885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 msgr2=0x563954f336d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f336d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 -- 172.21.15.22:0/1017613885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 msgr2=0x563954f27180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f27180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 -- 172.21.15.22:0/1017613885 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563954e13b00 con 0x563954f12d70 2024-03-24T06:40:41.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18ee000640 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00bfd0 tx=0x7f18dc010000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a2c47341974e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.238+0000 7f18db400640 1 -- 172.21.15.22:0/1017613885 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc023070 con 0x563954f12d70 2024-03-24T06:40:41.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18db400640 1 -- 172.21.15.22:0/1017613885 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18dc00dcf0 con 0x563954f12d70 2024-03-24T06:40:41.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18db400640 1 -- 172.21.15.22:0/1017613885 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc02c630 con 0x563954f12d70 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1017613885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 msgr2=0x563954f19220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00bfd0 tx=0x7f18dc010000 comp rx=0 tx=0).stop 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1017613885 shutdown_connections 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f1b950 0x563954f336d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f12d70 0x563954f19220 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1017613885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f27180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1017613885 >> 172.21.15.22:0/1017613885 conn(0x563954f0cb80 msgr2=0x563954f0bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.239+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1017613885 shutdown_connections 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.240+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1017613885 wait complete. 2024-03-24T06:40:41.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.240+0000 7f18f8bbb740 1 -- start start 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.240+0000 7f18f8bbb740 1 Processor -- start 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f15000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f1b950 0x563954f15540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563954f15a80 con 0x563954f12450 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563954f15bf0 con 0x563954f1b950 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563954f15db0 con 0x563954f12d70 2024-03-24T06:40:41.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18f8bbb740 10 client.0 _ll_register_callbacks cb 0x7f18e0005e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18eea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18eea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33324/0 (socket says 172.21.15.22:33324) 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18eea00640 1 -- 172.21.15.22:0/1862061562 learned_addr learned my addr 172.21.15.22:0/1862061562 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18ee000640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f15000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.241+0000 7f18ef400640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f1b950 0x563954f15540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 -- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 msgr2=0x563954f15000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f15000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 -- 172.21.15.22:0/1862061562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f1b950 msgr2=0x563954f15540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f1b950 0x563954f15540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 -- 172.21.15.22:0/1862061562 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18dc00d930 con 0x563954f12450 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18eea00640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f18e002c260 tx=0x7f18e002c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f401d2c378d9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.242+0000 7f18d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.244+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0005310 con 0x563954f12450 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.244+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f18e002c8b0 con 0x563954f12450 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.244+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18e00054e0 con 0x563954f12450 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.244+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18e0004d80 con 0x563954f12450 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 10 client.15585 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563954e396a0 con 0x563954f12450 2024-03-24T06:40:41.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18e00054e0 con 0x563954f12450 2024-03-24T06:40:41.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f18e003fa70 con 0x563954f12450 2024-03-24T06:40:41.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 10 client.15585 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 10 client.15585 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 20 client.15585 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f91c80 0x563954f94070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 4 client.15585 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.245+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563954c04890 con 0x563954f91c80 2024-03-24T06:40:41.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.246+0000 7f18ee000640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f91c80 0x563954f94070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.246+0000 7f18ee000640 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f91c80 0x563954f94070 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1da192c03ef8e57 server_cookie=9ee10d0b16bd99b in_seq=0 out_seq=0 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.246+0000 7f18d8c00640 10 client.15585 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18d8c00640 1 -- 172.21.15.22:0/1862061562 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563954c04890 con 0x563954f91c80 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18d8c00640 10 client.15585 handle_command_reply: tid=0 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 client.15585 shutdown 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 msgr2=0x563954f237b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f18e002c260 tx=0x7f18e002c840 comp rx=0 tx=0).stop 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 shutdown_connections 2024-03-24T06:40:41.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x563954f91c80 0x563954f94070 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563954f1b950 0x563954f15540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563954f12d70 0x563954f15000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 --2- 172.21.15.22:0/1862061562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563954f12450 0x563954f237b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 >> 172.21.15.22:0/1862061562 conn(0x563954f0cb80 msgr2=0x563954f0bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.247+0000 7f18d8c00640 0 client.15585 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:40:41.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.248+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 shutdown_connections 2024-03-24T06:40:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.248+0000 7f18f8bbb740 1 -- 172.21.15.22:0/1862061562 wait complete. 2024-03-24T06:40:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.248+0000 7f18f8bbb740 20 client.15585 trim_cache size 0 max 16384 2024-03-24T06:40:41.249 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:40:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.249+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 msgr2=0x7f18d40a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.249+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 0x7f18d40a47f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x563954f0dd10 tx=0x7f18dc005760 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.249+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 msgr2=0x7f18f01958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.249+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f18e800eb10 tx=0x7f18e800c2d0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 shutdown_connections 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f18d40a2330 0x7f18d40a47f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f0107660 0x7f18f01958a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18f0101f60 0x7f18f019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 --2- 172.21.15.22:0/206473792 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f18f01015a0 0x7f18f019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 >> 172.21.15.22:0/206473792 conn(0x7f18f00fd0b0 msgr2=0x7f18f00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 shutdown_connections 2024-03-24T06:40:41.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.250+0000 7f18efe00640 1 -- 172.21.15.22:0/206473792 wait complete. 2024-03-24T06:40:41.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:41 smithi022 ceph-mon[32003]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:41.251 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:41 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1864171555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:41.268 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 -- 172.21.15.22:0/165047971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8101f60 msgr2=0x7efec810e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 --2- 172.21.15.22:0/165047971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8101f60 0x7efec810e420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efec0009cd0 tx=0x7efec002c0e0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 -- 172.21.15.22:0/165047971 shutdown_connections 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 --2- 172.21.15.22:0/165047971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8101f60 0x7efec810e420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 --2- 172.21.15.22:0/165047971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec8101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 --2- 172.21.15.22:0/165047971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8107660 0x7efec8107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 -- 172.21.15.22:0/165047971 >> 172.21.15.22:0/165047971 conn(0x7efec80fd0b0 msgr2=0x7efec80ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 -- 172.21.15.22:0/165047971 shutdown_connections 2024-03-24T06:40:41.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.466+0000 7efece600640 1 -- 172.21.15.22:0/165047971 wait complete. 2024-03-24T06:40:41.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.467+0000 7efece600640 1 Processor -- start 2024-03-24T06:40:41.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.467+0000 7efece600640 1 -- start start 2024-03-24T06:40:41.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8101f60 0x7efec819dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8107660 0x7efec8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efec8197f50 con 0x7efec8101f60 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efec81980c0 con 0x7efec8107660 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efece600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efec8198280 con 0x7efec81015a0 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efecd200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efecd200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41624/0 (socket says 172.21.15.22:41624) 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efecd200640 1 -- 172.21.15.22:0/1704846169 learned_addr learned my addr 172.21.15.22:0/1704846169 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.468+0000 7efec7e00640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8101f60 0x7efec819dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecdc00640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8107660 0x7efec8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 -- 172.21.15.22:0/1704846169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8107660 msgr2=0x7efec8197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8107660 0x7efec8197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 -- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8101f60 msgr2=0x7efec819dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8101f60 0x7efec819dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 -- 172.21.15.22:0/1704846169 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec0009900 con 0x7efec81015a0 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.469+0000 7efecd200640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c920 tx=0x7efebc00cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.470+0000 7efec5600640 1 -- 172.21.15.22:0/1704846169 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc0079a0 con 0x7efec81015a0 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.470+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec8198560 con 0x7efec81015a0 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.470+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec8073b20 con 0x7efec81015a0 2024-03-24T06:40:41.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.470+0000 7efec5600640 1 -- 172.21.15.22:0/1704846169 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efebc015e20 con 0x7efec81015a0 2024-03-24T06:40:41.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.470+0000 7efec5600640 1 -- 172.21.15.22:0/1704846169 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc01b340 con 0x7efec81015a0 2024-03-24T06:40:41.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.473+0000 7efed27f8740 1 Processor -- start 2024-03-24T06:40:41.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.474+0000 7efed27f8740 1 -- start start 2024-03-24T06:40:41.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.474+0000 7efed27f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.474+0000 7efed27f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48fd70 0x55d3fd498410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.474+0000 7efed27f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd49ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.474+0000 7efed27f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd3fd6e0 con 0x55d3fd498950 2024-03-24T06:40:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efed27f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd3fac10 con 0x55d3fd48f450 2024-03-24T06:40:41.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efed27f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd3f63f0 con 0x55d3fd48fd70 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35174/0 (socket says 172.21.15.22:35174) 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 -- 172.21.15.22:0/3346735996 learned_addr learned my addr 172.21.15.22:0/3346735996 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efec7e00640 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd49ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 -- 172.21.15.22:0/3346735996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48fd70 msgr2=0x55d3fd498410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48fd70 0x55d3fd498410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 -- 172.21.15.22:0/3346735996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 msgr2=0x55d3fd49ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd49ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.475+0000 7efecdc00640 1 -- 172.21.15.22:0/3346735996 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3fd3b7e40 con 0x55d3fd48f450 2024-03-24T06:40:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.476+0000 7efec5600640 1 -- 172.21.15.22:0/1704846169 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7efebc007b00 con 0x7efec81015a0 2024-03-24T06:40:41.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.477+0000 7efecdc00640 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efec0009ca0 tx=0x7efec0032e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa690d3c04264355 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.478+0000 7efec5600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 0x7efeb00a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.479+0000 7efec5600640 1 -- 172.21.15.22:0/1704846169 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7efebc0d6280 con 0x7efec81015a0 2024-03-24T06:40:41.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.479+0000 7efecd200640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 0x7efeb00a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.479+0000 7efeba000640 1 -- 172.21.15.22:0/3346735996 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0034440 con 0x55d3fd48f450 2024-03-24T06:40:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.479+0000 7efeba000640 1 -- 172.21.15.22:0/3346735996 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec00345a0 con 0x55d3fd48f450 2024-03-24T06:40:41.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.479+0000 7efeba000640 1 -- 172.21.15.22:0/3346735996 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0003bd0 con 0x55d3fd48f450 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 -- 172.21.15.22:0/3346735996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 msgr2=0x55d3fd48f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efec0009ca0 tx=0x7efec0032e00 comp rx=0 tx=0).stop 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efecd200640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 0x7efeb00a4800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efebc0d36c0 tx=0x7efebc01d940 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 -- 172.21.15.22:0/3346735996 shutdown_connections 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd49ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48fd70 0x55d3fd498410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 --2- 172.21.15.22:0/3346735996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd48f830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.480+0000 7efed27f8740 1 -- 172.21.15.22:0/3346735996 >> 172.21.15.22:0/3346735996 conn(0x55d3fd489b80 msgr2=0x55d3fd489f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.481+0000 7efed27f8740 1 -- 172.21.15.22:0/3346735996 shutdown_connections 2024-03-24T06:40:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.481+0000 7efed27f8740 1 -- 172.21.15.22:0/3346735996 wait complete. 2024-03-24T06:40:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.481+0000 7efed27f8740 1 -- start start 2024-03-24T06:40:41.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 Processor -- start 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd495920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 0x55d3fd495e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd4963a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd39b060 con 0x55d3fd48fd70 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd39ad00 con 0x55d3fd48f450 2024-03-24T06:40:41.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd1641c0 con 0x55d3fd498950 2024-03-24T06:40:41.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.482+0000 7efed27f8740 10 client.0 _ll_register_callbacks cb 0x55d3fd106be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efecdc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd495920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 0x55d3fd495e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efecdc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd495920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35178/0 (socket says 172.21.15.22:35178) 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efecdc00640 1 -- 172.21.15.22:0/579945527 learned_addr learned my addr 172.21.15.22:0/579945527 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efecdc00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd4963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 -- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 msgr2=0x55d3fd4963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd4963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 -- 172.21.15.22:0/579945527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 msgr2=0x55d3fd495920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd495920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.483+0000 7efec7e00640 1 -- 172.21.15.22:0/579945527 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec810ed20 con 0x55d3fd48fd70 2024-03-24T06:40:41.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.484+0000 7efec7e00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 0x55d3fd495e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efec81990a0 tx=0x7efeb4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3096c37b319d27b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.484+0000 7efeaea00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.484+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4005220 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.484+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7efeb4005380 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.484+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d3fd0f57e0 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efeb4005540 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efed27f8740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3fd3f4a70 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d3fd0f57e0 con 0x55d3fd48fd70 2024-03-24T06:40:41.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efeb4010bf0 con 0x55d3fd48fd70 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efed27f8740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.485+0000 7efed27f8740 10 client.15594 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.486+0000 7efed27f8740 20 client.15594 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.486+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd4a8d70 0x55d3fd4ab190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.486+0000 7efed27f8740 4 client.15594 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:41.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.486+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3fd132720 con 0x55d3fd4a8d70 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.486+0000 7efecdc00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd4a8d70 0x55d3fd4ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.487+0000 7efeaea00640 10 client.15594 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.487+0000 7efecdc00640 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd4a8d70 0x55d3fd4ab190 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c2af5177fd62d6d server_cookie=4a03907db7f94f4b in_seq=0 out_seq=0 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.487+0000 7efeaea00640 1 -- 172.21.15.22:0/579945527 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3fd132720 con 0x55d3fd4a8d70 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.487+0000 7efeaea00640 10 client.15594 handle_command_reply: tid=0 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.487+0000 7efed27f8740 1 client.15594 shutdown 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 msgr2=0x55d3fd495e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 0x55d3fd495e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efec81990a0 tx=0x7efeb4012820 comp rx=0 tx=0).stop 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 shutdown_connections 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd4a8d70 0x55d3fd4ab190 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd4963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48fd70 0x55d3fd495e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 --2- 172.21.15.22:0/579945527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48f450 0x55d3fd495920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 >> 172.21.15.22:0/579945527 conn(0x55d3fd489b80 msgr2=0x55d3fd48b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efeaea00640 0 client.15594 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:40:41.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 shutdown_connections 2024-03-24T06:40:41.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 1 -- 172.21.15.22:0/579945527 wait complete. 2024-03-24T06:40:41.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.488+0000 7efed27f8740 20 client.15594 trim_cache size 0 max 16384 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 Processor -- start 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 -- start start 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 0x55d3fd498050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd4b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd132720 con 0x55d3fd498950 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd0f57e0 con 0x55d3fd48fd70 2024-03-24T06:40:41.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.503+0000 7efed27f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd48a980 con 0x55d3fd48f450 2024-03-24T06:40:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efecdc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 0x55d3fd498050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35180/0 (socket says 172.21.15.22:35180) 2024-03-24T06:40:41.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efecdc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 0x55d3fd498050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41648/0 (socket says 172.21.15.22:41648) 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 -- 172.21.15.22:0/1090491816 learned_addr learned my addr 172.21.15.22:0/1090491816 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efecdc00640 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd4b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 -- 172.21.15.22:0/1090491816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 msgr2=0x55d3fd498050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 0x55d3fd498050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 -- 172.21.15.22:0/1090491816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 msgr2=0x55d3fd4b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd4b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 -- 172.21.15.22:0/1090491816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3fd390b00 con 0x55d3fd48fd70 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.504+0000 7efec7e00640 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efeb400ba20 tx=0x7efeb400b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7cbe5ba978234d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efeb9600640 1 -- 172.21.15.22:0/1090491816 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4026760 con 0x55d3fd48fd70 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efeb9600640 1 -- 172.21.15.22:0/1090491816 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeb4005030 con 0x55d3fd48fd70 2024-03-24T06:40:41.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efeb9600640 1 -- 172.21.15.22:0/1090491816 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4010950 con 0x55d3fd48fd70 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 -- 172.21.15.22:0/1090491816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 msgr2=0x55d3fd4a41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efeb400ba20 tx=0x7efeb400b890 comp rx=0 tx=0).stop 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 -- 172.21.15.22:0/1090491816 shutdown_connections 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd498950 0x55d3fd4b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd4a41e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 --2- 172.21.15.22:0/1090491816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd48f450 0x55d3fd498050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.505+0000 7efed27f8740 1 -- 172.21.15.22:0/1090491816 >> 172.21.15.22:0/1090491816 conn(0x55d3fd489b80 msgr2=0x55d3fd488cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.506+0000 7efed27f8740 1 -- 172.21.15.22:0/1090491816 shutdown_connections 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.506+0000 7efed27f8740 1 -- 172.21.15.22:0/1090491816 wait complete. 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.506+0000 7efed27f8740 1 -- start start 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.506+0000 7efed27f8740 1 Processor -- start 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48f450 0x55d3fd4a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd49c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd49ca10 con 0x55d3fd48f450 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd49cb80 con 0x55d3fd48fd70 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d3fd50e0b0 con 0x55d3fd498950 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efed27f8740 10 client.0 _ll_register_callbacks cb 0x7efeb400bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efecdc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48f450 0x55d3fd4a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35182/0 (socket says 172.21.15.22:35182) 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 -- 172.21.15.22:0/957981630 learned_addr learned my addr 172.21.15.22:0/957981630 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efecdc00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd49c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 -- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 msgr2=0x55d3fd49c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd49c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 -- 172.21.15.22:0/957981630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48f450 msgr2=0x55d3fd4a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.507+0000 7efec7e00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48f450 0x55d3fd4a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efec7e00640 1 -- 172.21.15.22:0/957981630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3fd3f4a70 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efec7e00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efeb400bbf0 tx=0x7efeb4001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c851b03a34cdfcbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efeaea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4036070 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7efeb4026dd0 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec00345a0 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.508+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efeb40053b0 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.509+0000 7efed27f8740 10 client.25745 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.509+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3fd3b66a0 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.509+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec00345a0 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.509+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efeb4039020 con 0x55d3fd48fd70 2024-03-24T06:40:41.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.509+0000 7efed27f8740 10 client.25745 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 10 client.25745 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 20 client.25745 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd50f470 0x55d3fd511860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 4 client.25745 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3fd181890 con 0x55d3fd50f470 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efecdc00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd50f470 0x55d3fd511860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efecdc00640 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd50f470 0x55d3fd511860 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c9a6f6abba806f68 server_cookie=352c877d2834daf9 in_seq=0 out_seq=0 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efeaea00640 10 client.25745 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efeaea00640 1 -- 172.21.15.22:0/957981630 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3fd181890 con 0x55d3fd50f470 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efeaea00640 10 client.25745 handle_command_reply: tid=0 2024-03-24T06:40:41.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.510+0000 7efed27f8740 1 client.25745 shutdown 2024-03-24T06:40:41.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 msgr2=0x55d3fd49bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efeb400bbf0 tx=0x7efeb4001e50 comp rx=0 tx=0).stop 2024-03-24T06:40:41.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 shutdown_connections 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55d3fd50f470 0x55d3fd511860 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d3fd498950 0x55d3fd49c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d3fd48fd70 0x55d3fd49bdf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 --2- 172.21.15.22:0/957981630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d3fd48f450 0x55d3fd4a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 >> 172.21.15.22:0/957981630 conn(0x55d3fd489b80 msgr2=0x55d3fd488cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efeaea00640 0 client.25745 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 shutdown_connections 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 1 -- 172.21.15.22:0/957981630 wait complete. 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.511+0000 7efed27f8740 20 client.25745 trim_cache size 0 max 16384 2024-03-24T06:40:41.513 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 msgr2=0x7efeb00a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 0x7efeb00a4800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efebc0d36c0 tx=0x7efebc01d940 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 msgr2=0x7efec819d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c920 tx=0x7efebc00cde0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 shutdown_connections 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efeb00a2340 0x7efeb00a4800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efec8107660 0x7efec8197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efec8101f60 0x7efec819dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 --2- 172.21.15.22:0/1704846169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec81015a0 0x7efec819d700 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 >> 172.21.15.22:0/1704846169 conn(0x7efec80fd0b0 msgr2=0x7efec80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 shutdown_connections 2024-03-24T06:40:41.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.512+0000 7efece600640 1 -- 172.21.15.22:0/1704846169 wait complete. 2024-03-24T06:40:41.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:41 smithi155 ceph-mon[36262]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:41.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:41 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1864171555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:41.529 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:40:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:41 smithi143 ceph-mon[36520]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:41 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1864171555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:40:41.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 -- 172.21.15.22:0/1153002705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c107660 msgr2=0x7f459c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 --2- 172.21.15.22:0/1153002705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c107660 0x7f459c107a40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f458c009930 tx=0x7f458c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 -- 172.21.15.22:0/1153002705 shutdown_connections 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 --2- 172.21.15.22:0/1153002705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c101f60 0x7f459c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 --2- 172.21.15.22:0/1153002705 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c1015a0 0x7f459c101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 --2- 172.21.15.22:0/1153002705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c107660 0x7f459c107a40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.711+0000 7f459be00640 1 -- 172.21.15.22:0/1153002705 >> 172.21.15.22:0/1153002705 conn(0x7f459c0fd0b0 msgr2=0x7f459c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.712+0000 7f459be00640 1 -- 172.21.15.22:0/1153002705 shutdown_connections 2024-03-24T06:40:41.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.712+0000 7f459be00640 1 -- 172.21.15.22:0/1153002705 wait complete. 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 Processor -- start 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 -- start start 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 0x7f459c199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 0x7f459c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f459c19ec50 con 0x7f459c101f60 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f459c19edc0 con 0x7f459c107660 2024-03-24T06:40:41.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.713+0000 7f459be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f459c19ef80 con 0x7f459c1015a0 2024-03-24T06:40:41.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459a000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 0x7f459c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35198/0 (socket says 172.21.15.22:35198) 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 -- 172.21.15.22:0/4015016861 learned_addr learned my addr 172.21.15.22:0/4015016861 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459aa00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 0x7f459c199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 -- 172.21.15.22:0/4015016861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 msgr2=0x7f459c199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 0x7f459c199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 -- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 msgr2=0x7f459c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 0x7f459c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459b400640 1 -- 172.21.15.22:0/4015016861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c009560 con 0x7f459c107660 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459aa00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 0x7f459c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.714+0000 7f459a000640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 0x7f459c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f459b400640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f459400eaa0 tx=0x7f459400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f4593400640 1 -- 172.21.15.22:0/4015016861 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459400cdb0 con 0x7f459c107660 2024-03-24T06:40:41.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f4593400640 1 -- 172.21.15.22:0/4015016861 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4594005020 con 0x7f459c107660 2024-03-24T06:40:41.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459c19f260 con 0x7f459c107660 2024-03-24T06:40:41.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459c1a6ae0 con 0x7f459c107660 2024-03-24T06:40:41.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.715+0000 7f4593400640 1 -- 172.21.15.22:0/4015016861 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45940116d0 con 0x7f459c107660 2024-03-24T06:40:41.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.719+0000 7f45a43ce740 1 Processor -- start 2024-03-24T06:40:41.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.719+0000 7f4593400640 1 -- 172.21.15.22:0/4015016861 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f4594011830 con 0x7f459c107660 2024-03-24T06:40:41.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 -- start start 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327aeac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327aed550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558327a476e0 con 0x558327ad9c10 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558327a44c10 con 0x558327aeb160 2024-03-24T06:40:41.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.720+0000 7f45a43ce740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558327a403f0 con 0x558327ad9260 2024-03-24T06:40:41.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41682/0 (socket says 172.21.15.22:41682) 2024-03-24T06:40:41.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 -- 172.21.15.22:0/1406348100 learned_addr learned my addr 172.21.15.22:0/1406348100 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327aed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 -- 172.21.15.22:0/1406348100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 msgr2=0x558327aed550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327aed550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.721+0000 7f459a000640 1 -- 172.21.15.22:0/1406348100 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 msgr2=0x558327aeac20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:40:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f4593400640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 0x7f45800a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f459aa00640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327aeac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f4593400640 1 -- 172.21.15.22:0/4015016861 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f45940ce7c0 con 0x7f459c107660 2024-03-24T06:40:41.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f459a000640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327aeac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f459a000640 1 -- 172.21.15.22:0/1406348100 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558327a01e40 con 0x558327ad9260 2024-03-24T06:40:41.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.722+0000 7f459aa00640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327aeac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:41.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.723+0000 7f459a000640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 0x7f45800a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.724+0000 7f459a000640 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4588013930 tx=0x7f458800b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce038129581ef64e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.724+0000 7f459a000640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 0x7f45800a4880 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4588013fd0 tx=0x7f458801b000 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.724+0000 7f4587400640 1 -- 172.21.15.22:0/1406348100 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458800eeb0 con 0x558327ad9260 2024-03-24T06:40:41.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.724+0000 7f4587400640 1 -- 172.21.15.22:0/1406348100 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4588019030 con 0x558327ad9260 2024-03-24T06:40:41.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.725+0000 7f4587400640 1 -- 172.21.15.22:0/1406348100 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458801ec30 con 0x558327ad9260 2024-03-24T06:40:41.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 -- 172.21.15.22:0/1406348100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 msgr2=0x558327ad9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4588013930 tx=0x7f458800b480 comp rx=0 tx=0).stop 2024-03-24T06:40:41.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 -- 172.21.15.22:0/1406348100 shutdown_connections 2024-03-24T06:40:41.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327aed550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327aeac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1406348100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327ad9640 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 -- 172.21.15.22:0/1406348100 >> 172.21.15.22:0/1406348100 conn(0x558327ad3b80 msgr2=0x558327ad3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 -- 172.21.15.22:0/1406348100 shutdown_connections 2024-03-24T06:40:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.726+0000 7f45a43ce740 1 -- 172.21.15.22:0/1406348100 wait complete. 2024-03-24T06:40:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.727+0000 7f45a43ce740 1 -- start start 2024-03-24T06:40:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.727+0000 7f45a43ce740 1 Processor -- start 2024-03-24T06:40:41.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9260 0x558327ae81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 0x558327ae86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5583279e5060 con 0x558327ad9260 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5583279e4d00 con 0x558327ad9c10 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5583277ae1c0 con 0x558327aeb160 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f45a43ce740 10 client.0 _ll_register_callbacks cb 0x558327750be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f459b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f459b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41694/0 (socket says 172.21.15.22:41694) 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f459b400640 1 -- 172.21.15.22:0/1471543577 learned_addr learned my addr 172.21.15.22:0/1471543577 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.728+0000 7f459a000640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 0x558327ae86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9260 0x558327ae81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 -- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 msgr2=0x558327ae8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 -- 172.21.15.22:0/1471543577 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9260 msgr2=0x558327ae81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9260 0x558327ae81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 -- 172.21.15.22:0/1471543577 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4588013610 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459b400640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f459a000640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 0x558327ae86e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f458801e850 tx=0x7f458800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f582370299d31d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.729+0000 7f457be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458801edc0 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f458801e620 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55832773f7e0 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4588004020 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f45a43ce740 10 client.25757 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.730+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558327a3ea70 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55832773f7e0 con 0x558327ad9c10 2024-03-24T06:40:41.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f45880118a0 con 0x558327ad9c10 2024-03-24T06:40:41.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 10 client.25757 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 10 client.25757 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:40:41.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 20 client.25757 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327afb460 0x558327afd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 4 client.25757 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:41.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.731+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55832777c720 con 0x558327afb460 2024-03-24T06:40:41.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.732+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327afb460 0x558327afd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.732+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327afb460 0x558327afd880 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf175fc452c6a137 server_cookie=62d070f0ba0d649f in_seq=0 out_seq=0 2024-03-24T06:40:41.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.732+0000 7f457be00640 10 client.25757 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f457be00640 1 -- 172.21.15.22:0/1471543577 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55832777c720 con 0x558327afb460 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f457be00640 10 client.25757 handle_command_reply: tid=0 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 client.25757 shutdown 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 msgr2=0x558327ae86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 0x558327ae86e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f458801e850 tx=0x7f458800ef20 comp rx=0 tx=0).stop 2024-03-24T06:40:41.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 shutdown_connections 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327afb460 0x558327afd880 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327aeb160 0x558327ae8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9c10 0x558327ae86e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471543577 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9260 0x558327ae81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.733+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 >> 172.21.15.22:0/1471543577 conn(0x558327ad3b80 msgr2=0x558327ad58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.734+0000 7f457be00640 0 client.25757 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.734+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 shutdown_connections 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.734+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471543577 wait complete. 2024-03-24T06:40:41.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.734+0000 7f45a43ce740 20 client.25757 trim_cache size 0 max 16384 2024-03-24T06:40:41.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.753+0000 7f45a43ce740 1 Processor -- start 2024-03-24T06:40:41.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 -- start start 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327ae8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327b03190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55832777c720 con 0x558327ad9c10 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55832773f7e0 con 0x558327aeb160 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f45a43ce740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558327ad87b0 con 0x558327ad9260 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f459aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f459aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41696/0 (socket says 172.21.15.22:41696) 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f459aa00640 1 -- 172.21.15.22:0/1399321834 learned_addr learned my addr 172.21.15.22:0/1399321834 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f459a000640 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327ae8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.754+0000 7f459b400640 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327b03190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 -- 172.21.15.22:0/1399321834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 msgr2=0x558327b03190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327b03190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 -- 172.21.15.22:0/1399321834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 msgr2=0x558327ae8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327ae8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 -- 172.21.15.22:0/1399321834 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583279dab00 con 0x558327ad9260 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f459aa00640 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f458c02e010 tx=0x7f458c0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7e5a0690979d95b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.755+0000 7f4586a00640 1 -- 172.21.15.22:0/1399321834 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c045070 con 0x558327ad9260 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f4586a00640 1 -- 172.21.15.22:0/1399321834 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f458c0047a0 con 0x558327ad9260 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f4586a00640 1 -- 172.21.15.22:0/1399321834 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c02cea0 con 0x558327ad9260 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 -- 172.21.15.22:0/1399321834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 msgr2=0x558327af6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f458c02e010 tx=0x7f458c0058d0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 -- 172.21.15.22:0/1399321834 shutdown_connections 2024-03-24T06:40:41.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327aeb160 0x558327b03190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327ad9c10 0x558327ae8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.756+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1399321834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9260 0x558327af6a30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.757+0000 7f45a43ce740 1 -- 172.21.15.22:0/1399321834 >> 172.21.15.22:0/1399321834 conn(0x558327ad3b80 msgr2=0x558327ad2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.757+0000 7f45a43ce740 1 -- 172.21.15.22:0/1399321834 shutdown_connections 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.757+0000 7f45a43ce740 1 -- 172.21.15.22:0/1399321834 wait complete. 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.757+0000 7f45a43ce740 1 -- start start 2024-03-24T06:40:41.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.757+0000 7f45a43ce740 1 Processor -- start 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9260 0x558327af2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327aeb160 0x558327aee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558327aeece0 con 0x558327aeb160 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558327aeee50 con 0x558327ad9260 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558327aef010 con 0x558327ad9c10 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.758+0000 7f45a43ce740 10 client.0 _ll_register_callbacks cb 0x7f458800bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:41.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41704/0 (socket says 172.21.15.22:41704) 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 -- 172.21.15.22:0/1471403352 learned_addr learned my addr 172.21.15.22:0/1471403352 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9260 0x558327af2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459b400640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327aeb160 0x558327aee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 -- 172.21.15.22:0/1471403352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9260 msgr2=0x558327af2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9260 0x558327af2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 -- 172.21.15.22:0/1471403352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327aeb160 msgr2=0x558327aee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327aeb160 0x558327aee5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 -- 172.21.15.22:0/1471403352 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f459c10ed20 con 0x558327ad9c10 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f459a000640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4588013010 tx=0x7f458801e4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28cdfbd5336a7e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.759+0000 7f457be00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458801edc0 con 0x558327ad9c10 2024-03-24T06:40:41.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4588000e40 con 0x558327ad9c10 2024-03-24T06:40:41.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5583277cb890 con 0x558327ad9c10 2024-03-24T06:40:41.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f458800b900 con 0x558327ad9c10 2024-03-24T06:40:41.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f45a43ce740 10 client.25804 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:41.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.760+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558327a006a0 con 0x558327ad9c10 2024-03-24T06:40:41.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5583277cb890 con 0x558327ad9c10 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4588029ce0 con 0x558327ad9c10 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 10 client.25804 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 10 client.25804 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 20 client.25804 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327b618d0 0x558327b63cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 4 client.25804 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5583276eb700 con 0x558327b618d0 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.761+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327b618d0 0x558327b63cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f459aa00640 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327b618d0 0x558327b63cc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67becad69ef4182a server_cookie=f6f2221b4975bde4 in_seq=0 out_seq=0 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f457be00640 10 client.25804 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f457be00640 1 -- 172.21.15.22:0/1471403352 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5583276eb700 con 0x558327b618d0 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f457be00640 10 client.25804 handle_command_reply: tid=0 2024-03-24T06:40:41.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 client.25804 shutdown 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 msgr2=0x558327af34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4588013010 tx=0x7f458801e4e0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 shutdown_connections 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x558327b618d0 0x558327b63cc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.762+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558327aeb160 0x558327aee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558327ad9c10 0x558327af34d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 1 --2- 172.21.15.22:0/1471403352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558327ad9260 0x558327af2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 >> 172.21.15.22:0/1471403352 conn(0x558327ad3b80 msgr2=0x558327ad2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f457be00640 0 client.25804 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 shutdown_connections 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 1 -- 172.21.15.22:0/1471403352 wait complete. 2024-03-24T06:40:41.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.763+0000 7f45a43ce740 20 client.25804 trim_cache size 0 max 16384 2024-03-24T06:40:41.766 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:40:41.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 msgr2=0x7f45800a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 0x7f45800a4880 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4588013fd0 tx=0x7f458801b000 comp rx=0 tx=0).stop 2024-03-24T06:40:41.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 msgr2=0x7f459c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f459400eaa0 tx=0x7f459400ef60 comp rx=0 tx=0).stop 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 shutdown_connections 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f45800a23c0 0x7f45800a4880 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459c107660 0x7f459c19e650 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f459c101f60 0x7f459c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 --2- 172.21.15.22:0/4015016861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f459c1015a0 0x7f459c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 >> 172.21.15.22:0/4015016861 conn(0x7f459c0fd0b0 msgr2=0x7f459c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 shutdown_connections 2024-03-24T06:40:41.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:41.764+0000 7f459be00640 1 -- 172.21.15.22:0/4015016861 wait complete. 2024-03-24T06:40:41.781 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1950102855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 msgr2=0x7f8ef410e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1950102855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410e420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009c10 tx=0x7f8eec02bef0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1950102855 shutdown_connections 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1950102855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410e420 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1950102855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef4101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1950102855 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef4107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1950102855 >> 172.21.15.22:0/1950102855 conn(0x7f8ef40fd0b0 msgr2=0x7f8ef40ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1950102855 shutdown_connections 2024-03-24T06:40:42.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.035+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1950102855 wait complete. 2024-03-24T06:40:42.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.036+0000 7f8ef3e00640 1 Processor -- start 2024-03-24T06:40:42.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 -- start start 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef410e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef410bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410c370 con 0x7f8ef4107660 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef41aa520 con 0x7f8ef4101f60 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.037+0000 7f8ef3e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef41aa660 con 0x7f8ef41015a0 2024-03-24T06:40:42.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef410bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef410e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef410bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33472/0 (socket says 172.21.15.22:33472) 2024-03-24T06:40:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef3400640 1 -- 172.21.15.22:0/1142670102 learned_addr learned my addr 172.21.15.22:0/1142670102 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2000640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 -- 172.21.15.22:0/1142670102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 msgr2=0x7f8ef410b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 -- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 msgr2=0x7f8ef410bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef410bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.038+0000 7f8ef2a00640 1 -- 172.21.15.22:0/1142670102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee0009560 con 0x7f8ef41015a0 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef410e3d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400ea20 tx=0x7f8ee400eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8eeb400640 1 -- 172.21.15.22:0/1142670102 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400ccc0 con 0x7f8ef41015a0 2024-03-24T06:40:42.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8eeb400640 1 -- 172.21.15.22:0/1142670102 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee400ce20 con 0x7f8ef41015a0 2024-03-24T06:40:42.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eec009840 con 0x7f8ef41015a0 2024-03-24T06:40:42.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8eeb400640 1 -- 172.21.15.22:0/1142670102 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4018840 con 0x7f8ef41015a0 2024-03-24T06:40:42.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.039+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef41aad10 con 0x7f8ef41015a0 2024-03-24T06:40:42.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 Processor -- start 2024-03-24T06:40:42.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 -- start start 2024-03-24T06:40:42.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b75410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 0x55e3a8b77d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8ada6e0 con 0x55e3a8b6c450 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8ad7c10 con 0x55e3a8b6cd70 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.044+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8ad33f0 con 0x55e3a8b75950 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.045+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.045+0000 7f8ef3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33484/0 (socket says 172.21.15.22:33484) 2024-03-24T06:40:42.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.045+0000 7f8ef3400640 1 -- 172.21.15.22:0/2445676183 learned_addr learned my addr 172.21.15.22:0/2445676183 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.045+0000 7f8eeb400640 1 -- 172.21.15.22:0/1142670102 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7f8ee40189a0 con 0x7f8ef41015a0 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.045+0000 7f8ef2000640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 0x55e3a8b77d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2000640 1 -- 172.21.15.22:0/2445676183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 msgr2=0x55e3a8b75410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b75410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2000640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b75410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2000640 1 -- 172.21.15.22:0/2445676183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 msgr2=0x55e3a8b6c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2000640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2000640 1 -- 172.21.15.22:0/2445676183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a8a94e40 con 0x55e3a8b75950 2024-03-24T06:40:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b75410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.046+0000 7f8ef3400640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:42.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.047+0000 7f8ef2000640 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 0x55e3a8b77d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8ef410ccc0 tx=0x7f8ee0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=100de7ee41b46890 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.048+0000 7f8edf400640 1 -- 172.21.15.22:0/2445676183 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0004220 con 0x55e3a8b75950 2024-03-24T06:40:42.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.048+0000 7f8edf400640 1 -- 172.21.15.22:0/2445676183 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee0004380 con 0x55e3a8b75950 2024-03-24T06:40:42.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.048+0000 7f8edf400640 1 -- 172.21.15.22:0/2445676183 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0019950 con 0x55e3a8b75950 2024-03-24T06:40:42.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.048+0000 7f8eeb400640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 0x7f8ed80a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.048+0000 7f8eeb400640 1 -- 172.21.15.22:0/1142670102 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ee4014070 con 0x7f8ef41015a0 2024-03-24T06:40:42.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.049+0000 7f8efc1f8740 1 -- 172.21.15.22:0/2445676183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 msgr2=0x55e3a8b77d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.049+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 0x55e3a8b77d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8ef410ccc0 tx=0x7f8ee0011810 comp rx=0 tx=0).stop 2024-03-24T06:40:42.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.049+0000 7f8ef3400640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 0x7f8ed80a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 -- 172.21.15.22:0/2445676183 shutdown_connections 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b75950 0x55e3a8b77d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b75410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/2445676183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b6c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 -- 172.21.15.22:0/2445676183 >> 172.21.15.22:0/2445676183 conn(0x55e3a8b66b80 msgr2=0x55e3a8b66f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8ef3400640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 0x7f8ed80a4880 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8eec02cf80 tx=0x7f8eec009280 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 -- 172.21.15.22:0/2445676183 shutdown_connections 2024-03-24T06:40:42.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.050+0000 7f8efc1f8740 1 -- 172.21.15.22:0/2445676183 wait complete. 2024-03-24T06:40:42.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.051+0000 7f8efc1f8740 1 -- start start 2024-03-24T06:40:42.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.051+0000 7f8efc1f8740 1 Processor -- start 2024-03-24T06:40:42.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b73360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8a78060 con 0x55e3a8b6c450 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8a77d00 con 0x55e3a8b75950 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e3a88411c0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8efc1f8740 10 client.0 _ll_register_callbacks cb 0x55e3a87e3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8ef2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8ef2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8ef2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41746/0 (socket says 172.21.15.22:41746) 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8ef2000640 1 -- 172.21.15.22:0/50137531 learned_addr learned my addr 172.21.15.22:0/50137531 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.052+0000 7f8ef3400640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b73360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 -- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 msgr2=0x55e3a8b73360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b73360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 -- 172.21.15.22:0/50137531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 msgr2=0x55e3a8b728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 -- 172.21.15.22:0/50137531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee0009a10 con 0x55e3a8b6cd70 2024-03-24T06:40:42.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.053+0000 7f8ef2000640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00055c0 tx=0x7f8ee0019430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=397c9f8a63aabbcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.054+0000 7f8ed3e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.054+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0004990 con 0x55e3a8b6cd70 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.054+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ee0004af0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.054+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3a87d27e0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.054+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8ee0004cb0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 10 client.25816 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e3a8ad1a70 con 0x55e3a8b6cd70 2024-03-24T06:40:42.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e3a87d27e0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8ee0024830 con 0x55e3a8b6cd70 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 10 client.25816 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 10 client.25816 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 20 client.25816 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8b85b70 0x55e3a8b87f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 4 client.25816 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:42.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.055+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3a880f720 con 0x55e3a8b85b70 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.056+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8b85b70 0x55e3a8b87f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.056+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8b85b70 0x55e3a8b87f90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f78a71486600a882 server_cookie=6ecccccdd4025772 in_seq=0 out_seq=0 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.056+0000 7f8ed3e00640 10 client.25816 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.057+0000 7f8ed3e00640 1 -- 172.21.15.22:0/50137531 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e3a880f720 con 0x55e3a8b85b70 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.057+0000 7f8ed3e00640 10 client.25816 handle_command_reply: tid=0 2024-03-24T06:40:42.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.057+0000 7f8efc1f8740 1 client.25816 shutdown 2024-03-24T06:40:42.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 msgr2=0x55e3a8b72e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00055c0 tx=0x7f8ee0019430 comp rx=0 tx=0).stop 2024-03-24T06:40:42.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 shutdown_connections 2024-03-24T06:40:42.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8b85b70 0x55e3a8b87f90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b73360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b72e20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/50137531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 >> 172.21.15.22:0/50137531 conn(0x55e3a8b66b80 msgr2=0x55e3a8b688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8ed3e00640 0 client.25816 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 shutdown_connections 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 1 -- 172.21.15.22:0/50137531 wait complete. 2024-03-24T06:40:42.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.058+0000 7f8efc1f8740 20 client.25816 trim_cache size 0 max 16384 2024-03-24T06:40:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.086+0000 7f8efc1f8740 1 Processor -- start 2024-03-24T06:40:42.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.086+0000 7f8efc1f8740 1 -- start start 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8b6b890 con 0x55e3a8b6c450 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8b624e0 con 0x55e3a8b75950 2024-03-24T06:40:42.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8b67b80 con 0x55e3a8b6cd70 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8ef2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8ef2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41758/0 (socket says 172.21.15.22:41758) 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.087+0000 7f8ef2000640 1 -- 172.21.15.22:0/3193934954 learned_addr learned my addr 172.21.15.22:0/3193934954 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef3400640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2000640 1 -- 172.21.15.22:0/3193934954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 msgr2=0x55e3a8b8d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2000640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b8d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2000640 1 -- 172.21.15.22:0/3193934954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 msgr2=0x55e3a8b811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2000640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef2000640 1 -- 172.21.15.22:0/3193934954 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a8a6db00 con 0x55e3a8b6cd70 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.088+0000 7f8ef3400640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.089+0000 7f8ef2000640 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee00194a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27124c8c28766fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.089+0000 7f8edea00640 1 -- 172.21.15.22:0/3193934954 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0004b90 con 0x55e3a8b6cd70 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.089+0000 7f8edea00640 1 -- 172.21.15.22:0/3193934954 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee0031070 con 0x55e3a8b6cd70 2024-03-24T06:40:42.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.089+0000 7f8edea00640 1 -- 172.21.15.22:0/3193934954 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee00195e0 con 0x55e3a8b6cd70 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 -- 172.21.15.22:0/3193934954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 msgr2=0x55e3a8b73180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee00194a0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 -- 172.21.15.22:0/3193934954 shutdown_connections 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b75950 0x55e3a8b8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6cd70 0x55e3a8b73180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/3193934954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b6c450 0x55e3a8b811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 -- 172.21.15.22:0/3193934954 >> 172.21.15.22:0/3193934954 conn(0x55e3a8b66b80 msgr2=0x55e3a8b65430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 -- 172.21.15.22:0/3193934954 shutdown_connections 2024-03-24T06:40:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.090+0000 7f8efc1f8740 1 -- 172.21.15.22:0/3193934954 wait complete. 2024-03-24T06:40:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.091+0000 7f8efc1f8740 1 -- start start 2024-03-24T06:40:42.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.091+0000 7f8efc1f8740 1 Processor -- start 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8be3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 0x55e3a8be45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8be4c80 con 0x55e3a8b75950 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8b7c800 con 0x55e3a8b6cd70 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e3a8b7c9c0 con 0x55e3a8b6c450 2024-03-24T06:40:42.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8efc1f8740 10 client.0 _ll_register_callbacks cb 0x7f8ee0009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:42.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8ef2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8ef2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41774/0 (socket says 172.21.15.22:41774) 2024-03-24T06:40:42.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8ef2a00640 1 -- 172.21.15.22:0/842818571 learned_addr learned my addr 172.21.15.22:0/842818571 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8ef2000640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8be3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.092+0000 7f8ef3400640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 0x55e3a8be45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 -- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 msgr2=0x55e3a8be3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8be3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 -- 172.21.15.22:0/842818571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 msgr2=0x55e3a8be45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 0x55e3a8be45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 -- 172.21.15.22:0/842818571 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a8ad1a70 con 0x55e3a8b6c450 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef3400640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 0x55e3a8be45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ef2a00640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4097180 tx=0x7f8ee4010e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3a49be46993ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ed3e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee40a3410 con 0x55e3a8b6c450 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ee40cbd30 con 0x55e3a8b6c450 2024-03-24T06:40:42.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ee0004380 con 0x55e3a8b6c450 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.093+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8ee40cbef0 con 0x55e3a8b6c450 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 10 client.25822 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e3a8a936a0 con 0x55e3a8b6c450 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee0004380 con 0x55e3a8b6c450 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8ee409a350 con 0x55e3a8b6c450 2024-03-24T06:40:42.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 10 client.25822 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 10 client.25822 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 20 client.25822 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8bfcec0 0x55e3a8bff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 4 client.25822 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.094+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3a885e890 con 0x55e3a8bfcec0 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8ef2000640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8bfcec0 0x55e3a8bff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8ef2000640 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8bfcec0 0x55e3a8bff2b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=204269da4f5dcb67 server_cookie=31f9cac3c4f79f38 in_seq=0 out_seq=0 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8ed3e00640 10 client.25822 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8ed3e00640 1 -- 172.21.15.22:0/842818571 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3a885e890 con 0x55e3a8bfcec0 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8ed3e00640 10 client.25822 handle_command_reply: tid=0 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.095+0000 7f8efc1f8740 1 client.25822 shutdown 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 msgr2=0x55e3a8be7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4097180 tx=0x7f8ee4010e00 comp rx=0 tx=0).stop 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 shutdown_connections 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e3a8b75950 0x55e3a8be45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e3a8bfcec0 0x55e3a8bff2b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e3a8b6cd70 0x55e3a8be3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 --2- 172.21.15.22:0/842818571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e3a8b6c450 0x55e3a8be7950 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8ed3e00640 0 client.25822 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 >> 172.21.15.22:0/842818571 conn(0x55e3a8b66b80 msgr2=0x55e3a8b68760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 shutdown_connections 2024-03-24T06:40:42.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 1 -- 172.21.15.22:0/842818571 wait complete. 2024-03-24T06:40:42.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.096+0000 7f8efc1f8740 20 client.25822 trim_cache size 0 max 16384 2024-03-24T06:40:42.098 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.097+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 msgr2=0x7f8ed80a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.097+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 0x7f8ed80a4880 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8eec02cf80 tx=0x7f8eec009280 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.097+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 msgr2=0x7f8ef410e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.097+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef410e3d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400ea20 tx=0x7f8ee400eee0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 shutdown_connections 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8ed80a23c0 0x7f8ed80a4880 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8ef4107660 0x7f8ef410bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ef4101f60 0x7f8ef410b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 --2- 172.21.15.22:0/1142670102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ef41015a0 0x7f8ef410e3d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 >> 172.21.15.22:0/1142670102 conn(0x7f8ef40fd0b0 msgr2=0x7f8ef40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 shutdown_connections 2024-03-24T06:40:42.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.098+0000 7f8ef3e00640 1 -- 172.21.15.22:0/1142670102 wait complete. 2024-03-24T06:40:42.120 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:40:42.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 -- 172.21.15.22:0/2708817419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 msgr2=0x7ffbb410d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2708817419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb410d900 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009900 tx=0x7ffba002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 -- 172.21.15.22:0/2708817419 shutdown_connections 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2708817419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb4110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2708817419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb410d900 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2708817419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.340+0000 7ffbb3200640 1 -- 172.21.15.22:0/2708817419 >> 172.21.15.22:0/2708817419 conn(0x7ffbb40fd0b0 msgr2=0x7ffbb40ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.341+0000 7ffbb3200640 1 -- 172.21.15.22:0/2708817419 shutdown_connections 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.341+0000 7ffbb3200640 1 -- 172.21.15.22:0/2708817419 wait complete. 2024-03-24T06:40:42.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.342+0000 7ffbb3200640 1 Processor -- start 2024-03-24T06:40:42.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.342+0000 7ffbb3200640 1 -- start start 2024-03-24T06:40:42.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb419ecd0 con 0x7ffbb40743e0 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb419ee40 con 0x7ffbb410de40 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb419f000 con 0x7ffbb4073ac0 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb419a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33518/0 (socket says 172.21.15.22:33518) 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb1400640 1 -- 172.21.15.22:0/2924391846 learned_addr learned my addr 172.21.15.22:0/2924391846 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.343+0000 7ffbb1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 -- 172.21.15.22:0/2924391846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 msgr2=0x7ffbb419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 -- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 msgr2=0x7ffbb419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 -- 172.21.15.22:0/2924391846 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba0009560 con 0x7ffbb4073ac0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbb1e00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4199e10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffba400e9f0 tx=0x7ffba400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.344+0000 7ffbaaa00640 1 -- 172.21.15.22:0/2924391846 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba400ccc0 con 0x7ffbb4073ac0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.345+0000 7ffbaaa00640 1 -- 172.21.15.22:0/2924391846 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba400ce20 con 0x7ffbb4073ac0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.345+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb419f2e0 con 0x7ffbb4073ac0 2024-03-24T06:40:42.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.345+0000 7ffbaaa00640 1 -- 172.21.15.22:0/2924391846 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4004a90 con 0x7ffbb4073ac0 2024-03-24T06:40:42.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.345+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb41a6b60 con 0x7ffbb4073ac0 2024-03-24T06:40:42.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.348+0000 7ffbbb9a6740 1 Processor -- start 2024-03-24T06:40:42.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.348+0000 7ffbbb9a6740 1 -- start start 2024-03-24T06:40:42.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3672830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a367b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f2a35e06e0 con 0x55f2a3672d70 2024-03-24T06:40:42.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2a35ddc10 con 0x55f2a3672450 2024-03-24T06:40:42.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2a35d93f0 con 0x55f2a367b950 2024-03-24T06:40:42.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41792/0 (socket says 172.21.15.22:41792) 2024-03-24T06:40:42.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.349+0000 7ffbb1400640 1 -- 172.21.15.22:0/1217409764 learned_addr learned my addr 172.21.15.22:0/1217409764 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb2800640 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3672830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 -- 172.21.15.22:0/1217409764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 msgr2=0x55f2a3672830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3672830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 -- 172.21.15.22:0/1217409764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 msgr2=0x55f2a367b410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a367b410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 -- 172.21.15.22:0/1217409764 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a359ae40 con 0x55f2a367b950 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb2800640 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3672830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbb1400640 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffba002c2e0 tx=0x7ffba002c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cb29da772501112 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffbaaa00640 1 -- 172.21.15.22:0/2924391846 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7ffba4005140 con 0x7ffbb4073ac0 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffb9f000640 1 -- 172.21.15.22:0/1217409764 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0004880 con 0x55f2a367b950 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.350+0000 7ffb9f000640 1 -- 172.21.15.22:0/1217409764 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba00049e0 con 0x55f2a367b950 2024-03-24T06:40:42.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.351+0000 7ffb9f000640 1 -- 172.21.15.22:0/1217409764 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0005030 con 0x55f2a367b950 2024-03-24T06:40:42.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.351+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/1217409764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 msgr2=0x55f2a367dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.351+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffba002c2e0 tx=0x7ffba002c750 comp rx=0 tx=0).stop 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.351+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/1217409764 shutdown_connections 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a367dd40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a367b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/1217409764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3672830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/1217409764 >> 172.21.15.22:0/1217409764 conn(0x55f2a366cb80 msgr2=0x55f2a366cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/1217409764 shutdown_connections 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/1217409764 wait complete. 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.352+0000 7ffbbb9a6740 1 -- start start 2024-03-24T06:40:42.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 Processor -- start 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 0x55f2a3678a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a367b950 0x55f2a3679500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f2a357e060 con 0x55f2a367b950 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2a357dd00 con 0x55f2a3672d70 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2a33471c0 con 0x55f2a3672450 2024-03-24T06:40:42.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbbb9a6740 10 client.0 _ll_register_callbacks cb 0x55f2a32e9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.353+0000 7ffbb2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 0x55f2a3678a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35300/0 (socket says 172.21.15.22:35300) 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbb1400640 1 -- 172.21.15.22:0/2108859617 learned_addr learned my addr 172.21.15.22:0/2108859617 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbaaa00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 0x7ffb980a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbb1400640 1 -- 172.21.15.22:0/2108859617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 msgr2=0x55f2a3678a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.354+0000 7ffbaaa00640 1 -- 172.21.15.22:0/2924391846 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffba4014070 con 0x7ffbb4073ac0 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb1e00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 0x7ffb980a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb1400640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 0x55f2a3678a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb1400640 1 -- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a367b950 msgr2=0x55f2a3679500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb1400640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a367b950 0x55f2a3679500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb1400640 1 -- 172.21.15.22:0/2108859617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb4110ee0 con 0x55f2a3672d70 2024-03-24T06:40:42.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.355+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 0x55f2a3678a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffbb1400640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffba0005ab0 tx=0x7ffba00050a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b17b0cc945376b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffb93400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba002cc10 con 0x55f2a3672d70 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f2a32d87e0 con 0x55f2a3672d70 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffba005f030 con 0x55f2a3672d70 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffba0059460 con 0x55f2a3672d70 2024-03-24T06:40:42.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffbbb9a6740 10 client.25784 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.356+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f2a35d7a70 con 0x55f2a3672d70 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbb1e00640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 0x7ffb980a4790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffba40cb8d0 tx=0x7ffba4005020 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f2a32d87e0 con 0x55f2a3672d70 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffba003c050 con 0x55f2a3672d70 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 10 client.25784 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 20 client.25784 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a368bb70 0x55f2a368df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 4 client.25784 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.357+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a3315720 con 0x55f2a368bb70 2024-03-24T06:40:42.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.358+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a368bb70 0x55f2a368df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.359+0000 7ffb93400640 10 client.25784 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.359+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a368bb70 0x55f2a368df90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=defca2053bf31b4f server_cookie=3f1cfc781393685f in_seq=0 out_seq=0 2024-03-24T06:40:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffb93400640 1 -- 172.21.15.22:0/2108859617 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2a3315720 con 0x55f2a368bb70 2024-03-24T06:40:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffb93400640 10 client.25784 handle_command_reply: tid=0 2024-03-24T06:40:42.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 client.25784 shutdown 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 msgr2=0x55f2a3678fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ffba0005ab0 tx=0x7ffba00050a0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 shutdown_connections 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a367b950 0x55f2a3679500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a368bb70 0x55f2a368df90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672d70 0x55f2a3678fc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2108859617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672450 0x55f2a3678a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 >> 172.21.15.22:0/2108859617 conn(0x55f2a366cb80 msgr2=0x55f2a366e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.360+0000 7ffb93400640 0 client.25784 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:42.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.361+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 shutdown_connections 2024-03-24T06:40:42.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.361+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2108859617 wait complete. 2024-03-24T06:40:42.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.361+0000 7ffbbb9a6740 20 client.25784 trim_cache size 0 max 16384 2024-03-24T06:40:42.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.375+0000 7ffbbb9a6740 1 Processor -- start 2024-03-24T06:40:42.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 -- start start 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a3679320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a3693880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3642190 con 0x55f2a3672d70 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2a366c440 con 0x55f2a3672450 2024-03-24T06:40:42.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2a3668500 con 0x55f2a367b950 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbb2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbb2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35302/0 (socket says 172.21.15.22:35302) 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbb2800640 1 -- 172.21.15.22:0/2267143226 learned_addr learned my addr 172.21.15.22:0/2267143226 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a3693880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.376+0000 7ffbb1400640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a3679320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 -- 172.21.15.22:0/2267143226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 msgr2=0x55f2a3693880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a3693880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 -- 172.21.15.22:0/2267143226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 msgr2=0x55f2a3679320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a3679320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 -- 172.21.15.22:0/2267143226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a3573b00 con 0x55f2a3672450 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb1400640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a3679320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffbb2800640 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009fd0 tx=0x7ffbac003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4f3909591a1dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffb9e600640 1 -- 172.21.15.22:0/2267143226 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac020040 con 0x55f2a3672450 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.377+0000 7ffb9e600640 1 -- 172.21.15.22:0/2267143226 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbac021040 con 0x55f2a3672450 2024-03-24T06:40:42.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffb9e600640 1 -- 172.21.15.22:0/2267143226 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac005290 con 0x55f2a3672450 2024-03-24T06:40:42.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2267143226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 msgr2=0x55f2a3687180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009fd0 tx=0x7ffbac003a10 comp rx=0 tx=0).stop 2024-03-24T06:40:42.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2267143226 shutdown_connections 2024-03-24T06:40:42.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a367b950 0x55f2a3693880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672d70 0x55f2a3679320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/2267143226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a3672450 0x55f2a3687180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.378+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2267143226 >> 172.21.15.22:0/2267143226 conn(0x55f2a366cb80 msgr2=0x55f2a366e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.379+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2267143226 shutdown_connections 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.379+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/2267143226 wait complete. 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.379+0000 7ffbbb9a6740 1 -- start start 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.379+0000 7ffbbb9a6740 1 Processor -- start 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 0x55f2a3675590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a367b950 0x55f2a3676010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f2a36f0eb0 con 0x55f2a3672450 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2a36f1020 con 0x55f2a367b950 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2a36f11e0 con 0x55f2a3672d70 2024-03-24T06:40:42.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbbb9a6740 10 client.0 _ll_register_callbacks cb 0x7ffba0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:42.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbb1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41810/0 (socket says 172.21.15.22:41810) 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbb1400640 1 -- 172.21.15.22:0/3027817093 learned_addr learned my addr 172.21.15.22:0/3027817093 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbb2800640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 0x55f2a3675590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.380+0000 7ffbb2800640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a367b950 0x55f2a3676010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 -- 172.21.15.22:0/3027817093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a367b950 msgr2=0x55f2a3676010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a367b950 0x55f2a3676010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 -- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 msgr2=0x55f2a3675590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 0x55f2a3675590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 -- 172.21.15.22:0/3027817093 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac01bb90 con 0x55f2a3672d70 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb2800640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 0x55f2a3675590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffbb1400640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55f2a367a050 tx=0x7ffba0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52973fe9456e5150 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.381+0000 7ffb93400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba003fdb0 con 0x55f2a3672d70 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba00049e0 con 0x55f2a3672d70 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffba005f040 con 0x55f2a3672d70 2024-03-24T06:40:42.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffba00056b0 con 0x55f2a3672d70 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffbbb9a6740 10 client.25837 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f2a35996a0 con 0x55f2a3672d70 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.382+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba00049e0 con 0x55f2a3672d70 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffba003c040 con 0x55f2a3672d70 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 10 client.25837 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 20 client.25837 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a36f2490 0x55f2a36f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 4 client.25837 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:40:42.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2a3364890 con 0x55f2a36f2490 2024-03-24T06:40:42.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.383+0000 7ffbb2800640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a36f2490 0x55f2a36f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.384+0000 7ffb93400640 10 client.25837 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.384+0000 7ffbb2800640 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a36f2490 0x55f2a36f48b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=226b142ebe48894 server_cookie=f3be72fc69e83ea5 in_seq=0 out_seq=0 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.384+0000 7ffb93400640 1 -- 172.21.15.22:0/3027817093 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2a3364890 con 0x55f2a36f2490 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.384+0000 7ffb93400640 10 client.25837 handle_command_reply: tid=0 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.384+0000 7ffbbb9a6740 1 client.25837 shutdown 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 msgr2=0x55f2a3675ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55f2a367a050 tx=0x7ffba0005fd0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 shutdown_connections 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f2a36f2490 0x55f2a36f48b0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2a367b950 0x55f2a3676010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f2a3672d70 0x55f2a3675ad0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 --2- 172.21.15.22:0/3027817093 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f2a3672450 0x55f2a3675590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 >> 172.21.15.22:0/3027817093 conn(0x55f2a366cb80 msgr2=0x55f2a366db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffb93400640 0 client.25837 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 shutdown_connections 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 1 -- 172.21.15.22:0/3027817093 wait complete. 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.385+0000 7ffbbb9a6740 20 client.25837 trim_cache size 0 max 16384 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:40:42.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 msgr2=0x7ffb980a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 0x7ffb980a4790 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffba40cb8d0 tx=0x7ffba4005020 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 msgr2=0x7ffbb4199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4199e10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffba400e9f0 tx=0x7ffba400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 shutdown_connections 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb980a22d0 0x7ffb980a4790 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbb410de40 0x7ffbb419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbb40743e0 0x7ffbb419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 --2- 172.21.15.22:0/2924391846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbb4073ac0 0x7ffbb4199e10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.386+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 >> 172.21.15.22:0/2924391846 conn(0x7ffbb40fd0b0 msgr2=0x7ffbb40fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.387+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 shutdown_connections 2024-03-24T06:40:42.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:42.387+0000 7ffbb3200640 1 -- 172.21.15.22:0/2924391846 wait complete. 2024-03-24T06:40:43.408 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:40:43.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:40:43.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:40:43.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:43 smithi155 ceph-mon[36262]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:43.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.649+0000 7febd8e00640 1 -- 172.21.15.22:0/2661459296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 msgr2=0x7febd410e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.649+0000 7febd8e00640 1 --2- 172.21.15.22:0/2661459296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd410e410 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7febcc009cd0 tx=0x7febcc02c0e0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:43 smithi022 ceph-mon[32003]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:43.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 -- 172.21.15.22:0/2661459296 shutdown_connections 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 --2- 172.21.15.22:0/2661459296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd410e410 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 --2- 172.21.15.22:0/2661459296 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 --2- 172.21.15.22:0/2661459296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd4107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 -- 172.21.15.22:0/2661459296 >> 172.21.15.22:0/2661459296 conn(0x7febd40fd080 msgr2=0x7febd40ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 -- 172.21.15.22:0/2661459296 shutdown_connections 2024-03-24T06:40:43.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.650+0000 7febd8e00640 1 -- 172.21.15.22:0/2661459296 wait complete. 2024-03-24T06:40:43.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.651+0000 7febd8e00640 1 Processor -- start 2024-03-24T06:40:43.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 -- start start 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7febd419ecd0 con 0x7febd4101590 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7febd419ee40 con 0x7febd4101f50 2024-03-24T06:40:43.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7febd419f000 con 0x7febd4107650 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41818/0 (socket says 172.21.15.22:41818) 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.652+0000 7febd3e00640 1 -- 172.21.15.22:0/1280394920 learned_addr learned my addr 172.21.15.22:0/1280394920 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd2a00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3400640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 -- 172.21.15.22:0/1280394920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 msgr2=0x7febd419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 -- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 msgr2=0x7febd4199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 -- 172.21.15.22:0/1280394920 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc009900 con 0x7febd4107650 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3400640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.653+0000 7febd3e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7febcc005e30 tx=0x7febcc034a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.655+0000 7febcbe00640 1 -- 172.21.15.22:0/1280394920 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc0328b0 con 0x7febd4107650 2024-03-24T06:40:43.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.655+0000 7febcbe00640 1 -- 172.21.15.22:0/1280394920 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febcc003e50 con 0x7febd4107650 2024-03-24T06:40:43.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.655+0000 7febcbe00640 1 -- 172.21.15.22:0/1280394920 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc0047d0 con 0x7febd4107650 2024-03-24T06:40:43.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.655+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd419f280 con 0x7febd4107650 2024-03-24T06:40:43.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.655+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd41a6980 con 0x7febd4107650 2024-03-24T06:40:43.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.661+0000 7febdd20d740 1 Processor -- start 2024-03-24T06:40:43.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.662+0000 7febcbe00640 1 -- 172.21.15.22:0/1280394920 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7febcc00a040 con 0x7febd4107650 2024-03-24T06:40:43.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.662+0000 7febdd20d740 1 -- start start 2024-03-24T06:40:43.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x555584864220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584875d40 0x555584878130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5555847d22d0 con 0x555584863e40 2024-03-24T06:40:43.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5555847cf800 con 0x5555848647f0 2024-03-24T06:40:43.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.663+0000 7febdd20d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5555847cafe0 con 0x555584875d40 2024-03-24T06:40:43.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febd3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febd3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35348/0 (socket says 172.21.15.22:35348) 2024-03-24T06:40:43.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febd3400640 1 -- 172.21.15.22:0/1479635883 learned_addr learned my addr 172.21.15.22:0/1479635883 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febd2a00640 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584875d40 0x555584878130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febcbe00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 0x7febb80a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.664+0000 7febd2a00640 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x555584864220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febcbe00640 1 -- 172.21.15.22:0/1280394920 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7febcc0f0fd0 con 0x7febd4107650 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 -- 172.21.15.22:0/1479635883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584875d40 msgr2=0x555584878130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584875d40 0x555584878130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 -- 172.21.15.22:0/1479635883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 msgr2=0x555584864220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x555584864220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 -- 172.21.15.22:0/1479635883 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55558478ce90 con 0x5555848647f0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.665+0000 7febd3400640 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7febc400aa40 tx=0x7febc400af00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f4f7348b50ad57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.666+0000 7febd3e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 0x7febb80a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.666+0000 7febc8c00640 1 -- 172.21.15.22:0/1479635883 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc400ccc0 con 0x5555848647f0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.666+0000 7febc8c00640 1 -- 172.21.15.22:0/1479635883 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc400ce20 con 0x5555848647f0 2024-03-24T06:40:43.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.666+0000 7febc8c00640 1 -- 172.21.15.22:0/1479635883 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4018860 con 0x5555848647f0 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 -- 172.21.15.22:0/1479635883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 msgr2=0x555584875800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7febc400aa40 tx=0x7febc400af00 comp rx=0 tx=0).stop 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febd3e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 0x7febb80a4950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febcc0ee410 tx=0x7febcc034e90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 -- 172.21.15.22:0/1479635883 shutdown_connections 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584875d40 0x555584878130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x555584875800 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 --2- 172.21.15.22:0/1479635883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x555584864220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 -- 172.21.15.22:0/1479635883 >> 172.21.15.22:0/1479635883 conn(0x55558485e760 msgr2=0x55558485eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 -- 172.21.15.22:0/1479635883 shutdown_connections 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.667+0000 7febdd20d740 1 -- 172.21.15.22:0/1479635883 wait complete. 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.668+0000 7febdd20d740 1 -- start start 2024-03-24T06:40:43.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.668+0000 7febdd20d740 1 Processor -- start 2024-03-24T06:40:43.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584875d40 0x5555848737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55558476ffc0 con 0x5555848647f0 2024-03-24T06:40:43.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55558476fc60 con 0x555584875d40 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5555845130c0 con 0x555584863e40 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.669+0000 7febdd20d740 10 client.0 _ll_register_callbacks cb 0x555584787e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41838/0 (socket says 172.21.15.22:41838) 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 -- 172.21.15.22:0/2898978727 learned_addr learned my addr 172.21.15.22:0/2898978727 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584875d40 0x5555848737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 -- 172.21.15.22:0/2898978727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584875d40 msgr2=0x5555848737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584875d40 0x5555848737f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 -- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 msgr2=0x5555848732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 -- 172.21.15.22:0/2898978727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc400a640 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febd2a00640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7febbc009f20 tx=0x7febbc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5defd60de1de7100 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.670+0000 7febc1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc00ec30 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5555847efe90 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7febbc00ed90 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febbc004050 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febdd20d740 10 client.25846 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555847c9660 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.671+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5555847efe90 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7febbc003c70 con 0x555584863e40 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 10 client.25846 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:43.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 10 client.25846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 20 client.25846 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584886130 0x555584888560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 4 client.25846 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555847c1210 con 0x555584886130 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.672+0000 7febd3400640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584886130 0x555584888560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.673+0000 7febd3400640 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584886130 0x555584888560 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a0ff8019491448 server_cookie=4f6e623c72504c in_seq=0 out_seq=0 2024-03-24T06:40:43.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.673+0000 7febc1600640 10 client.25846 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:43.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febc1600640 1 -- 172.21.15.22:0/2898978727 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555847c1210 con 0x555584886130 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febc1600640 10 client.25846 handle_command_reply: tid=0 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 client.25846 shutdown 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 msgr2=0x555584872d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7febbc009f20 tx=0x7febbc004730 comp rx=0 tx=0).stop 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 shutdown_connections 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584886130 0x555584888560 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584875d40 0x5555848737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 --2- 172.21.15.22:0/2898978727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555584863e40 0x555584872d70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 >> 172.21.15.22:0/2898978727 conn(0x55558485e760 msgr2=0x5555848604b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.675+0000 7febc1600640 0 client.25846 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.676+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 shutdown_connections 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.676+0000 7febdd20d740 1 -- 172.21.15.22:0/2898978727 wait complete. 2024-03-24T06:40:43.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.676+0000 7febdd20d740 20 client.25846 trim_cache size 0 max 16384 2024-03-24T06:40:43.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:43 smithi143 ceph-mon[36520]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 Processor -- start 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 -- start start 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x55558488de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x55558487dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555584884a50 con 0x555584863e40 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55558484f730 con 0x5555848647f0 2024-03-24T06:40:43.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.707+0000 7febdd20d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5555848789e0 con 0x55558485e760 2024-03-24T06:40:43.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41844/0 (socket says 172.21.15.22:41844) 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 -- 172.21.15.22:0/3832326958 learned_addr learned my addr 172.21.15.22:0/3832326958 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x55558487dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd3400640 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x55558488de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 -- 172.21.15.22:0/3832326958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 msgr2=0x55558487dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x55558487dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 -- 172.21.15.22:0/3832326958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 msgr2=0x55558488de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x55558488de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 -- 172.21.15.22:0/3832326958 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555584765ae0 con 0x55558485e760 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.708+0000 7febd2a00640 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7febbc009ef0 tx=0x7febbc015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e6fea133155a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febc3e00640 1 -- 172.21.15.22:0/3832326958 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc020a50 con 0x55558485e760 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febc3e00640 1 -- 172.21.15.22:0/3832326958 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febbc020bb0 con 0x55558485e760 2024-03-24T06:40:43.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febc3e00640 1 -- 172.21.15.22:0/3832326958 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc004380 con 0x55558485e760 2024-03-24T06:40:43.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 -- 172.21.15.22:0/3832326958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 msgr2=0x55558485d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7febbc009ef0 tx=0x7febbc015040 comp rx=0 tx=0).stop 2024-03-24T06:40:43.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 -- 172.21.15.22:0/3832326958 shutdown_connections 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5555848647f0 0x55558487dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555584863e40 0x55558488de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 --2- 172.21.15.22:0/3832326958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558485d5a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.709+0000 7febdd20d740 1 -- 172.21.15.22:0/3832326958 >> 172.21.15.22:0/3832326958 conn(0x555584871a10 msgr2=0x555584874d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.710+0000 7febdd20d740 1 -- 172.21.15.22:0/3832326958 shutdown_connections 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.710+0000 7febdd20d740 1 -- 172.21.15.22:0/3832326958 wait complete. 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.710+0000 7febdd20d740 1 -- start start 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.710+0000 7febdd20d740 1 Processor -- start 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584863e40 0x55558487b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55558487b920 con 0x5555848647f0 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55558487ba90 con 0x555584863e40 2024-03-24T06:40:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5555848ec9f0 con 0x55558485e760 2024-03-24T06:40:43.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febdd20d740 10 client.0 _ll_register_callbacks cb 0x7febbc005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febd2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41858/0 (socket says 172.21.15.22:41858) 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febd2a00640 1 -- 172.21.15.22:0/1265910703 learned_addr learned my addr 172.21.15.22:0/1265910703 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febd3400640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584863e40 0x55558487b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.711+0000 7febd2a00640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 -- 172.21.15.22:0/1265910703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584863e40 msgr2=0x55558487b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584863e40 0x55558487b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 -- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 msgr2=0x5555848ec4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848ec4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 -- 172.21.15.22:0/1265910703 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555847c9660 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febd2a00640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7febbc005780 tx=0x7febbc004000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4c17c2ba27add43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febc1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc020a50 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7febbc032040 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.712+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febc400ce20 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.713+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febbc005180 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.713+0000 7febdd20d740 10 client.25852 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.713+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55558478b6f0 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.713+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febbc0052e0 con 0x55558485e760 2024-03-24T06:40:43.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7febbc011550 con 0x55558485e760 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 10 client.25852 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 10 client.25852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 20 client.25852 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584875d40 0x5555848efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 4 client.25852 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7febc400ce20 con 0x555584875d40 2024-03-24T06:40:43.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.714+0000 7febd3400640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584875d40 0x5555848efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.715+0000 7febc1600640 10 client.25852 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:43.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.715+0000 7febd3400640 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584875d40 0x5555848efdf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5be7802a0781ef02 server_cookie=bb4348fe0fcfe0b2 in_seq=0 out_seq=0 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.717+0000 7febc1600640 1 -- 172.21.15.22:0/1265910703 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7febc400ce20 con 0x555584875d40 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.717+0000 7febc1600640 10 client.25852 handle_command_reply: tid=0 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.717+0000 7febdd20d740 1 client.25852 shutdown 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 msgr2=0x55558487ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7febbc005780 tx=0x7febbc004000 comp rx=0 tx=0).stop 2024-03-24T06:40:43.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 shutdown_connections 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5555848647f0 0x5555848ec4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555584875d40 0x5555848efdf0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555584863e40 0x55558487b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febc1600640 0 client.25852 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 --2- 172.21.15.22:0/1265910703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55558485e760 0x55558487ac10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 >> 172.21.15.22:0/1265910703 conn(0x555584871a10 msgr2=0x555584874150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 shutdown_connections 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.718+0000 7febdd20d740 1 -- 172.21.15.22:0/1265910703 wait complete. 2024-03-24T06:40:43.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.719+0000 7febdd20d740 20 client.25852 trim_cache size 0 max 16384 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "a31bd469-a648-413c-926a-1df12e01dd2c", 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.719+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 msgr2=0x7febb80a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.719+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 0x7febb80a4950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febcc0ee410 tx=0x7febcc034e90 comp rx=0 tx=0).stop 2024-03-24T06:40:43.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.719+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 msgr2=0x7febd419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.719+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7febcc005e30 tx=0x7febcc034a90 comp rx=0 tx=0).stop 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 shutdown_connections 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7febb80a2490 0x7febb80a4950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febd4107650 0x7febd419e6d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febd4101f50 0x7febd419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 --2- 172.21.15.22:0/1280394920 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7febd4101590 0x7febd4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.720+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 >> 172.21.15.22:0/1280394920 conn(0x7febd40fd080 msgr2=0x7febd40fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.722+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 shutdown_connections 2024-03-24T06:40:43.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.724+0000 7febd8e00640 1 -- 172.21.15.22:0/1280394920 wait complete. 2024-03-24T06:40:43.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:40:43.739 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.979+0000 7fe317e00640 1 -- 172.21.15.22:0/1259524864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 msgr2=0x7fe318101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.979+0000 7fe317e00640 1 --2- 172.21.15.22:0/1259524864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 0x7fe318101a20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe304009930 tx=0x7fe30402bc30 comp rx=0 tx=0).stop 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 -- 172.21.15.22:0/1259524864 shutdown_connections 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 --2- 172.21.15.22:0/1259524864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31810e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 --2- 172.21.15.22:0/1259524864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 0x7fe318101a20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 --2- 172.21.15.22:0/1259524864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe318107660 0x7fe318107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 -- 172.21.15.22:0/1259524864 >> 172.21.15.22:0/1259524864 conn(0x7fe3180fd0b0 msgr2=0x7fe3180ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 -- 172.21.15.22:0/1259524864 shutdown_connections 2024-03-24T06:40:43.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.980+0000 7fe317e00640 1 -- 172.21.15.22:0/1259524864 wait complete. 2024-03-24T06:40:43.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 Processor -- start 2024-03-24T06:40:43.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 -- start start 2024-03-24T06:40:43.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 0x7fe318199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe318107660 0x7fe31819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe31819ec50 con 0x7fe318107660 2024-03-24T06:40:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe31819edc0 con 0x7fe318101f60 2024-03-24T06:40:43.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.981+0000 7fe317e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe31819ef80 con 0x7fe3181015a0 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35382/0 (socket says 172.21.15.22:35382) 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 -- 172.21.15.22:0/2880410317 learned_addr learned my addr 172.21.15.22:0/2880410317 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 -- 172.21.15.22:0/2880410317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 msgr2=0x7fe318199d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 0x7fe318199d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 -- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe318107660 msgr2=0x7fe31819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe318107660 0x7fe31819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 -- 172.21.15.22:0/2880410317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe304009560 con 0x7fe318101f60 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.982+0000 7fe316000640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe304000c00 tx=0x7fe304004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.983+0000 7fe30f400640 1 -- 172.21.15.22:0/2880410317 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304004440 con 0x7fe318101f60 2024-03-24T06:40:43.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.983+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31819f200 con 0x7fe318101f60 2024-03-24T06:40:43.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.983+0000 7fe30f400640 1 -- 172.21.15.22:0/2880410317 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe304046070 con 0x7fe318101f60 2024-03-24T06:40:43.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.983+0000 7fe30f400640 1 -- 172.21.15.22:0/2880410317 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304003850 con 0x7fe318101f60 2024-03-24T06:40:43.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.983+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe318108480 con 0x7fe318101f60 2024-03-24T06:40:43.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.986+0000 7fe320839740 1 Processor -- start 2024-03-24T06:40:43.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 -- start start 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 0x563e52f42990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f4e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563e52eb1790 con 0x563e52f43c40 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e52eaecc0 con 0x563e52f42f60 2024-03-24T06:40:43.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe320839740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563e52eaa4a0 con 0x563e52f4c2d0 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe316a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe316a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35394/0 (socket says 172.21.15.22:35394) 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.987+0000 7fe316a00640 1 -- 172.21.15.22:0/3712901002 learned_addr learned my addr 172.21.15.22:0/3712901002 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.988+0000 7fe316a00640 1 -- 172.21.15.22:0/3712901002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 msgr2=0x563e52f4e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.988+0000 7fe316a00640 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f4e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.988+0000 7fe316a00640 1 -- 172.21.15.22:0/3712901002 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 msgr2=0x563e52f42990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.988+0000 7fe30f400640 1 -- 172.21.15.22:0/2880410317 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136932+0+0 (secure 0 0 0) 0x7fe304004df0 con 0x7fe318101f60 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.989+0000 7fe317400640 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 0x563e52f42990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.989+0000 7fe316a00640 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 0x563e52f42990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.989+0000 7fe316a00640 1 -- 172.21.15.22:0/3712901002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e52e6bc20 con 0x563e52f42f60 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.989+0000 7fe317400640 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 0x563e52f42990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:40:43.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.989+0000 7fe316a00640 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe30800ba80 tx=0x7fe30800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81c479870cfd86ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.990+0000 7fe30f400640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 0x7fe2fc0a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.990+0000 7fe303e00640 1 -- 172.21.15.22:0/3712901002 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30800dca0 con 0x563e52f42f60 2024-03-24T06:40:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.990+0000 7fe30f400640 1 -- 172.21.15.22:0/2880410317 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe304048070 con 0x7fe318101f60 2024-03-24T06:40:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.990+0000 7fe303e00640 1 -- 172.21.15.22:0/3712901002 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30800de00 con 0x563e52f42f60 2024-03-24T06:40:43.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.990+0000 7fe303e00640 1 -- 172.21.15.22:0/3712901002 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30800dca0 con 0x563e52f42f60 2024-03-24T06:40:43.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.991+0000 7fe317400640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 0x7fe2fc0a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.991+0000 7fe320839740 1 -- 172.21.15.22:0/3712901002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 msgr2=0x563e52f433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.991+0000 7fe320839740 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe30800ba80 tx=0x7fe30800bf40 comp rx=0 tx=0).stop 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe317400640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 0x7fe2fc0a4800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe310009270 tx=0x7fe310009480 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 -- 172.21.15.22:0/3712901002 shutdown_connections 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f4e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f433e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 --2- 172.21.15.22:0/3712901002 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f43c40 0x563e52f42990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 -- 172.21.15.22:0/3712901002 >> 172.21.15.22:0/3712901002 conn(0x563e52f09b30 msgr2=0x563e52f3d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:43.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 -- 172.21.15.22:0/3712901002 shutdown_connections 2024-03-24T06:40:43.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.992+0000 7fe320839740 1 -- 172.21.15.22:0/3712901002 wait complete. 2024-03-24T06:40:43.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.993+0000 7fe320839740 1 -- start start 2024-03-24T06:40:43.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.993+0000 7fe320839740 1 Processor -- start 2024-03-24T06:40:43.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f49850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f49d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563e52e4ef20 con 0x563e52f4c2d0 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e52e4ebc0 con 0x563e52f43c40 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563e52c181c0 con 0x563e52f42f60 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe316a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.994+0000 7fe320839740 10 client.0 _ll_register_callbacks cb 0x563e52bbabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41884/0 (socket says 172.21.15.22:41884) 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 -- 172.21.15.22:0/1347880448 learned_addr learned my addr 172.21.15.22:0/1347880448 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe317400640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f49d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 -- 172.21.15.22:0/1347880448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 msgr2=0x563e52f49850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f49850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 -- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 msgr2=0x563e52f49d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f49d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:43.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 -- 172.21.15.22:0/1347880448 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30800b6b0 con 0x563e52f42f60 2024-03-24T06:40:43.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563e52ea2160 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe316a00640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d770 tx=0x7fe308004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca7d2c7096224f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.995+0000 7fe300c00640 10 client.25858 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe300c00640 1 -- 172.21.15.22:0/1347880448 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30801f070 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe300c00640 1 -- 172.21.15.22:0/1347880448 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe308004040 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe300c00640 1 -- 172.21.15.22:0/1347880448 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe308004240 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe320839740 10 client.25858 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({fsmap=29}) -- 0x563e52ea8b20 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.996+0000 7fe300c00640 1 -- 172.21.15.22:0/1347880448 <== mon.1 v2:172.21.15.155:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe3080043a0 con 0x563e52f42f60 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 10 client.25858 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 10 client.25858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 20 client.25858 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:43.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52f5c570 0x563e52f5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:43.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 4 client.25858 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:40:43.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e52ba97e0 con 0x563e52f5c570 2024-03-24T06:40:43.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.997+0000 7fe316000640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52f5c570 0x563e52f5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:43.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.998+0000 7fe316000640 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52f5c570 0x563e52f5e9b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6957a53e4ce5f0a server_cookie=2f03688555793a9d in_seq=0 out_seq=0 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.998+0000 7fe300c00640 10 client.25858 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe300c00640 1 -- 172.21.15.22:0/1347880448 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e52ba97e0 con 0x563e52f5c570 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe300c00640 10 client.25858 handle_command_reply: tid=0 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 client.25858 shutdown 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 msgr2=0x563e52f49310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:43.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d770 tx=0x7fe308004b20 comp rx=0 tx=0).stop 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 shutdown_connections 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f49d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52f5c570 0x563e52f5e9b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f49850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 --2- 172.21.15.22:0/1347880448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f42f60 0x563e52f49310 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 >> 172.21.15.22:0/1347880448 conn(0x563e52f09b30 msgr2=0x563e52f3f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:43.999+0000 7fe300c00640 0 client.25858 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.000+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 shutdown_connections 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.000+0000 7fe320839740 1 -- 172.21.15.22:0/1347880448 wait complete. 2024-03-24T06:40:44.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.000+0000 7fe320839740 20 client.25858 trim_cache size 0 max 16384 2024-03-24T06:40:44.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.032+0000 7fe320839740 1 Processor -- start 2024-03-24T06:40:44.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.032+0000 7fe320839740 1 -- start start 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f42f60 0x563e52f57c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f3ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe31819f3f0 con 0x563e52f42f60 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e52f3e9b0 con 0x563e52f43c40 2024-03-24T06:40:44.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.033+0000 7fe320839740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563e52f58160 con 0x563e52f4c2d0 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe316a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f42f60 0x563e52f57c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41894/0 (socket says 172.21.15.22:41894) 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 -- 172.21.15.22:0/958279332 learned_addr learned my addr 172.21.15.22:0/958279332 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe316000640 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f3ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 -- 172.21.15.22:0/958279332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 msgr2=0x563e52f3ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f3ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 -- 172.21.15.22:0/958279332 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f42f60 msgr2=0x563e52f57c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f42f60 0x563e52f57c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe317400640 1 -- 172.21.15.22:0/958279332 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e52e449a0 con 0x563e52f4c2d0 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.034+0000 7fe316000640 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f3ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.035+0000 7fe317400640 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe310009620 tx=0x7fe310012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7ec703c67da799 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.035+0000 7fe303400640 1 -- 172.21.15.22:0/958279332 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31001e6e0 con 0x563e52f4c2d0 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.035+0000 7fe303400640 1 -- 172.21.15.22:0/958279332 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe31001ed20 con 0x563e52f4c2d0 2024-03-24T06:40:44.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.035+0000 7fe303400640 1 -- 172.21.15.22:0/958279332 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310026df0 con 0x563e52f4c2d0 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 -- 172.21.15.22:0/958279332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 msgr2=0x563e52f64280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe310009620 tx=0x7fe310012000 comp rx=0 tx=0).stop 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 -- 172.21.15.22:0/958279332 shutdown_connections 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f4c2d0 0x563e52f64280 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f43c40 0x563e52f3ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 --2- 172.21.15.22:0/958279332 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f42f60 0x563e52f57c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.036+0000 7fe320839740 1 -- 172.21.15.22:0/958279332 >> 172.21.15.22:0/958279332 conn(0x563e52f09b30 msgr2=0x563e52f3c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:44.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.037+0000 7fe320839740 1 -- 172.21.15.22:0/958279332 shutdown_connections 2024-03-24T06:40:44.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.037+0000 7fe320839740 1 -- 172.21.15.22:0/958279332 wait complete. 2024-03-24T06:40:44.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.037+0000 7fe320839740 1 -- start start 2024-03-24T06:40:44.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.037+0000 7fe320839740 1 Processor -- start 2024-03-24T06:40:44.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f45ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563e52f54930 con 0x563e52f4c2d0 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e52f54aa0 con 0x563e52f42f60 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563e52f46000 con 0x563e52f43c40 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe320839740 10 client.0 _ll_register_callbacks cb 0x7fe30800b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe316a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe316000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe316a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35420/0 (socket says 172.21.15.22:35420) 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe316000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:41902/0 (socket says 172.21.15.22:41902) 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.038+0000 7fe316a00640 1 -- 172.21.15.22:0/3501847060 learned_addr learned my addr 172.21.15.22:0/3501847060 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 -- 172.21.15.22:0/3501847060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 msgr2=0x563e52f45580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 -- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 msgr2=0x563e52f45ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f45ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 -- 172.21.15.22:0/3501847060 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3100099e0 con 0x563e52f42f60 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316000640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:40:44.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe316a00640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe308004d00 tx=0x7fe30801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a49332e6e5ae9bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe300c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308004560 con 0x563e52f42f60 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.039+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30800de00 con 0x563e52f42f60 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe308000e40 con 0x563e52f42f60 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe308005470 con 0x563e52f42f60 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe320839740 10 client.25823 fetch_fsmap learned FSMap version 29 2024-03-24T06:40:44.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e52e6a480 con 0x563e52f42f60 2024-03-24T06:40:44.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30800de00 con 0x563e52f42f60 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.040+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe308010460 con 0x563e52f42f60 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 10 client.25823 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 10 client.25823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 20 client.25823 populate_metadata read hostname 'smithi022' 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52fc38c0 0x563e52fc5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 4 client.25823 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563e52be6720 con 0x563e52fc38c0 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.041+0000 7fe316000640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52fc38c0 0x563e52fc5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe316000640 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52fc38c0 0x563e52fc5cb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe5c269d2e699913 server_cookie=6fc97cbfeb160a69 in_seq=0 out_seq=0 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe300c00640 10 client.25823 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe300c00640 1 -- 172.21.15.22:0/3501847060 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563e52be6720 con 0x563e52fc38c0 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe300c00640 10 client.25823 handle_command_reply: tid=0 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 client.25823 shutdown 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 msgr2=0x563e52f54250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:44.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe308004d00 tx=0x7fe30801f000 comp rx=0 tx=0).stop 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 shutdown_connections 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563e52f4c2d0 0x563e52f45ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563e52f43c40 0x563e52f45580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563e52fc38c0 0x563e52fc5cb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 --2- 172.21.15.22:0/3501847060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e52f42f60 0x563e52f54250 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe308004d00 tx=0x7fe30801f000 comp rx=0 tx=0).stop 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.042+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 >> 172.21.15.22:0/3501847060 conn(0x563e52f09b30 msgr2=0x563e52f492d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.043+0000 7fe300c00640 0 client.25823 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.043+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 shutdown_connections 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.043+0000 7fe320839740 1 -- 172.21.15.22:0/3501847060 wait complete. 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.043+0000 7fe320839740 20 client.25823 trim_cache size 0 max 16384 2024-03-24T06:40:44.044 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "a31bd469-a648-413c-926a-1df12e01dd2c": { 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "a31bd469-a648-413c-926a-1df12e01dd2c", 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:40:44.045 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 msgr2=0x7fe2fc0a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 0x7fe2fc0a4800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe310009270 tx=0x7fe310009480 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 msgr2=0x7fe31819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe304000c00 tx=0x7fe304004040 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 shutdown_connections 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe2fc0a2340 0x7fe2fc0a4800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe318107660 0x7fe31819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe318101f60 0x7fe31819a2d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 --2- 172.21.15.22:0/2880410317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3181015a0 0x7fe318199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 >> 172.21.15.22:0/2880410317 conn(0x7fe3180fd0b0 msgr2=0x7fe318074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 shutdown_connections 2024-03-24T06:40:44.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:40:44.044+0000 7fe317e00640 1 -- 172.21.15.22:0/2880410317 wait complete. 2024-03-24T06:40:44.061 INFO:tasks.cephfs.filesystem:scrub status for tag:a31bd469-a648-413c-926a-1df12e01dd2c - {'path': '/', 'tag': 'a31bd469-a648-413c-926a-1df12e01dd2c', 'options': 'recursive,force'} 2024-03-24T06:40:45.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:45 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:40:45.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:45 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:45 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:40:45.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:45 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:45 smithi155 ceph-mon[36262]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:45.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:45 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:40:45.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:45 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:45 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:40:45.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:45 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:45 smithi022 ceph-mon[32003]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:45 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:40:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:45 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:45 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:40:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:45 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:40:45.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:45 smithi143 ceph-mon[36520]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:40:47.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:47 smithi155 ceph-mon[36262]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-24T06:40:47.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:47 smithi022 ceph-mon[32003]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-24T06:40:47.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:47 smithi143 ceph-mon[36520]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-24T06:40:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:48 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:40:48.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:48 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:40:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:48 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:40:49.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:49 smithi155 ceph-mon[36262]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:40:49.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:49 smithi022 ceph-mon[32003]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:40:49.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:49 smithi143 ceph-mon[36520]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-24T06:40:51.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:51 smithi155 ceph-mon[36262]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:40:51.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:51 smithi022 ceph-mon[32003]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:40:51.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:51 smithi143 ceph-mon[36520]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:40:53.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:53 smithi155 ceph-mon[36262]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-24T06:40:53.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:53 smithi022 ceph-mon[32003]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-24T06:40:53.677 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:53 smithi143 ceph-mon[36520]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-24T06:40:55.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:55 smithi155 ceph-mon[36262]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:40:55.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:55 smithi022 ceph-mon[32003]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:40:55.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:55 smithi143 ceph-mon[36520]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:40:57.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:57 smithi022 ceph-mon[32003]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-24T06:40:57.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:57 smithi143 ceph-mon[36520]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-24T06:40:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:57 smithi155 ceph-mon[36262]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-24T06:40:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:40:59 smithi022 ceph-mon[32003]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T06:40:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:40:59 smithi143 ceph-mon[36520]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T06:40:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:40:59 smithi155 ceph-mon[36262]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-24T06:41:01.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:41:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:41:00.772+0000 7ff3aa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:41:01.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:41:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:41:00.772+0000 7ff3aa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:41:01.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:41:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:41:00.772+0000 7ff3aa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:41:01.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:41:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:41:00.772+0000 7ff3aa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:41:01.149 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:41:00 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:41:00.772+0000 7ff3aa200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T06:41:01.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:01 smithi022 ceph-mon[32003]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:41:01.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:01 smithi143 ceph-mon[36520]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:41:01.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:01 smithi155 ceph-mon[36262]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T06:41:03.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:03 smithi022 ceph-mon[32003]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-24T06:41:03.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:03 smithi022 ceph-mon[32003]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-03-24T06:41:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:03 smithi143 ceph-mon[36520]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-24T06:41:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:03 smithi143 ceph-mon[36520]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-03-24T06:41:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:03 smithi155 ceph-mon[36262]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-24T06:41:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:03 smithi155 ceph-mon[36262]: mgrmap e26: x(active, since 62s), standbys: y, z 2024-03-24T06:41:05.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:05 smithi022 ceph-mon[32003]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T06:41:05.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:05 smithi143 ceph-mon[36520]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T06:41:05.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:05 smithi155 ceph-mon[36262]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T06:41:07.350 INFO:tasks.workunit.client.0.smithi022.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T06:41:07.350 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.350 INFO:tasks.workunit.client.0.smithi022.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr: git switch -c 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.351 INFO:tasks.workunit.client.0.smithi022.stderr:Or undo this operation with: 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr: git switch - 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T06:41:07.352 INFO:tasks.workunit.client.0.smithi022.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T06:41:07.374 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T06:41:07.437 INFO:tasks.workunit.client.0.smithi022.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T06:41:07.444 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T06:41:07.444 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T06:41:07.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:07 smithi022 ceph-mon[32003]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T06:41:07.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:07 smithi143 ceph-mon[36520]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T06:41:07.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:07 smithi155 ceph-mon[36262]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-24T06:41:08.363 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T06:41:08.494 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T06:41:08.587 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T06:41:08.594 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T06:41:08.594 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T06:41:08.676 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T06:41:08.686 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:41:08.686 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T06:41:08.743 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-24T06:41:08.744 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-24T06:41:08.744 DEBUG:teuthology.orchestra.run.smithi022:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-24T06:41:08.812 INFO:tasks.workunit.client.0.smithi022.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-24T06:41:08.814 INFO:tasks.workunit.client.0.smithi022.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-24T06:41:08.814 INFO:tasks.workunit.client.0.smithi022.stderr:+ trap -- retry EXIT 2024-03-24T06:41:08.814 INFO:tasks.workunit.client.0.smithi022.stderr:+ rm -rf ffsb 2024-03-24T06:41:08.818 INFO:tasks.workunit.client.0.smithi022.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-24T06:41:08.825 INFO:tasks.workunit.client.0.smithi022.stderr:Cloning into 'ffsb'... 2024-03-24T06:41:09.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:09 smithi022 ceph-mon[32003]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T06:41:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:09 smithi143 ceph-mon[36520]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T06:41:09.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:09 smithi155 ceph-mon[36262]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-24T06:41:10.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:10 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:41:10.669 INFO:tasks.workunit.client.0.smithi022.stderr:+ trap - EXIT 2024-03-24T06:41:10.669 INFO:tasks.workunit.client.0.smithi022.stderr:+ cd ffsb 2024-03-24T06:41:10.669 INFO:tasks.workunit.client.0.smithi022.stderr:+ ./configure 2024-03-24T06:41:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:10 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:41:10.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:10 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:41:10.979 INFO:tasks.workunit.client.0.smithi022.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T06:41:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:11 smithi022 ceph-mon[32003]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T06:41:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:11 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:41:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:11 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:41:11.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:11 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:41:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:11 smithi143 ceph-mon[36520]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T06:41:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:11 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:41:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:11 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:41:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:11 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:41:11.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:11 smithi155 ceph-mon[36262]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-24T06:41:11.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:11 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:41:11.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:11 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:41:11.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:11 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:41:12.008 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether build environment is sane... yes 2024-03-24T06:41:12.034 INFO:tasks.workunit.client.0.smithi022.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-24T06:41:12.034 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gawk... gawk 2024-03-24T06:41:12.068 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether make sets $(MAKE)... yes 2024-03-24T06:41:12.154 INFO:tasks.workunit.client.0.smithi022.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-24T06:41:12.154 INFO:tasks.workunit.client.0.smithi022.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-24T06:41:12.172 INFO:tasks.workunit.client.0.smithi022.stdout:checking for style of include used by make... GNU 2024-03-24T06:41:12.182 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc... gcc 2024-03-24T06:41:12.309 INFO:tasks.workunit.client.0.smithi022.stdout:checking for C compiler default output file name... a.out 2024-03-24T06:41:12.317 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether the C compiler works... yes 2024-03-24T06:41:12.325 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are cross compiling... no 2024-03-24T06:41:12.416 INFO:tasks.workunit.client.0.smithi022.stdout:checking for suffix of executables... 2024-03-24T06:41:12.496 INFO:tasks.workunit.client.0.smithi022.stdout:checking for suffix of object files... o 2024-03-24T06:41:12.570 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T06:41:12.652 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether gcc accepts -g... yes 2024-03-24T06:41:13.057 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-24T06:41:13.324 INFO:tasks.workunit.client.0.smithi022.stdout:checking dependency style of gcc... gcc3 2024-03-24T06:41:13.457 INFO:tasks.workunit.client.0.smithi022.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T06:41:13.551 INFO:tasks.workunit.client.0.smithi022.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-24T06:41:13.554 INFO:tasks.workunit.client.0.smithi022.stdout:checking for egrep... /usr/bin/grep -E 2024-03-24T06:41:13.578 INFO:tasks.workunit.client.0.smithi022.stdout:checking for AIX... no 2024-03-24T06:41:13.578 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:13 smithi155 ceph-mon[36262]: pgmap v39: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-03-24T06:41:13.613 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc... (cached) gcc 2024-03-24T06:41:13.646 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-24T06:41:13.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:13 smithi022 ceph-mon[32003]: pgmap v39: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-03-24T06:41:13.650 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-24T06:41:13.650 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-24T06:41:13.650 INFO:tasks.workunit.client.0.smithi022.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-24T06:41:13.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:13 smithi143 ceph-mon[36520]: pgmap v39: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-03-24T06:41:13.817 INFO:tasks.workunit.client.0.smithi022.stdout:checking for main in -lm... yes 2024-03-24T06:41:13.949 INFO:tasks.workunit.client.0.smithi022.stdout:checking for main in -lpthread... yes 2024-03-24T06:41:14.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:41:14.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:41:14.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 -- 172.21.15.22:0/31660894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97010df60 msgr2=0x7fd97010e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 --2- 172.21.15.22:0/31660894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97010df60 0x7fd97010e3e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd958009900 tx=0x7fd95802bc40 comp rx=0 tx=0).stop 2024-03-24T06:41:14.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 -- 172.21.15.22:0/31660894 shutdown_connections 2024-03-24T06:41:14.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 --2- 172.21.15.22:0/31660894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97011aef0 0x7fd97011d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 --2- 172.21.15.22:0/31660894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97010df60 0x7fd97010e3e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 --2- 172.21.15.22:0/31660894 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd97010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 -- 172.21.15.22:0/31660894 >> 172.21.15.22:0/31660894 conn(0x7fd97006c7f0 msgr2=0x7fd97006cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 -- 172.21.15.22:0/31660894 shutdown_connections 2024-03-24T06:41:14.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.330+0000 7fd974c00640 1 -- 172.21.15.22:0/31660894 wait complete. 2024-03-24T06:41:14.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.332+0000 7fd974c00640 1 Processor -- start 2024-03-24T06:41:14.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.332+0000 7fd974c00640 1 -- start start 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd9701acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97011aef0 0x7fd9701a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd9701a7030 con 0x7fd97010ff30 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd9701a7170 con 0x7fd97011aef0 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd974c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd9701a7330 con 0x7fd97010df60 2024-03-24T06:41:14.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd96de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd96de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42048/0 (socket says 172.21.15.22:42048) 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.333+0000 7fd96de00640 1 -- 172.21.15.22:0/2697770910 learned_addr learned my addr 172.21.15.22:0/2697770910 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96e800640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97011aef0 0x7fd9701a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96d400640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd9701acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96de00640 1 -- 172.21.15.22:0/2697770910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97011aef0 msgr2=0x7fd9701a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96de00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97011aef0 0x7fd9701a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96de00640 1 -- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 msgr2=0x7fd9701acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96de00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd9701acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96de00640 1 -- 172.21.15.22:0/2697770910 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd958009560 con 0x7fd97010df60 2024-03-24T06:41:14.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.334+0000 7fd96d400640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd9701acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:14.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.335+0000 7fd96de00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd96800ebf0 tx=0x7fd96800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.336+0000 7fd966a00640 1 -- 172.21.15.22:0/2697770910 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968010630 con 0x7fd97010df60 2024-03-24T06:41:14.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.336+0000 7fd966a00640 1 -- 172.21.15.22:0/2697770910 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96800ced0 con 0x7fd97010df60 2024-03-24T06:41:14.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.336+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9701a7610 con 0x7fd97010df60 2024-03-24T06:41:14.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.336+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9701108b0 con 0x7fd97010df60 2024-03-24T06:41:14.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.336+0000 7fd966a00640 1 -- 172.21.15.22:0/2697770910 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968018b30 con 0x7fd97010df60 2024-03-24T06:41:14.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.342+0000 7fd977983740 1 Processor -- start 2024-03-24T06:41:14.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.342+0000 7fd977983740 1 -- start start 2024-03-24T06:41:14.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.342+0000 7fd966a00640 1 -- 172.21.15.22:0/2697770910 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7fd968018c90 con 0x7fd97010df60 2024-03-24T06:41:14.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be294757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be294814e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be293e4790 con 0x55be2947f0f0 2024-03-24T06:41:14.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be293e1cc0 con 0x55be29476a60 2024-03-24T06:41:14.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd977983740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be293dd4a0 con 0x55be29475d80 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd96de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd96de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42060/0 (socket says 172.21.15.22:42060) 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd96de00640 1 -- 172.21.15.22:0/3733206729 learned_addr learned my addr 172.21.15.22:0/3733206729 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd96d400640 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be294814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.343+0000 7fd96e800640 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be294757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 -- 172.21.15.22:0/3733206729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 msgr2=0x55be294757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be294757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 -- 172.21.15.22:0/3733206729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 msgr2=0x55be294814e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be294814e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 -- 172.21.15.22:0/3733206729 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be2939ec20 con 0x55be29475d80 2024-03-24T06:41:14.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd96de00640 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cc090 tx=0x7fd9680d7650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abe2c7163b708bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.344+0000 7fd966a00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 0x7fd9540a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.345+0000 7fd966a00640 1 -- 172.21.15.22:0/2697770910 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd968014070 con 0x7fd97010df60 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.345+0000 7fd95f000640 1 -- 172.21.15.22:0/3733206729 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680973f0 con 0x55be29475d80 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.345+0000 7fd95f000640 1 -- 172.21.15.22:0/3733206729 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96809af20 con 0x55be29475d80 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.345+0000 7fd95f000640 1 -- 172.21.15.22:0/3733206729 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968018420 con 0x55be29475d80 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd96e800640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 0x7fd9540a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 -- 172.21.15.22:0/3733206729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 msgr2=0x55be29476200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cc090 tx=0x7fd9680d7650 comp rx=0 tx=0).stop 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 -- 172.21.15.22:0/3733206729 shutdown_connections 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be294814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29475d80 0x55be29476200 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 --2- 172.21.15.22:0/3733206729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be294757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.346+0000 7fd977983740 1 -- 172.21.15.22:0/3733206729 >> 172.21.15.22:0/3733206729 conn(0x55be2943cb30 msgr2=0x55be294707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.347+0000 7fd977983740 1 -- 172.21.15.22:0/3733206729 shutdown_connections 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.347+0000 7fd977983740 1 -- 172.21.15.22:0/3733206729 wait complete. 2024-03-24T06:41:14.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.347+0000 7fd977983740 1 -- start start 2024-03-24T06:41:14.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.347+0000 7fd96e800640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 0x7fd9540a4860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd960009a70 tx=0x7fd9600090e0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.347+0000 7fd977983740 1 Processor -- start 2024-03-24T06:41:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be2947c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2947cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be29381f20 con 0x55be2947f0f0 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be29381bc0 con 0x55be29475d80 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be2914b1c0 con 0x55be29476a60 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd96de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd977983740 10 client.0 _ll_register_callbacks cb 0x55be290edbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd96de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35896/0 (socket says 172.21.15.22:35896) 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.348+0000 7fd96de00640 1 -- 172.21.15.22:0/2715041690 learned_addr learned my addr 172.21.15.22:0/2715041690 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 -- 172.21.15.22:0/2715041690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 msgr2=0x55be2947c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be2947c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 -- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 msgr2=0x55be2947cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2947cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 -- 172.21.15.22:0/2715041690 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9701108b0 con 0x55be29475d80 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd96de00640 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cc700 tx=0x7fd968009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6804976e702bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd953400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680d2e50 con 0x55be29475d80 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd968010e00 con 0x55be29475d80 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.349+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be293d5160 con 0x55be29475d80 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd96809f0d0 con 0x55be29475d80 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 10 client.25841 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55be293dbb20 con 0x55be29475d80 2024-03-24T06:41:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55be293d5160 con 0x55be29475d80 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd96809b3b0 con 0x55be29475d80 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 10 client.25841 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 20 client.25841 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be2948f280 0x55be294916b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 4 client.25841 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.350+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be290dc7e0 con 0x55be2948f280 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.351+0000 7fd96d400640 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be2948f280 0x55be294916b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.351+0000 7fd96d400640 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be2948f280 0x55be294916b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9266e7a8fa4cd391 server_cookie=8e8c0980dd5a200d in_seq=0 out_seq=0 2024-03-24T06:41:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.351+0000 7fd953400640 10 client.25841 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd953400640 1 -- 172.21.15.22:0/2715041690 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be290dc7e0 con 0x55be2948f280 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd953400640 10 client.25841 handle_command_reply: tid=0 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 client.25841 shutdown 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 msgr2=0x55be2947c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd9680cc700 tx=0x7fd968009480 comp rx=0 tx=0).stop 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 shutdown_connections 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2947cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be2947c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be2948f280 0x55be294916b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 --2- 172.21.15.22:0/2715041690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947c120 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 >> 172.21.15.22:0/2715041690 conn(0x55be2943cb30 msgr2=0x55be29472520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.354+0000 7fd953400640 0 client.25841 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.355+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 shutdown_connections 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.355+0000 7fd977983740 1 -- 172.21.15.22:0/2715041690 wait complete. 2024-03-24T06:41:14.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.355+0000 7fd977983740 20 client.25841 trim_cache size 0 max 16384 2024-03-24T06:41:14.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 Processor -- start 2024-03-24T06:41:14.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 -- start start 2024-03-24T06:41:14.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2948aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be29481ac0 con 0x55be2947f0f0 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be29475000 con 0x55be29475d80 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd977983740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be2946ac30 con 0x55be29476a60 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd96d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd96de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd96d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42086/0 (socket says 172.21.15.22:42086) 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd96d400640 1 -- 172.21.15.22:0/811811641 learned_addr learned my addr 172.21.15.22:0/811811641 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.375+0000 7fd96e800640 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2948aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 -- 172.21.15.22:0/811811641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 msgr2=0x55be2947b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 -- 172.21.15.22:0/811811641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 msgr2=0x55be2948aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2948aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 -- 172.21.15.22:0/811811641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be293779a0 con 0x55be29476a60 2024-03-24T06:41:14.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96de00640 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.376+0000 7fd96d400640 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd9701a81d0 tx=0x7fd958009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc73d7bb010b7a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd95e600640 1 -- 172.21.15.22:0/811811641 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958047aa0 con 0x55be29476a60 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd95e600640 1 -- 172.21.15.22:0/811811641 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd958045040 con 0x55be29476a60 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd95e600640 1 -- 172.21.15.22:0/811811641 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9580472d0 con 0x55be29476a60 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd977983740 1 -- 172.21.15.22:0/811811641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 msgr2=0x55be29470300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd977983740 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd9701a81d0 tx=0x7fd958009260 comp rx=0 tx=0).stop 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd977983740 1 -- 172.21.15.22:0/811811641 shutdown_connections 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.377+0000 7fd977983740 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be2947f0f0 0x55be2948aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be29476a60 0x55be29470300 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 --2- 172.21.15.22:0/811811641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29475d80 0x55be2947b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 -- 172.21.15.22:0/811811641 >> 172.21.15.22:0/811811641 conn(0x55be2943cb30 msgr2=0x55be294724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 -- 172.21.15.22:0/811811641 shutdown_connections 2024-03-24T06:41:14.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 -- 172.21.15.22:0/811811641 wait complete. 2024-03-24T06:41:14.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 -- start start 2024-03-24T06:41:14.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.378+0000 7fd977983740 1 Processor -- start 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 0x55be2947fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be29480180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be29485140 con 0x55be29475d80 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be294852b0 con 0x55be29476a60 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be29485470 con 0x55be2947f0f0 2024-03-24T06:41:14.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd977983740 10 client.0 _ll_register_callbacks cb 0x7fd9680093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42102/0 (socket says 172.21.15.22:42102) 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96e800640 1 -- 172.21.15.22:0/3729351164 learned_addr learned my addr 172.21.15.22:0/3729351164 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96d400640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be29480180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96de00640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 0x55be2947fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96e800640 1 -- 172.21.15.22:0/3729351164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 msgr2=0x55be29480180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.379+0000 7fd96e800640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be29480180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd96e800640 1 -- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 msgr2=0x55be2947fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd96e800640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 0x55be2947fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd96e800640 1 -- 172.21.15.22:0/3729351164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be293dbb20 con 0x55be2947f0f0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd96de00640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 0x55be2947fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd96e800640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9600149c0 tx=0x7fd9600149f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46015e306f9959ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd953400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:14.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960023690 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be29470960 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.380+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd96001c040 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9600237f0 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 10 client.25882 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fd97011dc10 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55be29470960 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd960022c70 con 0x55be2947f0f0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 10 client.25882 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 20 client.25882 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be294f60e0 0x55be294f8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 4 client.25882 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55be29476700 con 0x55be294f60e0 2024-03-24T06:41:14.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.381+0000 7fd96de00640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be294f60e0 0x55be294f8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.382+0000 7fd96de00640 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be294f60e0 0x55be294f8500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d5f6fdd73f2190 server_cookie=31003a0dc7c6a79c in_seq=0 out_seq=0 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.382+0000 7fd953400640 10 client.25882 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd953400640 1 -- 172.21.15.22:0/3729351164 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55be29476700 con 0x55be294f60e0 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd953400640 10 client.25882 handle_command_reply: tid=0 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 client.25882 shutdown 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 msgr2=0x55be294806c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9600149c0 tx=0x7fd9600149f0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 shutdown_connections 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be2947f0f0 0x55be294806c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55be294f60e0 0x55be294f8500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.384+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be29476a60 0x55be29480180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.385+0000 7fd977983740 1 --2- 172.21.15.22:0/3729351164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be29475d80 0x55be2947fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.385+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 >> 172.21.15.22:0/3729351164 conn(0x55be2943cb30 msgr2=0x55be29471a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.385+0000 7fd953400640 0 client.25882 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:14.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.386+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 shutdown_connections 2024-03-24T06:41:14.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.386+0000 7fd977983740 1 -- 172.21.15.22:0/3729351164 wait complete. 2024-03-24T06:41:14.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.386+0000 7fd977983740 20 client.25882 trim_cache size 0 max 16384 2024-03-24T06:41:14.386 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 msgr2=0x7fd9540a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 0x7fd9540a4860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd960009a70 tx=0x7fd9600090e0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 msgr2=0x7fd9701ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd96800ebf0 tx=0x7fd96800c6a0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 shutdown_connections 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd9540a23a0 0x7fd9540a4860 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd97011aef0 0x7fd9701a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd97010ff30 0x7fd9701acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 --2- 172.21.15.22:0/2697770910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97010df60 0x7fd9701ac830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 >> 172.21.15.22:0/2697770910 conn(0x7fd97006c7f0 msgr2=0x7fd9700721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 shutdown_connections 2024-03-24T06:41:14.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.387+0000 7fd974c00640 1 -- 172.21.15.22:0/2697770910 wait complete. 2024-03-24T06:41:14.398 INFO:tasks.workunit.client.0.smithi022.stdout:checking for ANSI C header files... yes 2024-03-24T06:41:14.408 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:41:14.409 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:41:14.567 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-24T06:41:14.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 -- 172.21.15.22:0/1519332345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601038a0 msgr2=0x7f3760103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 --2- 172.21.15.22:0/1519332345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601038a0 0x7f3760103d20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f374c009900 tx=0x7f374c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:41:14.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 -- 172.21.15.22:0/1519332345 shutdown_connections 2024-03-24T06:41:14.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 --2- 172.21.15.22:0/1519332345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 0x7f3760110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 --2- 172.21.15.22:0/1519332345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601038a0 0x7f3760103d20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 --2- 172.21.15.22:0/1519332345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601098b0 0x7f3760109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 -- 172.21.15.22:0/1519332345 >> 172.21.15.22:0/1519332345 conn(0x7f37600ff370 msgr2=0x7f3760101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 -- 172.21.15.22:0/1519332345 shutdown_connections 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.642+0000 7f3768000640 1 -- 172.21.15.22:0/1519332345 wait complete. 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.643+0000 7f3768000640 1 Processor -- start 2024-03-24T06:41:14.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 -- start start 2024-03-24T06:41:14.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 0x7f3760199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 0x7f376019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 0x7f376019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f376019edf0 con 0x7f3760104260 2024-03-24T06:41:14.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f376019ef60 con 0x7f37601098b0 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.644+0000 7f3768000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f376019f120 con 0x7f37601038a0 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3765600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 0x7f3760199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 0x7f376019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3765600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 0x7f3760199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42128/0 (socket says 172.21.15.22:42128) 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3765600640 1 -- 172.21.15.22:0/1666067488 learned_addr learned my addr 172.21.15.22:0/1666067488 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:14.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3764c00640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 0x7f376019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 -- 172.21.15.22:0/1666067488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 msgr2=0x7f3760199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 0x7f3760199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 -- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 msgr2=0x7f376019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 0x7f376019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 -- 172.21.15.22:0/1666067488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c009560 con 0x7f37601098b0 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.645+0000 7f3766000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 0x7f376019e7f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3758009fd0 tx=0x7f375800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.646+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375800ce20 con 0x7f37601098b0 2024-03-24T06:41:14.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.646+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3758005020 con 0x7f37601098b0 2024-03-24T06:41:14.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.646+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758010690 con 0x7f37601098b0 2024-03-24T06:41:14.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.646+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376019f400 con 0x7f37601098b0 2024-03-24T06:41:14.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.646+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37601a6c80 con 0x7f37601098b0 2024-03-24T06:41:14.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.647+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3760068950 con 0x7f37601098b0 2024-03-24T06:41:14.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.649+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7f3758005500 con 0x7f37601098b0 2024-03-24T06:41:14.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.650+0000 7f375e000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 0x7f37480a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:14.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.650+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3758014070 con 0x7f37601098b0 2024-03-24T06:41:14.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.651+0000 7f3765600640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 0x7f37480a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:14.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.651+0000 7f3765600640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 0x7f37480a4890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c10 tx=0x7f3754005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:14.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.654+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3758097470 con 0x7f37601098b0 2024-03-24T06:41:14.666 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/types.h... yes 2024-03-24T06:41:14.758 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/stat.h... yes 2024-03-24T06:41:14.864 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdlib.h... yes 2024-03-24T06:41:14.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.954+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f376010be10 con 0x7f37601098b0 2024-03-24T06:41:14.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.955+0000 7f375e000640 1 -- 172.21.15.22:0/1666067488 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11939 (secure 0 0 0) 0x7f3758005190 con 0x7f37601098b0 2024-03-24T06:41:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:41:14.956 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:39:58.997320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:41:14.956 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 29 2024-03-24T06:41:14.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 msgr2=0x7f37480a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 0x7f37480a4890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c10 tx=0x7f3754005f90 comp rx=0 tx=0).stop 2024-03-24T06:41:14.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 msgr2=0x7f376019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 0x7f376019e7f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3758009fd0 tx=0x7f375800ef10 comp rx=0 tx=0).stop 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 shutdown_connections 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f37480a23d0 0x7f37480a4890 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37601098b0 0x7f376019e7f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3760104260 0x7f376019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 --2- 172.21.15.22:0/1666067488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37601038a0 0x7f3760199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 >> 172.21.15.22:0/1666067488 conn(0x7f37600ff370 msgr2=0x7f37600ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 shutdown_connections 2024-03-24T06:41:14.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:14.958+0000 7f3768000640 1 -- 172.21.15.22:0/1666067488 wait complete. 2024-03-24T06:41:14.973 INFO:tasks.workunit.client.0.smithi022.stdout:checking for string.h... yes 2024-03-24T06:41:14.982 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:41:15.081 INFO:tasks.workunit.client.0.smithi022.stdout:checking for memory.h... yes 2024-03-24T06:41:15.193 INFO:tasks.workunit.client.0.smithi022.stdout:checking for strings.h... yes 2024-03-24T06:41:15.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 -- 172.21.15.22:0/1122620257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4103790 msgr2=0x7f1ff4103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 --2- 172.21.15.22:0/1122620257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4103790 0x7f1ff4103c10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009860 tx=0x7f1fe002bb70 comp rx=0 tx=0).stop 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 -- 172.21.15.22:0/1122620257 shutdown_connections 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 --2- 172.21.15.22:0/1122620257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff4104150 0x7f1ff4110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 --2- 172.21.15.22:0/1122620257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4103790 0x7f1ff4103c10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 --2- 172.21.15.22:0/1122620257 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff41097a0 0x7f1ff4109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.194+0000 7f1ffa800640 1 -- 172.21.15.22:0/1122620257 >> 172.21.15.22:0/1122620257 conn(0x7f1ff40ff220 msgr2=0x7f1ff4101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.195+0000 7f1ffa800640 1 -- 172.21.15.22:0/1122620257 shutdown_connections 2024-03-24T06:41:15.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.195+0000 7f1ffa800640 1 -- 172.21.15.22:0/1122620257 wait complete. 2024-03-24T06:41:15.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.196+0000 7f1ffa800640 1 Processor -- start 2024-03-24T06:41:15.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.196+0000 7f1ffa800640 1 -- start start 2024-03-24T06:41:15.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff4103790 0x7f1ff4199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 0x7f1ff419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff419ed50 con 0x7f1ff4103790 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff419eec0 con 0x7f1ff4104150 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ffa800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff419f080 con 0x7f1ff41097a0 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ff8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff4103790 0x7f1ff4199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ff8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35448/0 (socket says 172.21.15.22:35448) 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ff8a00640 1 -- 172.21.15.22:0/3038005390 learned_addr learned my addr 172.21.15.22:0/3038005390 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.197+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 0x7f1ff419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 -- 172.21.15.22:0/3038005390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 msgr2=0x7f1ff419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 0x7f1ff419e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 -- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff4103790 msgr2=0x7f1ff4199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff4103790 0x7f1ff4199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 -- 172.21.15.22:0/3038005390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec009900 con 0x7f1ff4104150 2024-03-24T06:41:15.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 0x7f1ff419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:15.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0037040 tx=0x7f1fe00045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff2000640 1 -- 172.21.15.22:0/3038005390 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00349d0 con 0x7f1ff4104150 2024-03-24T06:41:15.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff2000640 1 -- 172.21.15.22:0/3038005390 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe002c9f0 con 0x7f1ff4104150 2024-03-24T06:41:15.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe0009490 con 0x7f1ff4104150 2024-03-24T06:41:15.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.198+0000 7f1ff2000640 1 -- 172.21.15.22:0/3038005390 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00035f0 con 0x7f1ff4104150 2024-03-24T06:41:15.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.199+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff419f530 con 0x7f1ff4104150 2024-03-24T06:41:15.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.202+0000 7f1ff2000640 1 -- 172.21.15.22:0/3038005390 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7f1fe003b040 con 0x7f1ff4104150 2024-03-24T06:41:15.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.203+0000 7f1ffeb87740 1 Processor -- start 2024-03-24T06:41:15.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.203+0000 7f1ffeb87740 1 -- start start 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.203+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.203+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd450f950 0x55bcd4511d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bcd44746e0 con 0x55bcd4506d70 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4471c10 con 0x55bcd4506450 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcd446d3f0 con 0x55bcd450f950 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35458/0 (socket says 172.21.15.22:35458) 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 -- 172.21.15.22:0/1435587163 learned_addr learned my addr 172.21.15.22:0/1435587163 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd450f950 0x55bcd4511d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9400640 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff2000640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 0x7f1fdc0a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 -- 172.21.15.22:0/1435587163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd450f950 msgr2=0x55bcd4511d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd450f950 0x55bcd4511d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 -- 172.21.15.22:0/1435587163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 msgr2=0x55bcd450f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.204+0000 7f1ff9e00640 1 -- 172.21.15.22:0/1435587163 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd442ee40 con 0x55bcd4506450 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1ff2000640 1 -- 172.21.15.22:0/3038005390 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fe00f0070 con 0x7f1ff4104150 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00eac0 tx=0x7f1fec00ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=487f8dd53a744f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 0x7f1fdc0a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1fe6600640 1 -- 172.21.15.22:0/1435587163 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0183f0 con 0x55bcd4506450 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1fe6600640 1 -- 172.21.15.22:0/1435587163 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec014040 con 0x55bcd4506450 2024-03-24T06:41:15.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.205+0000 7f1fe6600640 1 -- 172.21.15.22:0/1435587163 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec021a40 con 0x55bcd4506450 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 0x7f1fdc0a4890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009fd0 tx=0x7f1fec021c60 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1435587163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 msgr2=0x55bcd4506830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00eac0 tx=0x7f1fec00ef80 comp rx=0 tx=0).stop 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1435587163 shutdown_connections 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd450f950 0x55bcd4511d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1435587163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd4506830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1435587163 >> 172.21.15.22:0/1435587163 conn(0x55bcd4500b80 msgr2=0x55bcd4500f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1435587163 shutdown_connections 2024-03-24T06:41:15.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1435587163 wait complete. 2024-03-24T06:41:15.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.207+0000 7f1ffeb87740 1 -- start start 2024-03-24T06:41:15.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.208+0000 7f1ffeb87740 1 Processor -- start 2024-03-24T06:41:15.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd450d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4412060 con 0x55bcd4506d70 2024-03-24T06:41:15.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4411d00 con 0x55bcd450f950 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcd41db1c0 con 0x55bcd4506450 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ffeb87740 10 client.0 _ll_register_callbacks cb 0x55bcd417dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ff8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39584/0 (socket says 172.21.15.22:39584) 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ff9400640 1 -- 172.21.15.22:0/2800397611 learned_addr learned my addr 172.21.15.22:0/2800397611 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.209+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd450d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 -- 172.21.15.22:0/2800397611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 msgr2=0x55bcd450d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd450d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 -- 172.21.15.22:0/2800397611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 msgr2=0x55bcd450ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 -- 172.21.15.22:0/2800397611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec00e7a0 con 0x55bcd4506450 2024-03-24T06:41:15.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80126e0 tx=0x7f1fe8012ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5126815b29544f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1fdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8017070 con 0x55bcd4506450 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fe800f670 con 0x55bcd4506450 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.210+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bcd416c7e0 con 0x55bcd4506450 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1fe80122e0 con 0x55bcd4506450 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1ffeb87740 10 client.25900 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcd446ba70 con 0x55bcd4506450 2024-03-24T06:41:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bcd416c7e0 con 0x55bcd4506450 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1fe8010b50 con 0x55bcd4506450 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1ffeb87740 10 client.25900 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1ffeb87740 10 client.25900 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.211+0000 7f1ffeb87740 20 client.25900 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.212+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd451fdd0 0x55bcd4522230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.212+0000 7f1ffeb87740 4 client.25900 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.212+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcd41a9720 con 0x55bcd451fdd0 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.212+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd451fdd0 0x55bcd4522230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.213+0000 7f1fdaa00640 10 client.25900 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.212+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd451fdd0 0x55bcd4522230 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afc996d97879a8e0 server_cookie=cb007983ec550ce0 in_seq=0 out_seq=0 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2800397611 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bcd41a9720 con 0x55bcd451fdd0 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1fdaa00640 10 client.25900 handle_command_reply: tid=0 2024-03-24T06:41:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 client.25900 shutdown 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 msgr2=0x55bcd450c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80126e0 tx=0x7f1fe8012ae0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 shutdown_connections 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd451fdd0 0x55bcd4522230 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd450d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd450ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2800397611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd450c8c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1fdaa00640 0 client.25900 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.214+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 >> 172.21.15.22:0/2800397611 conn(0x55bcd4500b80 msgr2=0x55bcd45028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.215+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 shutdown_connections 2024-03-24T06:41:15.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.215+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2800397611 wait complete. 2024-03-24T06:41:15.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.215+0000 7f1ffeb87740 20 client.25900 trim_cache size 0 max 16384 2024-03-24T06:41:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 Processor -- start 2024-03-24T06:41:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 -- start start 2024-03-24T06:41:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506d70 0x55bcd451b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd450f950 0x55bcd4527b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4511d40 con 0x55bcd450f950 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bcd451b720 con 0x55bcd4506450 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcd44fbb50 con 0x55bcd4506d70 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.234+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35470/0 (socket says 172.21.15.22:35470) 2024-03-24T06:41:15.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 -- 172.21.15.22:0/1717506945 learned_addr learned my addr 172.21.15.22:0/1717506945 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506d70 0x55bcd451b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 -- 172.21.15.22:0/1717506945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506d70 msgr2=0x55bcd451b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506d70 0x55bcd451b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd450f950 0x55bcd4527b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 -- 172.21.15.22:0/1717506945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd450f950 msgr2=0x55bcd4527b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd450f950 0x55bcd4527b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 -- 172.21.15.22:0/1717506945 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd4407b00 con 0x55bcd4506450 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1ff9400640 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000c00 tx=0x7f1fe8012360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b631077140fe0362 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.235+0000 7f1fe5c00640 1 -- 172.21.15.22:0/1717506945 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8017040 con 0x55bcd4506450 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.236+0000 7f1fe5c00640 1 -- 172.21.15.22:0/1717506945 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe800f670 con 0x55bcd4506450 2024-03-24T06:41:15.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.236+0000 7f1fe5c00640 1 -- 172.21.15.22:0/1717506945 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80107a0 con 0x55bcd4506450 2024-03-24T06:41:15.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1717506945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 msgr2=0x55bcd450eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000c00 tx=0x7f1fe8012360 comp rx=0 tx=0).stop 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1717506945 shutdown_connections 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd450f950 0x55bcd4527b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506d70 0x55bcd451b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/1717506945 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd4506450 0x55bcd450eff0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1717506945 >> 172.21.15.22:0/1717506945 conn(0x55bcd4500b80 msgr2=0x55bcd4500f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1717506945 shutdown_connections 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- 172.21.15.22:0/1717506945 wait complete. 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.237+0000 7f1ffeb87740 1 -- start start 2024-03-24T06:41:15.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.238+0000 7f1ffeb87740 1 Processor -- start 2024-03-24T06:41:15.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.238+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.238+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd4508f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.238+0000 7f1ffeb87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd4509480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4517f70 con 0x55bcd4506d70 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bcd45099c0 con 0x55bcd450f950 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ffeb87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcd4509b80 con 0x55bcd4506450 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ffeb87740 10 client.0 _ll_register_callbacks cb 0x7f1fe8009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39604/0 (socket says 172.21.15.22:39604) 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 -- 172.21.15.22:0/2180340753 learned_addr learned my addr 172.21.15.22:0/2180340753 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9e00640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd4509480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd4508f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 -- 172.21.15.22:0/2180340753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 msgr2=0x55bcd4509480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd4509480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 -- 172.21.15.22:0/2180340753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 msgr2=0x55bcd4508f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd4508f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.239+0000 7f1ff9400640 1 -- 172.21.15.22:0/2180340753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd446ba70 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1ff9400640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8012b20 tx=0x7f1fe800fef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=454d5e201f88e2ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1fdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8017040 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fe8000e40 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fec014040 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1fe8005560 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1ffeb87740 10 client.25906 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.240+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bcd442d6a0 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec014040 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1fe80038f0 con 0x55bcd4506450 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 10 client.25906 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 10 client.25906 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:41:15.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 20 client.25906 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd45861c0 0x55bcd45885b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 4 client.25906 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:15.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcd41f8890 con 0x55bcd45861c0 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.241+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd45861c0 0x55bcd45885b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ff8a00640 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd45861c0 0x55bcd45885b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e66456fb718795a9 server_cookie=c07ec05481122bde in_seq=0 out_seq=0 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1fdaa00640 10 client.25906 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1fdaa00640 1 -- 172.21.15.22:0/2180340753 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcd41f8890 con 0x55bcd45861c0 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1fdaa00640 10 client.25906 handle_command_reply: tid=0 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 client.25906 shutdown 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 msgr2=0x55bcd4517810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8012b20 tx=0x7f1fe800fef0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 shutdown_connections 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55bcd45861c0 0x55bcd45885b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bcd450f950 0x55bcd4509480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bcd4506d70 0x55bcd4508f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 --2- 172.21.15.22:0/2180340753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcd4506450 0x55bcd4517810 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1fdaa00640 0 client.25906 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 >> 172.21.15.22:0/2180340753 conn(0x55bcd4500b80 msgr2=0x55bcd44ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 shutdown_connections 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.242+0000 7f1ffeb87740 1 -- 172.21.15.22:0/2180340753 wait complete. 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffeb87740 20 client.25906 trim_cache size 0 max 16384 2024-03-24T06:41:15.245 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 msgr2=0x7f1fdc0a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 0x7f1fdc0a4890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009fd0 tx=0x7f1fec021c60 comp rx=0 tx=0).stop 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 msgr2=0x7f1ff419a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0037040 tx=0x7f1fe00045b0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 shutdown_connections 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fdc0a23d0 0x7f1fdc0a4890 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ff41097a0 0x7f1ff419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ff4104150 0x7f1ff419a3d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 --2- 172.21.15.22:0/3038005390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1ff4103790 0x7f1ff4199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.243+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 >> 172.21.15.22:0/3038005390 conn(0x7f1ff40ff220 msgr2=0x7f1ff40ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.244+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 shutdown_connections 2024-03-24T06:41:15.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.244+0000 7f1ffa800640 1 -- 172.21.15.22:0/3038005390 wait complete. 2024-03-24T06:41:15.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:41:15.299 INFO:tasks.workunit.client.0.smithi022.stdout:checking for inttypes.h... yes 2024-03-24T06:41:15.410 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdint.h... yes 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 -- 172.21.15.22:0/1042377121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e8073ae0 msgr2=0x7f75e8073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 --2- 172.21.15.22:0/1042377121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e8073ae0 0x7f75e8073ec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009900 tx=0x7f75dc02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 -- 172.21.15.22:0/1042377121 shutdown_connections 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 --2- 172.21.15.22:0/1042377121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e810df40 0x7f75e8110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 --2- 172.21.15.22:0/1042377121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 0x7f75e810da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 --2- 172.21.15.22:0/1042377121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e8073ae0 0x7f75e8073ec0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 -- 172.21.15.22:0/1042377121 >> 172.21.15.22:0/1042377121 conn(0x7f75e8078b10 msgr2=0x7f75e8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 -- 172.21.15.22:0/1042377121 shutdown_connections 2024-03-24T06:41:15.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.506+0000 7f75f0000640 1 -- 172.21.15.22:0/1042377121 wait complete. 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.507+0000 7f75f0000640 1 Processor -- start 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.507+0000 7f75f0000640 1 -- start start 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.507+0000 7f75f0000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 0x7f75e8199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75f0000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 0x7f75e819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75f0000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 0x7f75e819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75f0000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f75e819ee30 con 0x7f75e8074400 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75f0000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f75e819efa0 con 0x7f75e8073ae0 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75f0000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75e819f160 con 0x7f75e810df40 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 0x7f75e8199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 0x7f75e8199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35482/0 (socket says 172.21.15.22:35482) 2024-03-24T06:41:15.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ed600640 1 -- 172.21.15.22:0/3926253393 learned_addr learned my addr 172.21.15.22:0/3926253393 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 0x7f75e819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ecc00640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 0x7f75e819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 -- 172.21.15.22:0/3926253393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 msgr2=0x7f75e8199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 0x7f75e8199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 -- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 msgr2=0x7f75e819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 0x7f75e819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.508+0000 7f75ee000640 1 -- 172.21.15.22:0/3926253393 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc009560 con 0x7f75e810df40 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75ee000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 0x7f75e819e830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f75e8074a30 tx=0x7f75e000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75e6000640 1 -- 172.21.15.22:0/3926253393 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e000cdf0 con 0x7f75e810df40 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75e6000640 1 -- 172.21.15.22:0/3926253393 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75e0005020 con 0x7f75e810df40 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75e6000640 1 -- 172.21.15.22:0/3926253393 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0010690 con 0x7f75e810df40 2024-03-24T06:41:15.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e819f440 con 0x7f75e810df40 2024-03-24T06:41:15.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.509+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e81a6cc0 con 0x7f75e810df40 2024-03-24T06:41:15.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.513+0000 7f75f2c1e740 1 Processor -- start 2024-03-24T06:41:15.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.514+0000 7f75e6000640 1 -- 172.21.15.22:0/3926253393 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7f75e0005190 con 0x7f75e810df40 2024-03-24T06:41:15.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.514+0000 7f75f2c1e740 1 -- start start 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c2d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c38b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5648d8b9b6e0 con 0x5648d8c36780 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648d8b98c10 con 0x5648d8c2dba0 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648d8b943f0 con 0x5648d8c2d280 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75ecc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c2d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35498/0 (socket says 172.21.15.22:35498) 2024-03-24T06:41:15.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75ed600640 1 -- 172.21.15.22:0/2564619026 learned_addr learned my addr 172.21.15.22:0/2564619026 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.515+0000 7f75ecc00640 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c38b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 -- 172.21.15.22:0/2564619026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 msgr2=0x5648d8c2d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c2d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 -- 172.21.15.22:0/2564619026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 msgr2=0x5648d8c38b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c38b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 -- 172.21.15.22:0/2564619026 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d8b55e40 con 0x5648d8c2dba0 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ecc00640 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c38b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75e6000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 0x7f75d00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75ed600640 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc02c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d9257bd0f58b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.516+0000 7f75e6000640 1 -- 172.21.15.22:0/3926253393 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75e0014070 con 0x7f75e810df40 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75da600640 1 -- 172.21.15.22:0/2564619026 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc004880 con 0x5648d8c2dba0 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75da600640 1 -- 172.21.15.22:0/2564619026 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75dc0049e0 con 0x5648d8c2dba0 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75da600640 1 -- 172.21.15.22:0/2564619026 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc005170 con 0x5648d8c2dba0 2024-03-24T06:41:15.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75ecc00640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 0x7f75d00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2564619026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 msgr2=0x5648d8c36240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75ecc00640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 0x7f75d00a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5648d8c28b40 tx=0x7f75d4013040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.517+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc02c750 comp rx=0 tx=0).stop 2024-03-24T06:41:15.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2564619026 shutdown_connections 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c38b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c36240 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2564619026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c2d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2564619026 >> 172.21.15.22:0/2564619026 conn(0x5648d8c279b0 msgr2=0x5648d8c27d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2564619026 shutdown_connections 2024-03-24T06:41:15.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2564619026 wait complete. 2024-03-24T06:41:15.519 INFO:tasks.workunit.client.0.smithi022.stdout:checking for unistd.h... yes 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.518+0000 7f75f2c1e740 1 -- start start 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 Processor -- start 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c33cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c36780 0x5648d8c34230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5648d8b39060 con 0x5648d8c2dba0 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648d8b38d00 con 0x5648d8c2d280 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648d89021c0 con 0x5648d8c36780 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75f2c1e740 10 client.0 _ll_register_callbacks cb 0x5648d88a4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75ed600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35508/0 (socket says 172.21.15.22:35508) 2024-03-24T06:41:15.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75ed600640 1 -- 172.21.15.22:0/4015295760 learned_addr learned my addr 172.21.15.22:0/4015295760 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75ecc00640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c33cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.519+0000 7f75ee000640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c36780 0x5648d8c34230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 -- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c36780 msgr2=0x5648d8c34230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c36780 0x5648d8c34230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 -- 172.21.15.22:0/4015295760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 msgr2=0x5648d8c33cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c33cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 -- 172.21.15.22:0/4015295760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e8111020 con 0x5648d8c2d280 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75ed600640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f75dc02f020 tx=0x7f75dc02ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58fb6851df0f7f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75cea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc004880 con 0x5648d8c2d280 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75dc0050f0 con 0x5648d8c2d280 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.520+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648d88937e0 con 0x5648d8c2d280 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f75dc02cd10 con 0x5648d8c2d280 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 10 client.25883 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648d8b92a70 con 0x5648d8c2d280 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5648d88937e0 con 0x5648d8c2d280 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f75dc03c050 con 0x5648d8c2d280 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 10 client.25883 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 10 client.25883 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 20 client.25883 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8c468f0 0x5648d8c48d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 4 client.25883 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.521+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648d88d0720 con 0x5648d8c468f0 2024-03-24T06:41:15.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.522+0000 7f75ecc00640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8c468f0 0x5648d8c48d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.523+0000 7f75ecc00640 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8c468f0 0x5648d8c48d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85260ab9f47e9be3 server_cookie=76c3bd097e524350 in_seq=0 out_seq=0 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.523+0000 7f75cea00640 10 client.25883 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75cea00640 1 -- 172.21.15.22:0/4015295760 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648d88d0720 con 0x5648d8c468f0 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75cea00640 10 client.25883 handle_command_reply: tid=0 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 client.25883 shutdown 2024-03-24T06:41:15.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 msgr2=0x5648d8c337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f75dc02f020 tx=0x7f75dc02ceb0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 shutdown_connections 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8c468f0 0x5648d8c48d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c36780 0x5648d8c34230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c33cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/4015295760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2d280 0x5648d8c337b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 >> 172.21.15.22:0/4015295760 conn(0x5648d8c279b0 msgr2=0x5648d8c29700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75cea00640 0 client.25883 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 shutdown_connections 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.524+0000 7f75f2c1e740 1 -- 172.21.15.22:0/4015295760 wait complete. 2024-03-24T06:41:15.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.525+0000 7f75f2c1e740 20 client.25883 trim_cache size 0 max 16384 2024-03-24T06:41:15.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 Processor -- start 2024-03-24T06:41:15.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 -- start start 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c34050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 0x5648d8c4e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5648d8c2c680 con 0x5648d8c2dba0 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648d8c289b0 con 0x5648d8c36780 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.547+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648d8c2ccf0 con 0x5648d8c2d280 2024-03-24T06:41:15.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ecc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c34050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39650/0 (socket says 172.21.15.22:39650) 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 -- 172.21.15.22:0/2381711789 learned_addr learned my addr 172.21.15.22:0/2381711789 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ee000640 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 0x5648d8c4e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 -- 172.21.15.22:0/2381711789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 msgr2=0x5648d8c4e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 0x5648d8c4e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 -- 172.21.15.22:0/2381711789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 msgr2=0x5648d8c34050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c34050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 -- 172.21.15.22:0/2381711789 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d8b2eb00 con 0x5648d8c2d280 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ee000640 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 0x5648d8c4e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:15.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.548+0000 7f75ed600640 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f75dc02c2e0 tx=0x7f75dc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a89a28bda0fbc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.549+0000 7f75d9c00640 1 -- 172.21.15.22:0/2381711789 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc005d50 con 0x5648d8c2d280 2024-03-24T06:41:15.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.549+0000 7f75d9c00640 1 -- 172.21.15.22:0/2381711789 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75dc004740 con 0x5648d8c2d280 2024-03-24T06:41:15.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.549+0000 7f75d9c00640 1 -- 172.21.15.22:0/2381711789 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc048a20 con 0x5648d8c2d280 2024-03-24T06:41:15.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2381711789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 msgr2=0x5648d8c35ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f75dc02c2e0 tx=0x7f75dc005fd0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2381711789 shutdown_connections 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c36780 0x5648d8c4e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c2dba0 0x5648d8c34050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/2381711789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c35ee0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2381711789 >> 172.21.15.22:0/2381711789 conn(0x5648d8c279b0 msgr2=0x5648d8c28d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2381711789 shutdown_connections 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- 172.21.15.22:0/2381711789 wait complete. 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.550+0000 7f75f2c1e740 1 -- start start 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 Processor -- start 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c3a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c3a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5648d8c3ee60 con 0x5648d8c36780 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648d8cabc90 con 0x5648d8c2dba0 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648d8cabe20 con 0x5648d8c2d280 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75f2c1e740 10 client.0 _ll_register_callbacks cb 0x7f75d4009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75ed600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75ecc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c3a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75ed600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39652/0 (socket says 172.21.15.22:39652) 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.551+0000 7f75ed600640 1 -- 172.21.15.22:0/411980467 learned_addr learned my addr 172.21.15.22:0/411980467 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ee000640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c3a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 -- 172.21.15.22:0/411980467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 msgr2=0x5648d8c3a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c3a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 -- 172.21.15.22:0/411980467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 msgr2=0x5648d8c3a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c3a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 -- 172.21.15.22:0/411980467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648d8b92a70 con 0x5648d8c2d280 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ecc00640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c3a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ee000640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c3a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75ed600640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75dc060040 tx=0x7f75dc0048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b57e623024b1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75cea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.552+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc0037f0 con 0x5648d8c2d280 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f75dc003950 con 0x5648d8c2d280 2024-03-24T06:41:15.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75dc0049e0 con 0x5648d8c2d280 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f75dc02cda0 con 0x5648d8c2d280 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 10 client.25924 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648d8b546a0 con 0x5648d8c2d280 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75dc0049e0 con 0x5648d8c2d280 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f75dc064070 con 0x5648d8c2d280 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 10 client.25924 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 10 client.25924 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 20 client.25924 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8cad0f0 0x5648d8caf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 4 client.25924 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.553+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648d891f890 con 0x5648d8cad0f0 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75ecc00640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8cad0f0 0x5648d8caf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75ecc00640 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8cad0f0 0x5648d8caf4e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49cff5e3c0f1d092 server_cookie=1a1bb1b8b7f366e9 in_seq=0 out_seq=0 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75cea00640 10 client.25924 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:15.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75cea00640 1 -- 172.21.15.22:0/411980467 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648d891f890 con 0x5648d8cad0f0 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75cea00640 10 client.25924 handle_command_reply: tid=0 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.554+0000 7f75f2c1e740 1 client.25924 shutdown 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 msgr2=0x5648d8c39ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f75dc060040 tx=0x7f75dc0048b0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 shutdown_connections 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5648d8cad0f0 0x5648d8caf4e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5648d8c36780 0x5648d8c3a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5648d8c2dba0 0x5648d8c3a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 --2- 172.21.15.22:0/411980467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648d8c2d280 0x5648d8c39ee0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75cea00640 0 client.25924 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 >> 172.21.15.22:0/411980467 conn(0x5648d8c279b0 msgr2=0x5648d8c29560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 shutdown_connections 2024-03-24T06:41:15.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 1 -- 172.21.15.22:0/411980467 wait complete. 2024-03-24T06:41:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.555+0000 7f75f2c1e740 20 client.25924 trim_cache size 0 max 16384 2024-03-24T06:41:15.559 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:15.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 msgr2=0x7f75d00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 0x7f75d00a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5648d8c28b40 tx=0x7f75d4013040 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 msgr2=0x7f75e819e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 0x7f75e819e830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f75e8074a30 tx=0x7f75e000ef10 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 shutdown_connections 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f75d00a2350 0x7f75d00a4810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e810df40 0x7f75e819e830 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e8074400 0x7f75e819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 --2- 172.21.15.22:0/3926253393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e8073ae0 0x7f75e8199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 >> 172.21.15.22:0/3926253393 conn(0x7f75e8078b10 msgr2=0x7f75e80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.556+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 shutdown_connections 2024-03-24T06:41:15.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.557+0000 7f75f0000640 1 -- 172.21.15.22:0/3926253393 wait complete. 2024-03-24T06:41:15.572 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:15 smithi022 ceph-mon[32003]: pgmap v40: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 16 op/s 2024-03-24T06:41:15.572 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:15 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1666067488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:15.576 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:41:15.648 INFO:tasks.workunit.client.0.smithi022.stdout:checking pthread.h usability... yes 2024-03-24T06:41:15.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:15 smithi143 ceph-mon[36520]: pgmap v40: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 16 op/s 2024-03-24T06:41:15.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:15 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1666067488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:15.695 INFO:tasks.workunit.client.0.smithi022.stdout:checking pthread.h presence... yes 2024-03-24T06:41:15.696 INFO:tasks.workunit.client.0.smithi022.stdout:checking for pthread.h... yes 2024-03-24T06:41:15.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:15 smithi155 ceph-mon[36262]: pgmap v40: 129 pgs: 129 active+clean; 2.8 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 16 op/s 2024-03-24T06:41:15.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:15 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1666067488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:15.817 INFO:tasks.workunit.client.0.smithi022.stdout:checking fcntl.h usability... yes 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 -- 172.21.15.22:0/3214142729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44301038b0 msgr2=0x7f4430103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 --2- 172.21.15.22:0/3214142729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44301038b0 0x7f4430103d30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f441c009900 tx=0x7f441c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 -- 172.21.15.22:0/3214142729 shutdown_connections 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 --2- 172.21.15.22:0/3214142729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4430104270 0x7f4430110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 --2- 172.21.15.22:0/3214142729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44301038b0 0x7f4430103d30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 --2- 172.21.15.22:0/3214142729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 0x7f4430109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 -- 172.21.15.22:0/3214142729 >> 172.21.15.22:0/3214142729 conn(0x7f44300ff3c0 msgr2=0x7f44301017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 -- 172.21.15.22:0/3214142729 shutdown_connections 2024-03-24T06:41:15.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.819+0000 7f4437400640 1 -- 172.21.15.22:0/3214142729 wait complete. 2024-03-24T06:41:15.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.820+0000 7f4437400640 1 Processor -- start 2024-03-24T06:41:15.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.820+0000 7f4437400640 1 -- start start 2024-03-24T06:41:15.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44301038b0 0x7f4430199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 0x7f443019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f443019edf0 con 0x7f44301038b0 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f443019ef60 con 0x7f4430104270 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4437400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f443019f120 con 0x7f44301098c0 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f442fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f442fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35548/0 (socket says 172.21.15.22:35548) 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f442fe00640 1 -- 172.21.15.22:0/1607163583 learned_addr learned my addr 172.21.15.22:0/1607163583 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4435400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 0x7f443019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.821+0000 7f4434a00640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44301038b0 0x7f4430199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 -- 172.21.15.22:0/1607163583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 msgr2=0x7f443019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 0x7f443019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 -- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44301038b0 msgr2=0x7f4430199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44301038b0 0x7f4430199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 -- 172.21.15.22:0/1607163583 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441c009560 con 0x7f4430104270 2024-03-24T06:41:15.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442fe00640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f441c000c00 tx=0x7f441c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442d600640 1 -- 172.21.15.22:0/1607163583 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c004500 con 0x7f4430104270 2024-03-24T06:41:15.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.822+0000 7f442d600640 1 -- 172.21.15.22:0/1607163583 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c0052d0 con 0x7f4430104270 2024-03-24T06:41:15.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.823+0000 7f442d600640 1 -- 172.21.15.22:0/1607163583 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c003e20 con 0x7f4430104270 2024-03-24T06:41:15.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.823+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443019f3a0 con 0x7f4430104270 2024-03-24T06:41:15.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.823+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44301a6c80 con 0x7f4430104270 2024-03-24T06:41:15.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f443a214740 1 Processor -- start 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f442d600640 1 -- 172.21.15.22:0/1607163583 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7f441c0035f0 con 0x7f4430104270 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f443a214740 1 -- start start 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f443a214740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f443a214740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.826+0000 7f443a214740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73beb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f443a214740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55daa73216e0 con 0x55daa73bc780 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f443a214740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55daa731ec10 con 0x55daa73b3280 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f443a214740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55daa731a3f0 con 0x55daa73b3ba0 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4435400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39676/0 (socket says 172.21.15.22:39676) 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 -- 172.21.15.22:0/4019458376 learned_addr learned my addr 172.21.15.22:0/4019458376 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f442fe00640 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73beb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 -- 172.21.15.22:0/4019458376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 msgr2=0x55daa73b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 -- 172.21.15.22:0/4019458376 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 msgr2=0x55daa73beb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73beb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f4434a00640 1 -- 172.21.15.22:0/4019458376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa72dbe40 con 0x55daa73b3ba0 2024-03-24T06:41:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.827+0000 7f442fe00640 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73beb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f4434a00640 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f442400bc90 tx=0x7f44240107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=604525cf365adb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f442d600640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 0x7f44180a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f442d600640 1 -- 172.21.15.22:0/1607163583 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f441c0f0d60 con 0x7f4430104270 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f4421c00640 1 -- 172.21.15.22:0/4019458376 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442401c480 con 0x55daa73b3ba0 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f4435400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 0x7f44180a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f4421c00640 1 -- 172.21.15.22:0/4019458376 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4424015070 con 0x55daa73b3ba0 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.828+0000 7f4421c00640 1 -- 172.21.15.22:0/4019458376 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240119e0 con 0x55daa73b3ba0 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f4435400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 0x7f44180a4890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4428005fd0 tx=0x7f4428009600 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 -- 172.21.15.22:0/4019458376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 msgr2=0x55daa73bc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f442400bc90 tx=0x7f44240107a0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 -- 172.21.15.22:0/4019458376 shutdown_connections 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73beb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73bc240 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 --2- 172.21.15.22:0/4019458376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 -- 172.21.15.22:0/4019458376 >> 172.21.15.22:0/4019458376 conn(0x55daa73ad9b0 msgr2=0x55daa73add90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 -- 172.21.15.22:0/4019458376 shutdown_connections 2024-03-24T06:41:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.829+0000 7f443a214740 1 -- 172.21.15.22:0/4019458376 wait complete. 2024-03-24T06:41:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.830+0000 7f443a214740 1 -- start start 2024-03-24T06:41:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.830+0000 7f443a214740 1 Processor -- start 2024-03-24T06:41:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73b9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 0x55daa73ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55daa72bf060 con 0x55daa73b3ba0 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55daa72bed00 con 0x55daa73b3280 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55daa70881c0 con 0x55daa73bc780 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f443a214740 10 client.0 _ll_register_callbacks cb 0x55daa702abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f442fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73b9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 0x55daa73ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4434a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f442fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73b9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42436/0 (socket says 172.21.15.22:42436) 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f442fe00640 1 -- 172.21.15.22:0/1960491900 learned_addr learned my addr 172.21.15.22:0/1960491900 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4434a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35570/0 (socket says 172.21.15.22:35570) 2024-03-24T06:41:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 -- 172.21.15.22:0/1960491900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 msgr2=0x55daa73b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 -- 172.21.15.22:0/1960491900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 msgr2=0x55daa73b9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73b9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4435400640 1 -- 172.21.15.22:0/1960491900 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442400b8c0 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.831+0000 7f4434a00640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f4435400640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 0x55daa73ba260 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4428011d40 tx=0x7f4428004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1f7002fd6826835 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f4416000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428020420 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55daa70197e0 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4428020580 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4428004b20 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f443a214740 10 client.25936 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.832+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55daa7318a70 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55daa70197e0 con 0x55daa73bc780 2024-03-24T06:41:15.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f442801d5d0 con 0x55daa73bc780 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 10 client.25936 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 10 client.25936 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 20 client.25936 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa73ccc10 0x55daa73cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 4 client.25936 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.833+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daa7056720 con 0x55daa73ccc10 2024-03-24T06:41:15.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.834+0000 7f4434a00640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa73ccc10 0x55daa73cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.834+0000 7f4434a00640 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa73ccc10 0x55daa73cf030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b063c8c60c9b313 server_cookie=37615d8e5505b2cc in_seq=0 out_seq=0 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.834+0000 7f4416000640 10 client.25936 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f4416000640 1 -- 172.21.15.22:0/1960491900 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55daa7056720 con 0x55daa73ccc10 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f4416000640 10 client.25936 handle_command_reply: tid=0 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 client.25936 shutdown 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 msgr2=0x55daa73ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 0x55daa73ba260 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4428011d40 tx=0x7f4428004a40 comp rx=0 tx=0).stop 2024-03-24T06:41:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 shutdown_connections 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa73ccc10 0x55daa73cf030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73bc780 0x55daa73ba260 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73b9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 --2- 172.21.15.22:0/1960491900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 >> 172.21.15.22:0/1960491900 conn(0x55daa73ad9b0 msgr2=0x55daa73af700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f4416000640 0 client.25936 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.835+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 shutdown_connections 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.836+0000 7f443a214740 1 -- 172.21.15.22:0/1960491900 wait complete. 2024-03-24T06:41:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.836+0000 7f443a214740 20 client.25936 trim_cache size 0 max 16384 2024-03-24T06:41:15.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 Processor -- start 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 -- start start 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73bc780 0x55daa73aca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55daa73ad4e0 con 0x55daa73b3ba0 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55daa73b4620 con 0x55daa73bc780 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f443a214740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55daa73ae9b0 con 0x55daa73b3280 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f4434a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.855+0000 7f4434a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39694/0 (socket says 172.21.15.22:39694) 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 -- 172.21.15.22:0/2585999042 learned_addr learned my addr 172.21.15.22:0/2585999042 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f442fe00640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4435400640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73bc780 0x55daa73aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 -- 172.21.15.22:0/2585999042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73bc780 msgr2=0x55daa73aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73bc780 0x55daa73aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 -- 172.21.15.22:0/2585999042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 msgr2=0x55daa73c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 -- 172.21.15.22:0/2585999042 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa72b4b00 con 0x55daa73b3280 2024-03-24T06:41:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f442fe00640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:15.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.856+0000 7f4434a00640 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4424011550 tx=0x7f44240114e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37edf397ee6fee9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.857+0000 7f4421200640 1 -- 172.21.15.22:0/2585999042 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424020070 con 0x55daa73b3280 2024-03-24T06:41:15.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.857+0000 7f4421200640 1 -- 172.21.15.22:0/2585999042 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4424021040 con 0x55daa73b3280 2024-03-24T06:41:15.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.857+0000 7f4421200640 1 -- 172.21.15.22:0/2585999042 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442402b460 con 0x55daa73b3280 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.857+0000 7f443a214740 1 -- 172.21.15.22:0/2585999042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 msgr2=0x55daa73bbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.857+0000 7f443a214740 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4424011550 tx=0x7f44240114e0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 -- 172.21.15.22:0/2585999042 shutdown_connections 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73bc780 0x55daa73aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73b3ba0 0x55daa73c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 --2- 172.21.15.22:0/2585999042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3280 0x55daa73bbf10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 -- 172.21.15.22:0/2585999042 >> 172.21.15.22:0/2585999042 conn(0x55daa73ad9b0 msgr2=0x55daa73add90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 -- 172.21.15.22:0/2585999042 shutdown_connections 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 -- 172.21.15.22:0/2585999042 wait complete. 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 -- start start 2024-03-24T06:41:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.858+0000 7f443a214740 1 Processor -- start 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73c1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55daa73c4ce0 con 0x55daa73bc780 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55daa73c4e50 con 0x55daa73b3280 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55daa73c2090 con 0x55daa73b3ba0 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f443a214740 10 client.0 _ll_register_callbacks cb 0x7f44280119c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f4434a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f4434a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35598/0 (socket says 172.21.15.22:35598) 2024-03-24T06:41:15.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f4434a00640 1 -- 172.21.15.22:0/2227413019 learned_addr learned my addr 172.21.15.22:0/2227413019 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f442fe00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.859+0000 7f4435400640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 -- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 msgr2=0x55daa73c1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73c1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 -- 172.21.15.22:0/2227413019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 msgr2=0x55daa73c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 -- 172.21.15.22:0/2227413019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa72dbe40 con 0x55daa73b3280 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4435400640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4434a00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44240057c0 tx=0x7f442400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=912ba58ccc576236 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4416000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424020050 con 0x55daa73b3280 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4424017070 con 0x55daa73b3280 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.860+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4424015070 con 0x55daa73b3280 2024-03-24T06:41:15.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4424005250 con 0x55daa73b3280 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 10 client.25907 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55daa72da6a0 con 0x55daa73b3280 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4424015070 con 0x55daa73b3280 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f44240037f0 con 0x55daa73b3280 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 10 client.25907 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 10 client.25907 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 20 client.25907 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa7433d80 0x55daa74361a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 4 client.25907 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.861+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55daa70a5890 con 0x55daa7433d80 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f442fe00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa7433d80 0x55daa74361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f442fe00640 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa7433d80 0x55daa74361a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd82a9e024b5cbfa server_cookie=f7d1b6c4732932a3 in_seq=0 out_seq=0 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f4416000640 10 client.25907 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f4416000640 1 -- 172.21.15.22:0/2227413019 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55daa70a5890 con 0x55daa7433d80 2024-03-24T06:41:15.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f4416000640 10 client.25907 handle_command_reply: tid=0 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.862+0000 7f443a214740 1 client.25907 shutdown 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 msgr2=0x55daa73c4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44240057c0 tx=0x7f442400bf60 comp rx=0 tx=0).stop 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 shutdown_connections 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55daa7433d80 0x55daa74361a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55daa73bc780 0x55daa73c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55daa73b3ba0 0x55daa73c1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 --2- 172.21.15.22:0/2227413019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55daa73b3280 0x55daa73c4640 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 >> 172.21.15.22:0/2227413019 conn(0x55daa73ad9b0 msgr2=0x55daa73ba030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f4416000640 0 client.25907 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:15.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 shutdown_connections 2024-03-24T06:41:15.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 1 -- 172.21.15.22:0/2227413019 wait complete. 2024-03-24T06:41:15.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.863+0000 7f443a214740 20 client.25907 trim_cache size 0 max 16384 2024-03-24T06:41:15.869 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 msgr2=0x7f44180a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 0x7f44180a4890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4428005fd0 tx=0x7f4428009600 comp rx=0 tx=0).stop 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 msgr2=0x7f443019a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f441c000c00 tx=0x7f441c004040 comp rx=0 tx=0).stop 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 shutdown_connections 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f44180a23d0 0x7f44180a4890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44301098c0 0x7f443019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4430104270 0x7f443019a4c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 --2- 172.21.15.22:0/1607163583 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44301038b0 0x7f4430199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 >> 172.21.15.22:0/1607163583 conn(0x7f44300ff3c0 msgr2=0x7f44300ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:15.871 INFO:tasks.workunit.client.0.smithi022.stdout:checking fcntl.h presence... yes 2024-03-24T06:41:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 shutdown_connections 2024-03-24T06:41:15.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:15.864+0000 7f4437400640 1 -- 172.21.15.22:0/1607163583 wait complete. 2024-03-24T06:41:15.872 INFO:tasks.workunit.client.0.smithi022.stdout:checking for fcntl.h... yes 2024-03-24T06:41:15.884 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:41:15.990 INFO:tasks.workunit.client.0.smithi022.stdout:checking limits.h usability... yes 2024-03-24T06:41:16.038 INFO:tasks.workunit.client.0.smithi022.stdout:checking limits.h presence... yes 2024-03-24T06:41:16.039 INFO:tasks.workunit.client.0.smithi022.stdout:checking for limits.h... yes 2024-03-24T06:41:16.073 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdint.h... (cached) yes 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 -- 172.21.15.22:0/3090309955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974073ab0 msgr2=0x7fa974073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 --2- 172.21.15.22:0/3090309955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974073ab0 0x7fa974073e90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa964009900 tx=0x7fa96402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 -- 172.21.15.22:0/3090309955 shutdown_connections 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 --2- 172.21.15.22:0/3090309955 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa974112690 0x7fa974114a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 --2- 172.21.15.22:0/3090309955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9740743d0 0x7fa974112150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 --2- 172.21.15.22:0/3090309955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974073ab0 0x7fa974073e90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 -- 172.21.15.22:0/3090309955 >> 172.21.15.22:0/3090309955 conn(0x7fa9740fd170 msgr2=0x7fa9740ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 -- 172.21.15.22:0/3090309955 shutdown_connections 2024-03-24T06:41:16.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.086+0000 7fa97ac00640 1 -- 172.21.15.22:0/3090309955 wait complete. 2024-03-24T06:41:16.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.087+0000 7fa97ac00640 1 Processor -- start 2024-03-24T06:41:16.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.087+0000 7fa97ac00640 1 -- start start 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9740743d0 0x7fa97419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa9741a3270 con 0x7fa9740743d0 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa9741a33e0 con 0x7fa974073ab0 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa97ac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa9741a35a0 con 0x7fa974112690 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa973e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39722/0 (socket says 172.21.15.22:39722) 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 -- 172.21.15.22:0/2449438065 learned_addr learned my addr 172.21.15.22:0/2449438065 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa973e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35616/0 (socket says 172.21.15.22:35616) 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa973400640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9740743d0 0x7fa97419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 -- 172.21.15.22:0/2449438065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 msgr2=0x7fa97419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 -- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9740743d0 msgr2=0x7fa97419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9740743d0 0x7fa97419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.088+0000 7fa978c00640 1 -- 172.21.15.22:0/2449438065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa964009560 con 0x7fa974112690 2024-03-24T06:41:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa973e00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:16.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa978c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00eb40 tx=0x7fa96c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa970c00640 1 -- 172.21.15.22:0/2449438065 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c004b20 con 0x7fa974112690 2024-03-24T06:41:16.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9741a3880 con 0x7fa974112690 2024-03-24T06:41:16.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa970c00640 1 -- 172.21.15.22:0/2449438065 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa96c004c80 con 0x7fa974112690 2024-03-24T06:41:16.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa970c00640 1 -- 172.21.15.22:0/2449438065 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96c0107b0 con 0x7fa974112690 2024-03-24T06:41:16.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.089+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9741ab100 con 0x7fa974112690 2024-03-24T06:41:16.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.093+0000 7fa970c00640 1 -- 172.21.15.22:0/2449438065 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7fa96c005020 con 0x7fa974112690 2024-03-24T06:41:16.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.093+0000 7fa97d9c6740 1 Processor -- start 2024-03-24T06:41:16.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 -- start start 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd79240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd79780 0x55faacd7bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55faaccde6e0 con 0x55faacd70ba0 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55faaccdbc10 con 0x55faacd79780 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55faaccd73f0 con 0x55faacd70280 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa973400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa973400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39728/0 (socket says 172.21.15.22:39728) 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.094+0000 7fa973400640 1 -- 172.21.15.22:0/37102739 learned_addr learned my addr 172.21.15.22:0/37102739 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd79780 0x55faacd7bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973e00640 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd79240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa970c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 0x7fa9580a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 -- 172.21.15.22:0/37102739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd79780 msgr2=0x55faacd7bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd79780 0x55faacd7bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 -- 172.21.15.22:0/37102739 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 msgr2=0x55faacd79240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd79240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 -- 172.21.15.22:0/37102739 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faacc98e40 con 0x55faacd70280 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa970c00640 1 -- 172.21.15.22:0/2449438065 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa96c014070 con 0x7fa974112690 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa978c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 0x7fa9580a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa973400640 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa960009560 tx=0x7fa960005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4af2850bc46957a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.095+0000 7fa969200640 1 -- 172.21.15.22:0/37102739 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960018070 con 0x55faacd70280 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.096+0000 7fa969200640 1 -- 172.21.15.22:0/37102739 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa960004510 con 0x55faacd70280 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.096+0000 7fa969200640 1 -- 172.21.15.22:0/37102739 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96000f580 con 0x55faacd70280 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.096+0000 7fa97d9c6740 1 -- 172.21.15.22:0/37102739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 msgr2=0x55faacd70660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.096+0000 7fa978c00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 0x7fa9580a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0cc030 tx=0x7fa96c005b40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.096+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa960009560 tx=0x7fa960005850 comp rx=0 tx=0).stop 2024-03-24T06:41:16.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 -- 172.21.15.22:0/37102739 shutdown_connections 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd79780 0x55faacd7bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd79240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/37102739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd70660 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 -- 172.21.15.22:0/37102739 >> 172.21.15.22:0/37102739 conn(0x55faacd6a9b0 msgr2=0x55faacd6ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 -- 172.21.15.22:0/37102739 shutdown_connections 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 -- 172.21.15.22:0/37102739 wait complete. 2024-03-24T06:41:16.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.097+0000 7fa97d9c6740 1 -- start start 2024-03-24T06:41:16.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.098+0000 7fa97d9c6740 1 Processor -- start 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.098+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.098+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd76d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd772d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55faacc7c060 con 0x55faacd70ba0 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55faacc7bd00 con 0x55faacd70280 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55faaca451c0 con 0x55faacd79780 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa97d9c6740 10 client.0 _ll_register_callbacks cb 0x55faac9e7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35632/0 (socket says 172.21.15.22:35632) 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 -- 172.21.15.22:0/3173849226 learned_addr learned my addr 172.21.15.22:0/3173849226 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd772d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd76d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 -- 172.21.15.22:0/3173849226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 msgr2=0x55faacd772d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd772d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 -- 172.21.15.22:0/3173849226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 msgr2=0x55faacd76d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd76d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.099+0000 7fa973400640 1 -- 172.21.15.22:0/3173849226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa960013460 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa973400640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa960013420 tx=0x7fa960015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b39808f2b6ab0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa95d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960018070 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa9600040d0 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55faac9d67e0 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa960004290 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa97d9c6740 10 client.25919 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.100+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55faaccd5a70 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55faac9d67e0 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa960004b30 con 0x55faacd70280 2024-03-24T06:41:16.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 10 client.25919 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:16.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 10 client.25919 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:41:16.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 20 client.25919 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:16.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacd89b70 0x55faacd8bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 4 client.25919 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:16.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faaca13720 con 0x55faacd89b70 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.101+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacd89b70 0x55faacd8bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.102+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacd89b70 0x55faacd8bf60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1506273139d60007 server_cookie=4f6f3d3de27fc600 in_seq=0 out_seq=0 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.102+0000 7fa95d600640 10 client.25919 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa95d600640 1 -- 172.21.15.22:0/3173849226 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55faaca13720 con 0x55faacd89b70 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa95d600640 10 client.25919 handle_command_reply: tid=0 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 client.25919 shutdown 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 msgr2=0x55faacd76850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa960013420 tx=0x7fa960015040 comp rx=0 tx=0).stop 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 shutdown_connections 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd772d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd76d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacd89b70 0x55faacd8bf60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173849226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd76850 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa95d600640 0 client.25919 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 >> 172.21.15.22:0/3173849226 conn(0x55faacd6a9b0 msgr2=0x55faacd6c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 shutdown_connections 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.103+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173849226 wait complete. 2024-03-24T06:41:16.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.104+0000 7fa97d9c6740 20 client.25919 trim_cache size 0 max 16384 2024-03-24T06:41:16.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.118+0000 7fa97d9c6740 1 Processor -- start 2024-03-24T06:41:16.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 -- start start 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 0x55faacd770f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55faacd86720 con 0x55faacd79780 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55faacd6f370 con 0x55faacd70ba0 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55faacd6fcf0 con 0x55faacd70280 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa973e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa973e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35638/0 (socket says 172.21.15.22:35638) 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa973e00640 1 -- 172.21.15.22:0/3173006192 learned_addr learned my addr 172.21.15.22:0/3173006192 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa973400640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.119+0000 7fa973400640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 0x55faacd770f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 -- 172.21.15.22:0/3173006192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 msgr2=0x55faacd757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 -- 172.21.15.22:0/3173006192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 msgr2=0x55faacd770f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 0x55faacd770f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 -- 172.21.15.22:0/3173006192 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faacc71b00 con 0x55faacd70ba0 2024-03-24T06:41:16.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973400640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 0x55faacd770f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:16.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa973e00640 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa964004c50 tx=0x7fa964009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1f5abd567f55412 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa95fe00640 1 -- 172.21.15.22:0/3173006192 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96404c070 con 0x55faacd70ba0 2024-03-24T06:41:16.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa95fe00640 1 -- 172.21.15.22:0/3173006192 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa964004200 con 0x55faacd70ba0 2024-03-24T06:41:16.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.120+0000 7fa95fe00640 1 -- 172.21.15.22:0/3173006192 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964045070 con 0x55faacd70ba0 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173006192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 msgr2=0x55faacd85010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa964004c50 tx=0x7fa964009260 comp rx=0 tx=0).stop 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173006192 shutdown_connections 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd79780 0x55faacd770f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70ba0 0x55faacd85010 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/3173006192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd70280 0x55faacd757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173006192 >> 172.21.15.22:0/3173006192 conn(0x55faacd6a9b0 msgr2=0x55faacd69a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173006192 shutdown_connections 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.121+0000 7fa97d9c6740 1 -- 172.21.15.22:0/3173006192 wait complete. 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.122+0000 7fa97d9c6740 1 -- start start 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.122+0000 7fa97d9c6740 1 Processor -- start 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.122+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.122+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd73aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.122+0000 7fa97d9c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd73fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55faacdeee60 con 0x55faacd70ba0 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55faacdeefa0 con 0x55faacd70280 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa97d9c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55faacdef160 con 0x55faacd79780 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa97d9c6740 10 client.0 _ll_register_callbacks cb 0x7fa9600136a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35648/0 (socket says 172.21.15.22:35648) 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd73aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 -- 172.21.15.22:0/1549059240 learned_addr learned my addr 172.21.15.22:0/1549059240 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd73fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 -- 172.21.15.22:0/1549059240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 msgr2=0x55faacd73fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd73fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 -- 172.21.15.22:0/1549059240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 msgr2=0x55faacd73aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd73aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 -- 172.21.15.22:0/1549059240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9741153b0 con 0x55faacd70280 2024-03-24T06:41:16.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa973400640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa960033050 tx=0x7fa960000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4a30cc026ae0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.123+0000 7fa95d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960018040 con 0x55faacd70280 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa9600040d0 con 0x55faacd70280 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa960004510 con 0x55faacd70280 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa960004290 con 0x55faacd70280 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 10 client.25925 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:16.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55faacc976a0 con 0x55faacd70280 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa960004510 con 0x55faacd70280 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa960004b40 con 0x55faacd70280 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 10 client.25925 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 10 client.25925 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 20 client.25925 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacdf04e0 0x55faacdf2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 4 client.25925 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.124+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55faaca62890 con 0x55faacdf04e0 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa973e00640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacdf04e0 0x55faacdf2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa973e00640 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacdf04e0 0x55faacdf2900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4c872a5349f4f81 server_cookie=fed7cd79988aaacf in_seq=0 out_seq=0 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa95d600640 10 client.25925 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:16.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa95d600640 1 -- 172.21.15.22:0/1549059240 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55faaca62890 con 0x55faacdf04e0 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa95d600640 10 client.25925 handle_command_reply: tid=0 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.125+0000 7fa97d9c6740 1 client.25925 shutdown 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 msgr2=0x55faacd73560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa960033050 tx=0x7fa960000bc0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 shutdown_connections 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55faacd79780 0x55faacd73fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55faacd70ba0 0x55faacd73aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55faacdf04e0 0x55faacdf2900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 --2- 172.21.15.22:0/1549059240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55faacd70280 0x55faacd73560 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 >> 172.21.15.22:0/1549059240 conn(0x55faacd6a9b0 msgr2=0x55faacd6bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa95d600640 0 client.25925 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 shutdown_connections 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 1 -- 172.21.15.22:0/1549059240 wait complete. 2024-03-24T06:41:16.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.126+0000 7fa97d9c6740 20 client.25925 trim_cache size 0 max 16384 2024-03-24T06:41:16.130 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:16.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 msgr2=0x7fa9580a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 0x7fa9580a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0cc030 tx=0x7fa96c005b40 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 msgr2=0x7fa9741a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00eb40 tx=0x7fa96c00c2d0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 shutdown_connections 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9580a2350 0x7fa9580a4810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa974112690 0x7fa9741a2c70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9740743d0 0x7fa97419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 --2- 172.21.15.22:0/2449438065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa974073ab0 0x7fa97419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 >> 172.21.15.22:0/2449438065 conn(0x7fa9740fd170 msgr2=0x7fa9740fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 shutdown_connections 2024-03-24T06:41:16.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.127+0000 7fa97ac00640 1 -- 172.21.15.22:0/2449438065 wait complete. 2024-03-24T06:41:16.144 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:41:16.179 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/time.h usability... yes 2024-03-24T06:41:16.221 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/time.h presence... yes 2024-03-24T06:41:16.223 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/time.h... yes 2024-03-24T06:41:16.251 INFO:tasks.workunit.client.0.smithi022.stdout:checking for unistd.h... (cached) yes 2024-03-24T06:41:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 -- 172.21.15.22:0/298596824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d81076b0 msgr2=0x7fb2d8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 --2- 172.21.15.22:0/298596824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d81076b0 0x7fb2d8107a90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009900 tx=0x7fb2c802bc30 comp rx=0 tx=0).stop 2024-03-24T06:41:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 -- 172.21.15.22:0/298596824 shutdown_connections 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 --2- 172.21.15.22:0/298596824 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d8102060 0x7fb2d810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 --2- 172.21.15.22:0/298596824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 --2- 172.21.15.22:0/298596824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d81076b0 0x7fb2d8107a90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 -- 172.21.15.22:0/298596824 >> 172.21.15.22:0/298596824 conn(0x7fb2d8078b10 msgr2=0x7fb2d8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 -- 172.21.15.22:0/298596824 shutdown_connections 2024-03-24T06:41:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.373+0000 7fb2dde00640 1 -- 172.21.15.22:0/298596824 wait complete. 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.375+0000 7fb2dde00640 1 Processor -- start 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.375+0000 7fb2dde00640 1 -- start start 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 0x7fb2d8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81981a0 con 0x7fb2d81076b0 2024-03-24T06:41:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8198310 con 0x7fb2d8102060 2024-03-24T06:41:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2dde00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81984d0 con 0x7fb2d81016a0 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 0x7fb2d8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35654/0 (socket says 172.21.15.22:35654) 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d819d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39784/0 (socket says 172.21.15.22:39784) 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 -- 172.21.15.22:0/1298123234 learned_addr learned my addr 172.21.15.22:0/1298123234 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 0x7fb2d8197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42526/0 (socket says 172.21.15.22:42526) 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 -- 172.21.15.22:0/1298123234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 msgr2=0x7fb2d819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d819d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 -- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 msgr2=0x7fb2d8197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 0x7fb2d8197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.376+0000 7fb2d6000640 1 -- 172.21.15.22:0/1298123234 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c8009560 con 0x7fb2d8102060 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2d6000640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e9f0 tx=0x7fb2c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2dca00640 1 -- 172.21.15.22:0/1298123234 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4009700 con 0x7fb2d8102060 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2dca00640 1 -- 172.21.15.22:0/1298123234 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c400cd80 con 0x7fb2d8102060 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d81987b0 con 0x7fb2d8102060 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2dca00640 1 -- 172.21.15.22:0/1298123234 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4018920 con 0x7fb2d8102060 2024-03-24T06:41:16.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.377+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d81a49f0 con 0x7fb2d8102060 2024-03-24T06:41:16.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.381+0000 7fb2e0a02740 1 Processor -- start 2024-03-24T06:41:16.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.381+0000 7fb2e0a02740 1 -- start start 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.381+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579961396e0 con 0x5579961cb280 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557996136c10 con 0x5579961cbba0 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579961323f0 con 0x5579961d4780 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42538/0 (socket says 172.21.15.22:42538) 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2d7400640 1 -- 172.21.15.22:0/1528412080 learned_addr learned my addr 172.21.15.22:0/1528412080 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2d6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2dca00640 1 -- 172.21.15.22:0/1298123234 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7fb2c4018a80 con 0x7fb2d8102060 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.382+0000 7fb2d6000640 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 -- 172.21.15.22:0/1528412080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 msgr2=0x5579961d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 -- 172.21.15.22:0/1528412080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 msgr2=0x5579961d4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 -- 172.21.15.22:0/1528412080 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579960f3e40 con 0x5579961cb280 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f50 tx=0x7fb2d00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d5153367a2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.383+0000 7fb2c3e00640 1 -- 172.21.15.22:0/1528412080 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0004d20 con 0x5579961cb280 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.384+0000 7fb2c3e00640 1 -- 172.21.15.22:0/1528412080 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d0004e80 con 0x5579961cb280 2024-03-24T06:41:16.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.384+0000 7fb2c3e00640 1 -- 172.21.15.22:0/1528412080 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0005280 con 0x5579961cb280 2024-03-24T06:41:16.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.385+0000 7fb2e0a02740 1 -- 172.21.15.22:0/1528412080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 msgr2=0x5579961cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.385+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f50 tx=0x7fb2d00118d0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.385+0000 7fb2dca00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 0x7fb2bc0a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2dca00640 1 -- 172.21.15.22:0/1298123234 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb2c4014070 con 0x7fb2d8102060 2024-03-24T06:41:16.386 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/vfs.h usability... yes 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 -- 172.21.15.22:0/1528412080 shutdown_connections 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/1528412080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961cb660 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 -- 172.21.15.22:0/1528412080 >> 172.21.15.22:0/1528412080 conn(0x5579961c59b0 msgr2=0x5579961c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 0x7fb2bc0a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 -- 172.21.15.22:0/1528412080 shutdown_connections 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.386+0000 7fb2e0a02740 1 -- 172.21.15.22:0/1528412080 wait complete. 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.387+0000 7fb2d7400640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 0x7fb2bc0a4910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009250 tx=0x7fb2d0000f40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.387+0000 7fb2e0a02740 1 -- start start 2024-03-24T06:41:16.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.387+0000 7fb2e0a02740 1 Processor -- start 2024-03-24T06:41:16.389 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579960d7060 con 0x5579961cb280 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579960d6d00 con 0x5579961cbba0 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557995ea01c0 con 0x5579961d4780 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2e0a02740 10 client.0 _ll_register_callbacks cb 0x557995e42be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39806/0 (socket says 172.21.15.22:39806) 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2d7400640 1 -- 172.21.15.22:0/134346237 learned_addr learned my addr 172.21.15.22:0/134346237 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.388+0000 7fb2d6000640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 -- 172.21.15.22:0/134346237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 msgr2=0x5579961d1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 -- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 msgr2=0x5579961d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 -- 172.21.15.22:0/134346237 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d0009c00 con 0x5579961d4780 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d6000640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2d7400640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f50 tx=0x7fb2d0004620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b41fcf8a586e897 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.389+0000 7fb2c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000f680 con 0x5579961d4780 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557995e6e720 con 0x5579961d4780 2024-03-24T06:41:16.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb2d002a050 con 0x5579961d4780 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2d00059e0 con 0x5579961d4780 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2e0a02740 10 client.25972 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557996130a70 con 0x5579961d4780 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.390+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557995e6e720 con 0x5579961d4780 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb2d0019890 con 0x5579961d4780 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 10 client.25972 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:16.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 10 client.25972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 20 client.25972 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579961e4a60 0x5579961e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 4 client.25972 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557995ebd890 con 0x5579961e4a60 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.391+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579961e4a60 0x5579961e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.392+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579961e4a60 0x5579961e6e50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0b58356f6b14531 server_cookie=f5f2c6b953e22c8a in_seq=0 out_seq=0 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.392+0000 7fb2c0c00640 10 client.25972 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.393+0000 7fb2c0c00640 1 -- 172.21.15.22:0/134346237 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557995ebd890 con 0x5579961e4a60 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.393+0000 7fb2c0c00640 10 client.25972 handle_command_reply: tid=0 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.393+0000 7fb2e0a02740 1 client.25972 shutdown 2024-03-24T06:41:16.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 msgr2=0x5579961d21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009f50 tx=0x7fb2d0004620 comp rx=0 tx=0).stop 2024-03-24T06:41:16.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 shutdown_connections 2024-03-24T06:41:16.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d21a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5579961e4a60 0x5579961e6e50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x5579961d1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/134346237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x5579961d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 >> 172.21.15.22:0/134346237 conn(0x5579961c59b0 msgr2=0x5579961c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2c0c00640 0 client.25972 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 shutdown_connections 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 1 -- 172.21.15.22:0/134346237 wait complete. 2024-03-24T06:41:16.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.394+0000 7fb2e0a02740 20 client.25972 trim_cache size 0 max 16384 2024-03-24T06:41:16.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.410+0000 7fb2e0a02740 1 Processor -- start 2024-03-24T06:41:16.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.410+0000 7fb2e0a02740 1 -- start start 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.410+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 0x5579961e0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.410+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cbba0 0x5579961c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x557995ebd890 con 0x5579961cbba0 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557995e6e720 con 0x5579961cb280 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579961caee0 con 0x5579961d4780 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 0x5579961e0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d6000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cbba0 0x5579961c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39814/0 (socket says 172.21.15.22:39814) 2024-03-24T06:41:16.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 -- 172.21.15.22:0/344549505 learned_addr learned my addr 172.21.15.22:0/344549505 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 -- 172.21.15.22:0/344549505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 msgr2=0x5579961e0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 0x5579961e0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 -- 172.21.15.22:0/344549505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cbba0 msgr2=0x5579961c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cbba0 0x5579961c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d7400640 1 -- 172.21.15.22:0/344549505 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579960ccb00 con 0x5579961d4780 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.411+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 0x5579961e0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.412+0000 7fb2d7400640 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0012fd0 tx=0x7fb2d0005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3acafeefdf2340fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.412+0000 7fb2c3400640 1 -- 172.21.15.22:0/344549505 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00059e0 con 0x5579961d4780 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.412+0000 7fb2c3400640 1 -- 172.21.15.22:0/344549505 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d002db70 con 0x5579961d4780 2024-03-24T06:41:16.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.412+0000 7fb2c3400640 1 -- 172.21.15.22:0/344549505 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d002d320 con 0x5579961d4780 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 -- 172.21.15.22:0/344549505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 msgr2=0x5579961ec720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0012fd0 tx=0x7fb2d0005000 comp rx=0 tx=0).stop 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 -- 172.21.15.22:0/344549505 shutdown_connections 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961ec720 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cbba0 0x5579961c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/344549505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cb280 0x5579961e0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 -- 172.21.15.22:0/344549505 >> 172.21.15.22:0/344549505 conn(0x5579961c59b0 msgr2=0x5579961c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 -- 172.21.15.22:0/344549505 shutdown_connections 2024-03-24T06:41:16.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.413+0000 7fb2e0a02740 1 -- 172.21.15.22:0/344549505 wait complete. 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.414+0000 7fb2e0a02740 1 -- start start 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.414+0000 7fb2e0a02740 1 Processor -- start 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.414+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x557996243130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5579961d7da0 con 0x5579961cb280 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5579961d7f10 con 0x5579961cbba0 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579961d80d0 con 0x5579961d4780 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2e0a02740 10 client.0 _ll_register_callbacks cb 0x7fb2d0009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:16.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x557996243130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35696/0 (socket says 172.21.15.22:35696) 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 -- 172.21.15.22:0/3062323867 learned_addr learned my addr 172.21.15.22:0/3062323867 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d7400640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 -- 172.21.15.22:0/3062323867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 msgr2=0x5579961d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 -- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 msgr2=0x557996243130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x557996243130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6000640 1 -- 172.21.15.22:0/3062323867 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d810ee70 con 0x5579961cbba0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.415+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x557996243130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2d6000640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4097820 tx=0x7fb2c409aaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b52a5225067a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:16.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c409a4f0 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb2c40aee40 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557995e317e0 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2c409a650 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2e0a02740 10 client.25943 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.416+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fb2d002de70 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c40a7080 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb2c4096c80 con 0x5579961cbba0 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 10 client.25943 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 20 client.25943 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55799624b070 0x55799624d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 4 client.25943 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557995ddd700 con 0x55799624b070 2024-03-24T06:41:16.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.417+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55799624b070 0x55799624d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2d6a00640 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55799624b070 0x55799624d490 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87e55da9a77408 server_cookie=9662b26879d57182 in_seq=0 out_seq=0 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2c0c00640 10 client.25943 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2c0c00640 1 -- 172.21.15.22:0/3062323867 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557995ddd700 con 0x55799624b070 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2c0c00640 10 client.25943 handle_command_reply: tid=0 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 client.25943 shutdown 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 msgr2=0x557996243670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4097820 tx=0x7fb2c409aaa0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 shutdown_connections 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579961d4780 0x5579961d77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55799624b070 0x55799624d490 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5579961cbba0 0x557996243670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 --2- 172.21.15.22:0/3062323867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5579961cb280 0x557996243130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2c0c00640 0 client.25943 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:16.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.418+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 >> 172.21.15.22:0/3062323867 conn(0x5579961c59b0 msgr2=0x5579961c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 shutdown_connections 2024-03-24T06:41:16.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2e0a02740 1 -- 172.21.15.22:0/3062323867 wait complete. 2024-03-24T06:41:16.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2e0a02740 20 client.25943 trim_cache size 0 max 16384 2024-03-24T06:41:16.422 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 msgr2=0x7fb2bc0a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 0x7fb2bc0a4910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009250 tx=0x7fb2d0000f40 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 msgr2=0x7fb2d819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.419+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e9f0 tx=0x7fb2c400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 shutdown_connections 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb2bc0a2450 0x7fb2bc0a4910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2d81076b0 0x7fb2d8197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2d8102060 0x7fb2d819de90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 --2- 172.21.15.22:0/1298123234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2d81016a0 0x7fb2d819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 >> 172.21.15.22:0/1298123234 conn(0x7fb2d8078b10 msgr2=0x7fb2d8104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 shutdown_connections 2024-03-24T06:41:16.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:16.420+0000 7fb2dde00640 1 -- 172.21.15.22:0/1298123234 wait complete. 2024-03-24T06:41:16.434 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/vfs.h presence... yes 2024-03-24T06:41:16.435 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/vfs.h... yes 2024-03-24T06:41:16.532 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/limits.h usability... no 2024-03-24T06:41:16.586 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/limits.h presence... no 2024-03-24T06:41:16.588 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/limits.h... no 2024-03-24T06:41:16.671 INFO:tasks.workunit.client.0.smithi022.stdout:checking for an ANSI C-conforming const... yes 2024-03-24T06:41:16.758 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-24T06:41:16.761 INFO:tasks.workunit.client.0.smithi022.stdout:checking for function prototypes... yes 2024-03-24T06:41:16.834 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether setvbuf arguments are reversed... no 2024-03-24T06:41:16.997 INFO:tasks.workunit.client.0.smithi022.stdout:checking for system... yes 2024-03-24T06:41:17.149 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gettimeofday... yes 2024-03-24T06:41:17.319 INFO:tasks.workunit.client.0.smithi022.stdout:checking for mkdir... yes 2024-03-24T06:41:17.442 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:41:17.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:41:17.443 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:41:17.481 INFO:tasks.workunit.client.0.smithi022.stdout:checking for strerror... yes 2024-03-24T06:41:17.626 INFO:tasks.workunit.client.0.smithi022.stdout:checking for open64... yes 2024-03-24T06:41:17.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 -- 172.21.15.22:0/1286331453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 msgr2=0x7f24ec112040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 --2- 172.21.15.22:0/1286331453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec112040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009860 tx=0x7f24d802bc10 comp rx=0 tx=0).stop 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 -- 172.21.15.22:0/1286331453 shutdown_connections 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 --2- 172.21.15.22:0/1286331453 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec112580 0x7f24ec114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 --2- 172.21.15.22:0/1286331453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec112040 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 --2- 172.21.15.22:0/1286331453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec073ab0 0x7f24ec073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.647+0000 7f24f1200640 1 -- 172.21.15.22:0/1286331453 >> 172.21.15.22:0/1286331453 conn(0x7f24ec0fd080 msgr2=0x7f24ec0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.648+0000 7f24f1200640 1 -- 172.21.15.22:0/1286331453 shutdown_connections 2024-03-24T06:41:17.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.648+0000 7f24f1200640 1 -- 172.21.15.22:0/1286331453 wait complete. 2024-03-24T06:41:17.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:17 smithi022 ceph-mon[32003]: pgmap v41: 129 pgs: 129 active+clean; 4.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 316 KiB/s wr, 40 op/s 2024-03-24T06:41:17.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.649+0000 7f24f1200640 1 Processor -- start 2024-03-24T06:41:17.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.649+0000 7f24f1200640 1 -- start start 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.649+0000 7f24f1200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec073ab0 0x7f24ec072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24f1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24f1200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec112580 0x7f24ec06bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24f1200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec06c770 con 0x7f24ec073ab0 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24f1200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec06c8e0 con 0x7f24ec112580 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24f1200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec06caa0 con 0x7f24ec0743d0 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39846/0 (socket says 172.21.15.22:39846) 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec073ab0 0x7f24ec072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 -- 172.21.15.22:0/1512758326 learned_addr learned my addr 172.21.15.22:0/1512758326 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24ebe00640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec112580 0x7f24ec06bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 -- 172.21.15.22:0/1512758326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec112580 msgr2=0x7f24ec06bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec112580 0x7f24ec06bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 -- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec073ab0 msgr2=0x7f24ec072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec073ab0 0x7f24ec072e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.650+0000 7f24eaa00640 1 -- 172.21.15.22:0/1512758326 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e4009900 con 0x7f24ec0743d0 2024-03-24T06:41:17.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f24d802c150 tx=0x7f24d80344c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24e3e00640 1 -- 172.21.15.22:0/1512758326 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d802c5b0 con 0x7f24ec0743d0 2024-03-24T06:41:17.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24e3e00640 1 -- 172.21.15.22:0/1512758326 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d802cbf0 con 0x7f24ec0743d0 2024-03-24T06:41:17.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d8009490 con 0x7f24ec0743d0 2024-03-24T06:41:17.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24e3e00640 1 -- 172.21.15.22:0/1512758326 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8040450 con 0x7f24ec0743d0 2024-03-24T06:41:17.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.651+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ec070b20 con 0x7f24ec0743d0 2024-03-24T06:41:17.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.656+0000 7f24f55b5740 1 Processor -- start 2024-03-24T06:41:17.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.656+0000 7f24f55b5740 1 -- start start 2024-03-24T06:41:17.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.656+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a10920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24f55b5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5582709732d0 con 0x558270a05950 2024-03-24T06:41:17.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24f55b5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558270970800 con 0x558270a05030 2024-03-24T06:41:17.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24f55b5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55827096bfe0 con 0x558270a0e530 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24e3e00640 1 -- 172.21.15.22:0/1512758326 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7f24d8003780 con 0x7f24ec0743d0 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35728/0 (socket says 172.21.15.22:35728) 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 -- 172.21.15.22:0/691117121 learned_addr learned my addr 172.21.15.22:0/691117121 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24eaa00640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a10920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24eb400640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 -- 172.21.15.22:0/691117121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 msgr2=0x558270a10920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a10920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 -- 172.21.15.22:0/691117121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 msgr2=0x558270a0dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24ebe00640 1 -- 172.21.15.22:0/691117121 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55827092de90 con 0x558270a05030 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24eaa00640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a10920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.657+0000 7f24eb400640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.658+0000 7f24ebe00640 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f24e400e580 tx=0x7f24e40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f71bb7fc9d41b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.658+0000 7f24e0c00640 1 -- 172.21.15.22:0/691117121 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e401b070 con 0x558270a05030 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.658+0000 7f24e0c00640 1 -- 172.21.15.22:0/691117121 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4003e50 con 0x558270a05030 2024-03-24T06:41:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.658+0000 7f24e0c00640 1 -- 172.21.15.22:0/691117121 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4013040 con 0x558270a05030 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24e3e00640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 0x7f24d00a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 -- 172.21.15.22:0/691117121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 msgr2=0x558270a05410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f24e400e580 tx=0x7f24e40095c0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24e3e00640 1 -- 172.21.15.22:0/1512758326 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f24d80f0440 con 0x7f24ec0743d0 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24eb400640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 0x7f24d00a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 -- 172.21.15.22:0/691117121 shutdown_connections 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a10920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 --2- 172.21.15.22:0/691117121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05030 0x558270a05410 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 -- 172.21.15.22:0/691117121 >> 172.21.15.22:0/691117121 conn(0x5582709ff760 msgr2=0x5582709ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.659+0000 7f24f55b5740 1 -- 172.21.15.22:0/691117121 shutdown_connections 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.660+0000 7f24f55b5740 1 -- 172.21.15.22:0/691117121 wait complete. 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.660+0000 7f24f55b5740 1 -- start start 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.660+0000 7f24eb400640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 0x7f24d00a4890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x558270a04b00 tx=0x7f24dc005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.660+0000 7f24f55b5740 1 Processor -- start 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558270910fc0 con 0x558270a05950 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558270910c60 con 0x558270a0e530 2024-03-24T06:41:17.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5582706b40c0 con 0x558270a05030 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24f55b5740 10 client.0 _ll_register_callbacks cb 0x558270928e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24eb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24eb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39866/0 (socket says 172.21.15.22:39866) 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24eb400640 1 -- 172.21.15.22:0/283689996 learned_addr learned my addr 172.21.15.22:0/283689996 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24eaa00640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.661+0000 7f24ebe00640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 -- 172.21.15.22:0/283689996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 msgr2=0x558270a0c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 -- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 msgr2=0x558270a0bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 -- 172.21.15.22:0/283689996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e4009d70 con 0x558270a05030 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24ebe00640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:17.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24eb400640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f24dc004310 tx=0x7f24dc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=295df9648bc93248 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24d5600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc017070 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f24dc0044c0 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.662+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558270990e90 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24dc01f640 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 10 client.25990 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55827096a660 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558270990e90 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24dc029a00 con 0x558270a05030 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 10 client.25990 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 10 client.25990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:17.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 20 client.25990 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a1ea60 0x558270a20ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 4 client.25990 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.663+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558270962210 con 0x558270a1ea60 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.664+0000 7f24eaa00640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a1ea60 0x558270a20ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.664+0000 7f24eaa00640 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a1ea60 0x558270a20ec0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c05bb29886eb4e4 server_cookie=c65f9f8646490ab4 in_seq=0 out_seq=0 2024-03-24T06:41:17.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.664+0000 7f24d5600640 10 client.25990 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24d5600640 1 -- 172.21.15.22:0/283689996 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558270962210 con 0x558270a1ea60 2024-03-24T06:41:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24d5600640 10 client.25990 handle_command_reply: tid=0 2024-03-24T06:41:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 client.25990 shutdown 2024-03-24T06:41:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 msgr2=0x558270a0b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f24dc004310 tx=0x7f24dc004730 comp rx=0 tx=0).stop 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 shutdown_connections 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a1ea60 0x558270a20ec0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05950 0x558270a0bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 --2- 172.21.15.22:0/283689996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05030 0x558270a0b590 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 >> 172.21.15.22:0/283689996 conn(0x5582709ff760 msgr2=0x558270a014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24d5600640 0 client.25990 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 shutdown_connections 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.666+0000 7f24f55b5740 1 -- 172.21.15.22:0/283689996 wait complete. 2024-03-24T06:41:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.667+0000 7f24f55b5740 20 client.25990 trim_cache size 0 max 16384 2024-03-24T06:41:17.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:17 smithi143 ceph-mon[36520]: pgmap v41: 129 pgs: 129 active+clean; 4.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 316 KiB/s wr, 40 op/s 2024-03-24T06:41:17.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 Processor -- start 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 -- start start 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a19dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 0x558270a267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558270a04960 con 0x558270a05030 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5582709fef40 con 0x558270a0e530 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.686+0000 7f24f55b5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5582709fab60 con 0x558270a05950 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 0x558270a267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 0x558270a267d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39870/0 (socket says 172.21.15.22:39870) 2024-03-24T06:41:17.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24eaa00640 1 -- 172.21.15.22:0/2689966324 learned_addr learned my addr 172.21.15.22:0/2689966324 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24eb400640 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a19dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 -- 172.21.15.22:0/2689966324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 msgr2=0x558270a267d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 0x558270a267d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 -- 172.21.15.22:0/2689966324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 msgr2=0x558270a19dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a19dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.687+0000 7f24ebe00640 1 -- 172.21.15.22:0/2689966324 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558270906ae0 con 0x558270a0e530 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.688+0000 7f24eb400640 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a19dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.688+0000 7f24ebe00640 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0be30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009c20 tx=0x7f24e400df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c792f073e1adab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.688+0000 7f24d7e00640 1 -- 172.21.15.22:0/2689966324 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4013030 con 0x558270a0e530 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.688+0000 7f24d7e00640 1 -- 172.21.15.22:0/2689966324 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e401c040 con 0x558270a0e530 2024-03-24T06:41:17.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.688+0000 7f24d7e00640 1 -- 172.21.15.22:0/2689966324 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e401b070 con 0x558270a0e530 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 -- 172.21.15.22:0/2689966324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 msgr2=0x558270a0be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0be30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009c20 tx=0x7f24e400df20 comp rx=0 tx=0).stop 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 -- 172.21.15.22:0/2689966324 shutdown_connections 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a0e530 0x558270a0be30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a05950 0x558270a267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 --2- 172.21.15.22:0/2689966324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a19dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.689+0000 7f24f55b5740 1 -- 172.21.15.22:0/2689966324 >> 172.21.15.22:0/2689966324 conn(0x5582709ff760 msgr2=0x558270a01420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.690+0000 7f24f55b5740 1 -- 172.21.15.22:0/2689966324 shutdown_connections 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.690+0000 7f24f55b5740 1 -- 172.21.15.22:0/2689966324 wait complete. 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.690+0000 7f24f55b5740 1 -- start start 2024-03-24T06:41:17.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.690+0000 7f24f55b5740 1 Processor -- start 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a16290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a12810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558270a12ee0 con 0x558270a05030 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558270a13050 con 0x558270a05950 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558270a13210 con 0x558270a0e530 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24f55b5740 10 client.0 _ll_register_callbacks cb 0x7f24dc012bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.691+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35764/0 (socket says 172.21.15.22:35764) 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 -- 172.21.15.22:0/1348943144 learned_addr learned my addr 172.21.15.22:0/1348943144 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24ebe00640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a12810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eb400640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a16290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 -- 172.21.15.22:0/1348943144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 msgr2=0x558270a12810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a12810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 -- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 msgr2=0x558270a16290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a16290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 -- 172.21.15.22:0/1348943144 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55827096a660 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24eaa00640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24d80f2890 tx=0x7f24d8003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac3c8f2ac6f3c02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24d5600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d80bd030 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f24d80bc930 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.692+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4003e50 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24d80c0a80 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 10 client.25961 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55827092de90 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4003e50 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24d80d5250 con 0x558270a05950 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 10 client.25961 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:17.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 20 client.25961 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a85df0 0x558270a881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 4 client.25961 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:41:17.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.693+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55827066b860 con 0x558270a85df0 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.694+0000 7f24eb400640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a85df0 0x558270a881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.694+0000 7f24eb400640 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a85df0 0x558270a881e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f69f3400c1eada server_cookie=6881011038d73210 in_seq=0 out_seq=0 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.694+0000 7f24d5600640 10 client.25961 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.695+0000 7f24d5600640 1 -- 172.21.15.22:0/1348943144 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55827066b860 con 0x558270a85df0 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.695+0000 7f24d5600640 10 client.25961 handle_command_reply: tid=0 2024-03-24T06:41:17.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.695+0000 7f24f55b5740 1 client.25961 shutdown 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 msgr2=0x558270a167d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f24d80f2890 tx=0x7f24d8003b20 comp rx=0 tx=0).stop 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 shutdown_connections 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558270a0e530 0x558270a12810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558270a85df0 0x558270a881e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558270a05950 0x558270a167d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 --2- 172.21.15.22:0/1348943144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558270a05030 0x558270a16290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 >> 172.21.15.22:0/1348943144 conn(0x5582709ff760 msgr2=0x558270a00b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24d5600640 0 client.25961 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 shutdown_connections 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.696+0000 7f24f55b5740 1 -- 172.21.15.22:0/1348943144 wait complete. 2024-03-24T06:41:17.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.697+0000 7f24f55b5740 20 client.25961 trim_cache size 0 max 16384 2024-03-24T06:41:17.699 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:17.699 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:41:17.699 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "62629e3b-3ef1-49be-98b3-eeeb38d2fe04", 2024-03-24T06:41:17.699 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:41:17.699 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 msgr2=0x7f24d00a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 0x7f24d00a4890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x558270a04b00 tx=0x7f24dc005f90 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 msgr2=0x7f24ec073360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f24d802c150 tx=0x7f24d80344c0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 shutdown_connections 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24d00a23d0 0x7f24d00a4890 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24ec112580 0x7f24ec06bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24ec0743d0 0x7f24ec073360 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 --2- 172.21.15.22:0/1512758326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24ec073ab0 0x7f24ec072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 >> 172.21.15.22:0/1512758326 conn(0x7f24ec0fd080 msgr2=0x7f24ec0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 shutdown_connections 2024-03-24T06:41:17.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.698+0000 7f24f1200640 1 -- 172.21.15.22:0/1512758326 wait complete. 2024-03-24T06:41:17.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:41:17.722 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:41:17.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:17 smithi155 ceph-mon[36262]: pgmap v41: 129 pgs: 129 active+clean; 4.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 316 KiB/s wr, 40 op/s 2024-03-24T06:41:17.791 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stat64... yes 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 -- 172.21.15.22:0/3040574213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8102740 msgr2=0x7fd3c8102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3040574213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8102740 0x7fd3c8102b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009900 tx=0x7fd3b802bcc0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 -- 172.21.15.22:0/3040574213 shutdown_connections 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3040574213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c810bb60 0x7fd3c810e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3040574213 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c8103060 0x7fd3c810b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3040574213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8102740 0x7fd3c8102b20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 -- 172.21.15.22:0/3040574213 >> 172.21.15.22:0/3040574213 conn(0x7fd3c80fbf60 msgr2=0x7fd3c80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 -- 172.21.15.22:0/3040574213 shutdown_connections 2024-03-24T06:41:17.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.973+0000 7fd3cca00640 1 -- 172.21.15.22:0/3040574213 wait complete. 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.974+0000 7fd3cca00640 1 Processor -- start 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.974+0000 7fd3cca00640 1 -- start start 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.974+0000 7fd3cca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3cca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 0x7fd3c81008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3cca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c810bb60 0x7fd3c8100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3cca00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c81a5fa0 con 0x7fd3c810bb60 2024-03-24T06:41:17.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3cca00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c81a6110 con 0x7fd3c8102740 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3cca00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c81a62d0 con 0x7fd3c8103060 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 0x7fd3c81008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35778/0 (socket says 172.21.15.22:35778) 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 -- 172.21.15.22:0/3446622403 learned_addr learned my addr 172.21.15.22:0/3446622403 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c810bb60 0x7fd3c8100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 -- 172.21.15.22:0/3446622403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 msgr2=0x7fd3c81008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 0x7fd3c81008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 -- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c810bb60 msgr2=0x7fd3c8100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c810bb60 0x7fd3c8100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c7400640 1 -- 172.21.15.22:0/3446622403 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b8009560 con 0x7fd3c8102740 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.975+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 0x7fd3c81008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:17.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3c7400640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8000c00 tx=0x7fd3b802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3bfe00640 1 -- 172.21.15.22:0/3446622403 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b802dbd0 con 0x7fd3c8102740 2024-03-24T06:41:17.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3bfe00640 1 -- 172.21.15.22:0/3446622403 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b802dd30 con 0x7fd3c8102740 2024-03-24T06:41:17.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c81a6550 con 0x7fd3c8102740 2024-03-24T06:41:17.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3bfe00640 1 -- 172.21.15.22:0/3446622403 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8003b50 con 0x7fd3c8102740 2024-03-24T06:41:17.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.976+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c81a6ab0 con 0x7fd3c8102740 2024-03-24T06:41:17.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 Processor -- start 2024-03-24T06:41:17.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 -- start start 2024-03-24T06:41:17.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fc1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc1f60 0x55d278fc23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fcd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d278f30790 con 0x55d278fc2c40 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d278f2dcc0 con 0x55d278fc1f60 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.980+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d278f294a0 con 0x55d278fcb2d0 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fc1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fcd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fc1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42670/0 (socket says 172.21.15.22:42670) 2024-03-24T06:41:17.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c7e00640 1 -- 172.21.15.22:0/707901588 learned_addr learned my addr 172.21.15.22:0/707901588 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 -- 172.21.15.22:0/707901588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc1f60 msgr2=0x55d278fc23e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc1f60 0x55d278fc23e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 -- 172.21.15.22:0/707901588 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 msgr2=0x55d278fc1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fc1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 -- 172.21.15.22:0/707901588 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d278eeac20 con 0x55d278fcb2d0 2024-03-24T06:41:17.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3bfe00640 1 -- 172.21.15.22:0/3446622403 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136989+0+0 (secure 0 0 0) 0x7fd3b8005460 con 0x7fd3c8102740 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.981+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fcd6c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009f90 tx=0x7fd3b40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41d4b8b8e20518f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.982+0000 7fd3b3e00640 1 -- 172.21.15.22:0/707901588 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4004030 con 0x55d278fcb2d0 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.982+0000 7fd3b3e00640 1 -- 172.21.15.22:0/707901588 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b4005500 con 0x55d278fcb2d0 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.982+0000 7fd3b3e00640 1 -- 172.21.15.22:0/707901588 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4004ca0 con 0x55d278fcb2d0 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.982+0000 7fd3d0e30740 1 -- 172.21.15.22:0/707901588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 msgr2=0x55d278fcd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.982+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fcd6c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009f90 tx=0x7fd3b40118d0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 -- 172.21.15.22:0/707901588 shutdown_connections 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fcd6c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc1f60 0x55d278fc23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/707901588 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fc1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 -- 172.21.15.22:0/707901588 >> 172.21.15.22:0/707901588 conn(0x55d278f88b30 msgr2=0x55d278fbc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 -- 172.21.15.22:0/707901588 shutdown_connections 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 -- 172.21.15.22:0/707901588 wait complete. 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3bfe00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 0x7fd3ac0a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 -- start start 2024-03-24T06:41:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3bfe00640 1 -- 172.21.15.22:0/3446622403 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd3b80f0bb0 con 0x7fd3c8102740 2024-03-24T06:41:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 0x7fd3ac0a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.983+0000 7fd3d0e30740 1 Processor -- start 2024-03-24T06:41:17.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 0x55d278fc8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d278ecdf20 con 0x55d278fcb2d0 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d278ecdbc0 con 0x55d278fc2c40 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d278c971c0 con 0x55d278fc1f60 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3d0e30740 10 client.0 _ll_register_callbacks cb 0x55d278c39be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35788/0 (socket says 172.21.15.22:35788) 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 0x55d278fc8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c6a00640 1 -- 172.21.15.22:0/4006518088 learned_addr learned my addr 172.21.15.22:0/4006518088 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:17.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.984+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 0x7fd3ac0a47a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40047e0 tx=0x7fd3b4000bc0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 -- 172.21.15.22:0/4006518088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 msgr2=0x55d278fc82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 -- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 msgr2=0x55d278fc8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 0x55d278fc8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 -- 172.21.15.22:0/4006518088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b4009bf0 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c7e00640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 0x55d278fc8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400fdb0 tx=0x7fd3b40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4831999e40832e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4005670 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd3b4029040 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.985+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d278f21160 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd3b4019450 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 10 client.25970 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d278f27b20 con 0x55d278fc2c40 2024-03-24T06:41:17.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d278f21160 con 0x55d278fc2c40 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b4024850 con 0x55d278fc2c40 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 10 client.25970 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 10 client.25970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 20 client.25970 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d278fdb6f0 0x55d278fddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 4 client.25970 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.986+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d278c287e0 con 0x55d278fdb6f0 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.987+0000 7fd3c7400640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d278fdb6f0 0x55d278fddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.987+0000 7fd3b0c00640 10 client.25970 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.987+0000 7fd3c7400640 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d278fdb6f0 0x55d278fddb10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd439f004fed0d91 server_cookie=5f7e64a9480d3e0c in_seq=0 out_seq=0 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3b0c00640 1 -- 172.21.15.22:0/4006518088 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d278c287e0 con 0x55d278fdb6f0 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3b0c00640 10 client.25970 handle_command_reply: tid=0 2024-03-24T06:41:17.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 client.25970 shutdown 2024-03-24T06:41:17.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 msgr2=0x55d278fc87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:17.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400fdb0 tx=0x7fd3b40118d0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 shutdown_connections 2024-03-24T06:41:17.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fcb2d0 0x55d278fc8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d278fdb6f0 0x55d278fddb10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fc87f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/4006518088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3b0c00640 0 client.25970 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 >> 172.21.15.22:0/4006518088 conn(0x55d278f88b30 msgr2=0x55d278fbe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 shutdown_connections 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.988+0000 7fd3d0e30740 1 -- 172.21.15.22:0/4006518088 wait complete. 2024-03-24T06:41:17.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:17.989+0000 7fd3d0e30740 20 client.25970 trim_cache size 0 max 16384 2024-03-24T06:41:18.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 Processor -- start 2024-03-24T06:41:18.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 -- start start 2024-03-24T06:41:18.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fd6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fcb2d0 0x55d278fc8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d278fc11e0 con 0x55d278fc2c40 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d278fcdce0 con 0x55d278fcb2d0 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.023+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d278fb85f0 con 0x55d278fc1f60 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fcb2d0 0x55d278fc8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:39918/0 (socket says 172.21.15.22:39918) 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 -- 172.21.15.22:0/922105121 learned_addr learned my addr 172.21.15.22:0/922105121 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fd6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 -- 172.21.15.22:0/922105121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fcb2d0 msgr2=0x55d278fc8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fcb2d0 0x55d278fc8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 -- 172.21.15.22:0/922105121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 msgr2=0x55d278fd6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fd6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.024+0000 7fd3c7400640 1 -- 172.21.15.22:0/922105121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d278ec39a0 con 0x55d278fc1f60 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.025+0000 7fd3c7400640 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80edf30 tx=0x7fd3b80035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd7b387cd388977 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.025+0000 7fd3b3400640 1 -- 172.21.15.22:0/922105121 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80bd130 con 0x55d278fc1f60 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.025+0000 7fd3b3400640 1 -- 172.21.15.22:0/922105121 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b80c1280 con 0x55d278fc1f60 2024-03-24T06:41:18.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.025+0000 7fd3b3400640 1 -- 172.21.15.22:0/922105121 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80c5610 con 0x55d278fc1f60 2024-03-24T06:41:18.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 -- 172.21.15.22:0/922105121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 msgr2=0x55d278fc7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:18.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3b80edf30 tx=0x7fd3b80035f0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 -- 172.21.15.22:0/922105121 shutdown_connections 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fcb2d0 0x55d278fc8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc2c40 0x55d278fd6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/922105121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fc1f60 0x55d278fc7210 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 -- 172.21.15.22:0/922105121 >> 172.21.15.22:0/922105121 conn(0x55d278f88b30 msgr2=0x55d278fbe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 -- 172.21.15.22:0/922105121 shutdown_connections 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.026+0000 7fd3d0e30740 1 -- 172.21.15.22:0/922105121 wait complete. 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.027+0000 7fd3d0e30740 1 -- start start 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.027+0000 7fd3d0e30740 1 Processor -- start 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc1f60 0x55d278fcc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fd17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d278fcd1b0 con 0x55d278fc1f60 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d278fd1d20 con 0x55d278fc2c40 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d278fd1ee0 con 0x55d278fcb2d0 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3d0e30740 10 client.0 _ll_register_callbacks cb 0x7fd3c0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc1f60 0x55d278fcc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35820/0 (socket says 172.21.15.22:35820) 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 -- 172.21.15.22:0/2733464270 learned_addr learned my addr 172.21.15.22:0/2733464270 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c7e00640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fd17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 -- 172.21.15.22:0/2733464270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 msgr2=0x55d278fd17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fd17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 -- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc1f60 msgr2=0x55d278fcc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:18.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc1f60 0x55d278fcc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.028+0000 7fd3c6a00640 1 -- 172.21.15.22:0/2733464270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c81a6ab0 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3c6a00640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40059f0 tx=0x7fd3b4005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6874a043dfbad30c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4024640 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd3b40247a0 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b4005500 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd3b4009310 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3d0e30740 10 client.25976 fetch_fsmap learned FSMap version 29 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d278ee9480 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.029+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b4005500 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b4019450 con 0x55d278fc2c40 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 10 client.25976 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 10 client.25976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:18.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 20 client.25976 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:18.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d27904ad20 0x55d27904d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:18.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 4 client.25976 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:41:18.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d278c65720 con 0x55d27904ad20 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.030+0000 7fd3c7400640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d27904ad20 0x55d27904d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3c7400640 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d27904ad20 0x55d27904d110 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=660606601302a8a5 server_cookie=1dd9745be76716dd in_seq=0 out_seq=0 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3b0c00640 10 client.25976 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3b0c00640 1 -- 172.21.15.22:0/2733464270 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d278c65720 con 0x55d27904ad20 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3b0c00640 10 client.25976 handle_command_reply: tid=0 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 client.25976 shutdown 2024-03-24T06:41:18.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 msgr2=0x55d278fcca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40059f0 tx=0x7fd3b4005f10 comp rx=0 tx=0).stop 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 shutdown_connections 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d278fcb2d0 0x55d278fd17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d27904ad20 0x55d27904d110 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d278fc2c40 0x55d278fcca50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 --2- 172.21.15.22:0/2733464270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d278fc1f60 0x55d278fcc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 >> 172.21.15.22:0/2733464270 conn(0x55d278f88b30 msgr2=0x55d278fbdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3b0c00640 0 client.25976 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 shutdown_connections 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 1 -- 172.21.15.22:0/2733464270 wait complete. 2024-03-24T06:41:18.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.031+0000 7fd3d0e30740 20 client.25976 trim_cache size 0 max 16384 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "62629e3b-3ef1-49be-98b3-eeeb38d2fe04": { 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "62629e3b-3ef1-49be-98b3-eeeb38d2fe04", 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:41:18.036 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 msgr2=0x7fd3ac0a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 0x7fd3ac0a47a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40047e0 tx=0x7fd3b4000bc0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 msgr2=0x7fd3c8100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8000c00 tx=0x7fd3b802c6c0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 shutdown_connections 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd3ac0a22e0 0x7fd3ac0a47a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3c810bb60 0x7fd3c8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3c8103060 0x7fd3c81008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 --2- 172.21.15.22:0/3446622403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c8102740 0x7fd3c8100390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:18.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 >> 172.21.15.22:0/3446622403 conn(0x7fd3c80fbf60 msgr2=0x7fd3c80fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:18.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 shutdown_connections 2024-03-24T06:41:18.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:18.032+0000 7fd3cca00640 1 -- 172.21.15.22:0/3446622403 wait complete. 2024-03-24T06:41:18.054 INFO:tasks.cephfs.filesystem:scrub status for tag:62629e3b-3ef1-49be-98b3-eeeb38d2fe04 - {'path': '/', 'tag': '62629e3b-3ef1-49be-98b3-eeeb38d2fe04', 'options': 'recursive,force'} 2024-03-24T06:41:18.302 INFO:tasks.workunit.client.0.smithi022.stdout:checking for fseeko64... yes 2024-03-24T06:41:18.442 INFO:tasks.workunit.client.0.smithi022.stdout:checking for lrand48_r... yes 2024-03-24T06:41:18.590 INFO:tasks.workunit.client.0.smithi022.stdout:checking for srand48_r... yes 2024-03-24T06:41:18.646 INFO:tasks.workunit.client.0.smithi022.stdout:configure: creating ./config.status 2024-03-24T06:41:18.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:18 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:41:18.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:18 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:18 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:41:18.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:18 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:18 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:41:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:18 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:18 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:41:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:18 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:18 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:41:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:18 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:18 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:41:18.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:18 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:18.944 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: creating Makefile 2024-03-24T06:41:18.982 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: creating config.h 2024-03-24T06:41:19.038 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: executing depfiles commands 2024-03-24T06:41:19.191 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: executing default commands 2024-03-24T06:41:19.314 INFO:tasks.workunit.client.0.smithi022.stderr:+ make 2024-03-24T06:41:19.333 INFO:tasks.workunit.client.0.smithi022.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-24T06:41:19.345 INFO:tasks.workunit.client.0.smithi022.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-24T06:41:19.346 INFO:tasks.workunit.client.0.smithi022.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-24T06:41:19.346 INFO:tasks.workunit.client.0.smithi022.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-24T06:41:19.346 INFO:tasks.workunit.client.0.smithi022.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-24T06:41:19.346 INFO:tasks.workunit.client.0.smithi022.stderr: any GNU archive site. 2024-03-24T06:41:19.352 INFO:tasks.workunit.client.0.smithi022.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-24T06:41:19.357 INFO:tasks.workunit.client.0.smithi022.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-24T06:41:19.359 INFO:tasks.workunit.client.0.smithi022.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-24T06:41:19.359 INFO:tasks.workunit.client.0.smithi022.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-24T06:41:19.359 INFO:tasks.workunit.client.0.smithi022.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-24T06:41:19.360 INFO:tasks.workunit.client.0.smithi022.stderr: Grab them from any GNU archive site. 2024-03-24T06:41:19.488 INFO:tasks.workunit.client.0.smithi022.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-24T06:41:19.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:19 smithi022 ceph-mon[32003]: pgmap v42: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:19.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:19 smithi143 ceph-mon[36520]: pgmap v42: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:19.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:19 smithi155 ceph-mon[36262]: pgmap v42: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:19.946 INFO:tasks.workunit.client.0.smithi022.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-24T06:41:19.947 INFO:tasks.workunit.client.0.smithi022.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-24T06:41:19.947 INFO:tasks.workunit.client.0.smithi022.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-24T06:41:19.947 INFO:tasks.workunit.client.0.smithi022.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-24T06:41:20.142 INFO:tasks.workunit.client.0.smithi022.stdout:/bin/sh ./config.status --recheck 2024-03-24T06:41:20.325 INFO:tasks.workunit.client.0.smithi022.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-24T06:41:20.605 INFO:tasks.workunit.client.0.smithi022.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T06:41:21.625 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether build environment is sane... yes 2024-03-24T06:41:21.642 INFO:tasks.workunit.client.0.smithi022.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-24T06:41:21.643 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gawk... gawk 2024-03-24T06:41:21.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:21 smithi022 ceph-mon[32003]: pgmap v43: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:21.671 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether make sets $(MAKE)... yes 2024-03-24T06:41:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:21 smithi143 ceph-mon[36520]: pgmap v43: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:21.757 INFO:tasks.workunit.client.0.smithi022.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-24T06:41:21.757 INFO:tasks.workunit.client.0.smithi022.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-24T06:41:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:21 smithi155 ceph-mon[36262]: pgmap v43: 129 pgs: 129 active+clean; 4.8 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 326 KiB/s wr, 40 op/s 2024-03-24T06:41:21.779 INFO:tasks.workunit.client.0.smithi022.stdout:checking for style of include used by make... GNU 2024-03-24T06:41:21.788 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc... gcc 2024-03-24T06:41:21.985 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether the C compiler works... yes 2024-03-24T06:41:21.985 INFO:tasks.workunit.client.0.smithi022.stdout:checking for C compiler default output file name... a.out 2024-03-24T06:41:22.073 INFO:tasks.workunit.client.0.smithi022.stdout:checking for suffix of executables... 2024-03-24T06:41:22.155 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are cross compiling... no 2024-03-24T06:41:22.241 INFO:tasks.workunit.client.0.smithi022.stdout:checking for suffix of object files... o 2024-03-24T06:41:22.301 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T06:41:22.370 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether gcc accepts -g... yes 2024-03-24T06:41:22.508 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-24T06:41:22.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:22 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:41:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:22 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:41:22.730 INFO:tasks.workunit.client.0.smithi022.stdout:checking dependency style of gcc... gcc3 2024-03-24T06:41:22.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:22 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:41:22.841 INFO:tasks.workunit.client.0.smithi022.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T06:41:22.970 INFO:tasks.workunit.client.0.smithi022.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-24T06:41:22.975 INFO:tasks.workunit.client.0.smithi022.stdout:checking for egrep... /usr/bin/grep -E 2024-03-24T06:41:23.255 INFO:tasks.workunit.client.0.smithi022.stdout:checking for ANSI C header files... yes 2024-03-24T06:41:23.333 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/types.h... yes 2024-03-24T06:41:23.420 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/stat.h... yes 2024-03-24T06:41:23.514 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdlib.h... yes 2024-03-24T06:41:23.612 INFO:tasks.workunit.client.0.smithi022.stdout:checking for string.h... yes 2024-03-24T06:41:23.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:23 smithi022 ceph-mon[32003]: pgmap v44: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 402 KiB/s wr, 50 op/s 2024-03-24T06:41:23.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:23 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:23.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:23 smithi143 ceph-mon[36520]: pgmap v44: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 402 KiB/s wr, 50 op/s 2024-03-24T06:41:23.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:23 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:23.696 INFO:tasks.workunit.client.0.smithi022.stdout:checking for memory.h... yes 2024-03-24T06:41:23.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:23 smithi155 ceph-mon[36262]: pgmap v44: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 402 KiB/s wr, 50 op/s 2024-03-24T06:41:23.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:23 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:23.788 INFO:tasks.workunit.client.0.smithi022.stdout:checking for strings.h... yes 2024-03-24T06:41:23.887 INFO:tasks.workunit.client.0.smithi022.stdout:checking for inttypes.h... yes 2024-03-24T06:41:23.978 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdint.h... yes 2024-03-24T06:41:24.071 INFO:tasks.workunit.client.0.smithi022.stdout:checking for unistd.h... yes 2024-03-24T06:41:24.188 INFO:tasks.workunit.client.0.smithi022.stdout:checking minix/config.h usability... no 2024-03-24T06:41:24.262 INFO:tasks.workunit.client.0.smithi022.stdout:checking minix/config.h presence... no 2024-03-24T06:41:24.262 INFO:tasks.workunit.client.0.smithi022.stdout:checking for minix/config.h... no 2024-03-24T06:41:24.333 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-24T06:41:24.335 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc... (cached) gcc 2024-03-24T06:41:24.433 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-24T06:41:24.433 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-24T06:41:24.433 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-24T06:41:24.433 INFO:tasks.workunit.client.0.smithi022.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-24T06:41:24.617 INFO:tasks.workunit.client.0.smithi022.stdout:checking for main in -lm... yes 2024-03-24T06:41:24.702 INFO:tasks.workunit.client.0.smithi022.stdout:checking for main in -lpthread... yes 2024-03-24T06:41:24.705 INFO:tasks.workunit.client.0.smithi022.stdout:checking for ANSI C header files... (cached) yes 2024-03-24T06:41:24.784 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-24T06:41:24.959 INFO:tasks.workunit.client.0.smithi022.stdout:checking pthread.h usability... yes 2024-03-24T06:41:25.003 INFO:tasks.workunit.client.0.smithi022.stdout:checking pthread.h presence... yes 2024-03-24T06:41:25.003 INFO:tasks.workunit.client.0.smithi022.stdout:checking for pthread.h... yes 2024-03-24T06:41:25.107 INFO:tasks.workunit.client.0.smithi022.stdout:checking fcntl.h usability... yes 2024-03-24T06:41:25.148 INFO:tasks.workunit.client.0.smithi022.stdout:checking fcntl.h presence... yes 2024-03-24T06:41:25.148 INFO:tasks.workunit.client.0.smithi022.stdout:checking for fcntl.h... yes 2024-03-24T06:41:25.265 INFO:tasks.workunit.client.0.smithi022.stdout:checking limits.h usability... yes 2024-03-24T06:41:25.302 INFO:tasks.workunit.client.0.smithi022.stdout:checking limits.h presence... yes 2024-03-24T06:41:25.302 INFO:tasks.workunit.client.0.smithi022.stdout:checking for limits.h... yes 2024-03-24T06:41:25.332 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stdint.h... (cached) yes 2024-03-24T06:41:25.407 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/time.h usability... yes 2024-03-24T06:41:25.449 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/time.h presence... yes 2024-03-24T06:41:25.449 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/time.h... yes 2024-03-24T06:41:25.478 INFO:tasks.workunit.client.0.smithi022.stdout:checking for unistd.h... (cached) yes 2024-03-24T06:41:25.601 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/vfs.h usability... yes 2024-03-24T06:41:25.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:25 smithi022 ceph-mon[32003]: pgmap v45: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 275 KiB/s wr, 38 op/s 2024-03-24T06:41:25.657 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/vfs.h presence... yes 2024-03-24T06:41:25.658 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/vfs.h... yes 2024-03-24T06:41:25.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:25 smithi143 ceph-mon[36520]: pgmap v45: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 275 KiB/s wr, 38 op/s 2024-03-24T06:41:25.762 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/limits.h usability... no 2024-03-24T06:41:25.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:25 smithi155 ceph-mon[36262]: pgmap v45: 129 pgs: 129 active+clean; 5.5 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 275 KiB/s wr, 38 op/s 2024-03-24T06:41:25.822 INFO:tasks.workunit.client.0.smithi022.stdout:checking sys/limits.h presence... no 2024-03-24T06:41:25.822 INFO:tasks.workunit.client.0.smithi022.stdout:checking for sys/limits.h... no 2024-03-24T06:41:25.881 INFO:tasks.workunit.client.0.smithi022.stdout:checking for an ANSI C-conforming const... yes 2024-03-24T06:41:25.950 INFO:tasks.workunit.client.0.smithi022.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-24T06:41:26.081 INFO:tasks.workunit.client.0.smithi022.stdout:checking for system... yes 2024-03-24T06:41:26.225 INFO:tasks.workunit.client.0.smithi022.stdout:checking for gettimeofday... yes 2024-03-24T06:41:26.564 INFO:tasks.workunit.client.0.smithi022.stdout:checking for mkdir... yes 2024-03-24T06:41:26.685 INFO:tasks.workunit.client.0.smithi022.stdout:checking for strerror... yes 2024-03-24T06:41:26.810 INFO:tasks.workunit.client.0.smithi022.stdout:checking for open64... yes 2024-03-24T06:41:26.925 INFO:tasks.workunit.client.0.smithi022.stdout:checking for stat64... yes 2024-03-24T06:41:27.045 INFO:tasks.workunit.client.0.smithi022.stdout:checking for fseeko64... yes 2024-03-24T06:41:27.170 INFO:tasks.workunit.client.0.smithi022.stdout:checking for lrand48_r... yes 2024-03-24T06:41:27.300 INFO:tasks.workunit.client.0.smithi022.stdout:checking for srand48_r... yes 2024-03-24T06:41:27.356 INFO:tasks.workunit.client.0.smithi022.stdout:configure: creating ./config.status 2024-03-24T06:41:27.517 INFO:tasks.workunit.client.0.smithi022.stdout: /bin/sh ./config.status 2024-03-24T06:41:27.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:27 smithi022 ceph-mon[32003]: pgmap v46: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 531 KiB/s wr, 78 op/s 2024-03-24T06:41:27.661 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: creating Makefile 2024-03-24T06:41:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:27 smithi143 ceph-mon[36520]: pgmap v46: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 531 KiB/s wr, 78 op/s 2024-03-24T06:41:27.696 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: creating config.h 2024-03-24T06:41:27.774 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: executing depfiles commands 2024-03-24T06:41:27.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:27 smithi155 ceph-mon[36262]: pgmap v46: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 531 KiB/s wr, 78 op/s 2024-03-24T06:41:27.816 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: executing default commands 2024-03-24T06:41:27.880 INFO:tasks.workunit.client.0.smithi022.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-24T06:41:28.179 INFO:tasks.workunit.client.0.smithi022.stdout:rm -f stamp-h1 2024-03-24T06:41:28.187 INFO:tasks.workunit.client.0.smithi022.stdout:touch config.h.in 2024-03-24T06:41:28.193 INFO:tasks.workunit.client.0.smithi022.stdout:cd . && /bin/sh ./config.status config.h 2024-03-24T06:41:28.313 INFO:tasks.workunit.client.0.smithi022.stdout:config.status: creating config.h 2024-03-24T06:41:28.356 INFO:tasks.workunit.client.0.smithi022.stdout:make all-am 2024-03-24T06:41:28.369 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-24T06:41:28.369 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-24T06:41:28.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:28 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:28.655 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-24T06:41:28.667 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-24T06:41:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:28 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:28.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:28 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:41:28.840 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-24T06:41:28.851 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-24T06:41:28.989 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-24T06:41:28.997 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-24T06:41:29.136 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-24T06:41:29.143 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-24T06:41:29.327 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-24T06:41:29.337 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-24T06:41:29.432 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-24T06:41:29.439 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-24T06:41:29.524 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-24T06:41:29.530 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-24T06:41:29.616 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-24T06:41:29.625 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-24T06:41:29.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:29 smithi022 ceph-mon[32003]: pgmap v47: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 343 KiB/s wr, 54 op/s 2024-03-24T06:41:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:29 smithi143 ceph-mon[36520]: pgmap v47: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 343 KiB/s wr, 54 op/s 2024-03-24T06:41:29.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:29 smithi155 ceph-mon[36262]: pgmap v47: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 343 KiB/s wr, 54 op/s 2024-03-24T06:41:29.903 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-24T06:41:29.910 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-24T06:41:30.094 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-24T06:41:30.104 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-24T06:41:30.325 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-24T06:41:30.334 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-24T06:41:30.453 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-24T06:41:30.460 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-24T06:41:30.592 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-24T06:41:30.599 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-24T06:41:30.760 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-24T06:41:30.769 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr:In file included from parser.h:22, 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr: from parser.c:25: 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c: In function ‘init_filesys’: 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.826 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-24T06:41:30.827 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.828 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.829 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.830 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.831 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.832 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.833 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.834 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-24T06:41:30.835 INFO:tasks.workunit.client.0.smithi022.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: | ^ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: 36 | container_of(ptr, type, member) 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-24T06:41:30.836 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c: In function ‘get_optstr’: 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:200:14: note: source object declared here 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr: 200 | char temp[BUFSIZE]; 2024-03-24T06:41:30.885 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~ 2024-03-24T06:41:30.893 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c: In function ‘size64_convert’: 2024-03-24T06:41:30.893 INFO:tasks.workunit.client.0.smithi022.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-24T06:41:30.893 INFO:tasks.workunit.client.0.smithi022.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-24T06:41:30.894 INFO:tasks.workunit.client.0.smithi022.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-24T06:41:31.261 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-24T06:41:31.267 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-24T06:41:31.372 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-24T06:41:31.382 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-24T06:41:31.530 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-24T06:41:31.541 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-24T06:41:31.601 INFO:tasks.workunit.client.0.smithi022.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-24T06:41:31.608 INFO:tasks.workunit.client.0.smithi022.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-24T06:41:31.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:31 smithi022 ceph-mon[32003]: pgmap v48: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 333 KiB/s wr, 50 op/s 2024-03-24T06:41:31.664 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-24T06:41:31.665 INFO:tasks.workunit.client.0.smithi022.stderr:+ cd .. 2024-03-24T06:41:31.665 INFO:tasks.workunit.client.0.smithi022.stderr:+ mkdir tmp 2024-03-24T06:41:31.670 INFO:tasks.workunit.client.0.smithi022.stderr:+ cd tmp 2024-03-24T06:41:31.670 INFO:tasks.workunit.client.0.smithi022.stderr:+ for f in $mydir/*.ffsb 2024-03-24T06:41:31.671 INFO:tasks.workunit.client.0.smithi022.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-24T06:41:31.673 INFO:tasks.workunit.client.0.smithi022.stdout:FFSB version 6.0-RC2 started 2024-03-24T06:41:31.673 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:31 smithi143 ceph-mon[36520]: pgmap v48: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 333 KiB/s wr, 50 op/s 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout:benchmark time = 300 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout:ThreadGroup 0 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout:================ 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout: num_threads = 32 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.679 INFO:tasks.workunit.client.0.smithi022.stdout: read_random = off 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: read_size = 0 (0B) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: read_blocksize = 0 (0B) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: read_skip = off 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: read_skipsize = 0 (0B) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: write_random = on 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: write_size = 5242880 (5MB) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: fsync_file = 0 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: write_blocksize = 4096 (4KB) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: wait time = 0 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: op weights 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: read = 0 (0.00%) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: readall = 0 (0.00%) 2024-03-24T06:41:31.680 INFO:tasks.workunit.client.0.smithi022.stdout: write = 1 (100.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: create = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: append = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: delete = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: metaop = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: createdir = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: stat = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: writeall = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: writeall_fsync = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: open_close = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: write_fsync = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: create_fsync = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: append_fsync = 0 (0.00%) 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout:FileSystem . 2024-03-24T06:41:31.681 INFO:tasks.workunit.client.0.smithi022.stdout:========== 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: num_dirs = 0 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: starting files = 128 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: min file size = 104857600 (100MB) 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: max file size = 104857600 (100MB) 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: directio = off 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: alignedio = on 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: bufferedio = off 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: aging is off 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: current utilization = 0.04% 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:41:31.682 INFO:tasks.workunit.client.0.smithi022.stdout:checking existing fs: . 2024-03-24T06:41:31.683 INFO:tasks.workunit.client.0.smithi022.stderr:opendir: No such file or directory 2024-03-24T06:41:31.683 INFO:tasks.workunit.client.0.smithi022.stdout:recreating new fileset 2024-03-24T06:41:31.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:31 smithi155 ceph-mon[36262]: pgmap v48: 129 pgs: 129 active+clean; 7.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 333 KiB/s wr, 50 op/s 2024-03-24T06:41:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:33 smithi022 ceph-mon[32003]: pgmap v49: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 383 KiB/s wr, 61 op/s 2024-03-24T06:41:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:33 smithi022 ceph-mon[32003]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-24T06:41:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:33 smithi143 ceph-mon[36520]: pgmap v49: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 383 KiB/s wr, 61 op/s 2024-03-24T06:41:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:33 smithi143 ceph-mon[36520]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-24T06:41:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:33 smithi155 ceph-mon[36262]: pgmap v49: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 383 KiB/s wr, 61 op/s 2024-03-24T06:41:33.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:33 smithi155 ceph-mon[36262]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-24T06:41:35.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:35 smithi022 ceph-mon[32003]: pgmap v50: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 307 KiB/s wr, 51 op/s 2024-03-24T06:41:35.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:35 smithi143 ceph-mon[36520]: pgmap v50: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 307 KiB/s wr, 51 op/s 2024-03-24T06:41:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:35 smithi155 ceph-mon[36262]: pgmap v50: 129 pgs: 129 active+clean; 8.3 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 307 KiB/s wr, 51 op/s 2024-03-24T06:41:37.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:37 smithi022 ceph-mon[32003]: pgmap v51: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 4.8 MiB/s wr, 82 op/s 2024-03-24T06:41:37.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:37 smithi143 ceph-mon[36520]: pgmap v51: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 4.8 MiB/s wr, 82 op/s 2024-03-24T06:41:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:37 smithi155 ceph-mon[36262]: pgmap v51: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 4.8 MiB/s wr, 82 op/s 2024-03-24T06:41:39.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:39 smithi143 ceph-mon[36520]: pgmap v52: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:39.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:39 smithi155 ceph-mon[36262]: pgmap v52: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:39.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:39 smithi022 ceph-mon[32003]: pgmap v52: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:41 smithi143 ceph-mon[36520]: pgmap v53: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:41.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:41 smithi155 ceph-mon[36262]: pgmap v53: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:41.901 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:41 smithi022 ceph-mon[32003]: pgmap v53: 129 pgs: 129 active+clean; 62 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 4.6 MiB/s wr, 41 op/s 2024-03-24T06:41:43.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:43 smithi155 ceph-mon[36262]: pgmap v54: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 32 MiB/s wr, 71 op/s 2024-03-24T06:41:43.902 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:43 smithi022 ceph-mon[32003]: pgmap v54: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 32 MiB/s wr, 71 op/s 2024-03-24T06:41:43.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:43 smithi143 ceph-mon[36520]: pgmap v54: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 32 MiB/s wr, 71 op/s 2024-03-24T06:41:45.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:45 smithi155 ceph-mon[36262]: pgmap v55: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 32 MiB/s wr, 59 op/s 2024-03-24T06:41:45.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:45 smithi022 ceph-mon[32003]: pgmap v55: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 32 MiB/s wr, 59 op/s 2024-03-24T06:41:45.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:45 smithi143 ceph-mon[36520]: pgmap v55: 129 pgs: 129 active+clean; 395 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 32 MiB/s wr, 59 op/s 2024-03-24T06:41:47.730 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:47 smithi143 ceph-mon[36520]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 144 MiB/s wr, 102 op/s 2024-03-24T06:41:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:47 smithi155 ceph-mon[36262]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 144 MiB/s wr, 102 op/s 2024-03-24T06:41:47.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:47 smithi022 ceph-mon[32003]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 144 MiB/s wr, 102 op/s 2024-03-24T06:41:48.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:41:48.055 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:41:48.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.428+0000 7fa937400640 1 -- 172.21.15.22:0/3931076198 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9300337e0 con 0x7fa938112430 2024-03-24T06:41:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 -- 172.21.15.22:0/3931076198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938112430 msgr2=0x7fa938114820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 --2- 172.21.15.22:0/3931076198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938112430 0x7fa938114820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa930009c10 tx=0x7fa93002bf00 comp rx=0 tx=0).stop 2024-03-24T06:41:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 -- 172.21.15.22:0/3931076198 shutdown_connections 2024-03-24T06:41:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 --2- 172.21.15.22:0/3931076198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938112430 0x7fa938114820 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 --2- 172.21.15.22:0/3931076198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938076560 0x7fa938111ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 --2- 172.21.15.22:0/3931076198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa938076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.429+0000 7fa93e000640 1 -- 172.21.15.22:0/3931076198 >> 172.21.15.22:0/3931076198 conn(0x7fa9380ff220 msgr2=0x7fa938101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.431+0000 7fa93e000640 1 -- 172.21.15.22:0/3931076198 shutdown_connections 2024-03-24T06:41:48.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.431+0000 7fa93e000640 1 -- 172.21.15.22:0/3931076198 wait complete. 2024-03-24T06:41:48.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.432+0000 7fa93e000640 1 Processor -- start 2024-03-24T06:41:48.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.433+0000 7fa93e000640 1 -- start start 2024-03-24T06:41:48.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.433+0000 7fa93e000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa93819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.433+0000 7fa93e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 0x7fa93819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa93819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa93e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 0x7fa9381a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa93e000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa9381a3180 con 0x7fa938075c40 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa93819e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44684/0 (socket says 172.21.15.22:44684) 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa93cc00640 1 -- 172.21.15.22:0/3975760378 learned_addr learned my addr 172.21.15.22:0/3975760378 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.434+0000 7fa937e00640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 0x7fa93819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.435+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa9381a32f0 con 0x7fa938112430 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.435+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa9381a34b0 con 0x7fa938076560 2024-03-24T06:41:48.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.435+0000 7fa93d600640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 0x7fa9381a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 -- 172.21.15.22:0/3975760378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 msgr2=0x7fa9381a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 0x7fa9381a2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 -- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 msgr2=0x7fa93819e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa93819e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 -- 172.21.15.22:0/3975760378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa930009840 con 0x7fa938076560 2024-03-24T06:41:48.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa93d600640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 0x7fa9381a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa937e00640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 0x7fa93819e850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa924009fd0 tx=0x7fa92400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.436+0000 7fa935600640 1 -- 172.21.15.22:0/3975760378 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924011830 con 0x7fa938076560 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.437+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9381a3790 con 0x7fa938076560 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.437+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9381ab010 con 0x7fa938076560 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.437+0000 7fa935600640 1 -- 172.21.15.22:0/3975760378 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa924011e70 con 0x7fa938076560 2024-03-24T06:41:48.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.437+0000 7fa935600640 1 -- 172.21.15.22:0/3975760378 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92400f2e0 con 0x7fa938076560 2024-03-24T06:41:48.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.444+0000 7fa935600640 1 -- 172.21.15.22:0/3975760378 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa92400f440 con 0x7fa938076560 2024-03-24T06:41:48.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.449+0000 7fa9423bf740 1 Processor -- start 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.449+0000 7fa935600640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 0x7fa9200a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.449+0000 7fa935600640 1 -- 172.21.15.22:0/3975760378 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa9240ced70 con 0x7fa938076560 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.449+0000 7fa93d600640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 0x7fa9200a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 -- start start 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9b9c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9b9d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562df9b0b790 con 0x562df9b9cf60 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562df9b08cc0 con 0x562df9ba62d0 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.450+0000 7fa9423bf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562df9b044a0 con 0x562df9b9dc40 2024-03-24T06:41:48.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.451+0000 7fa93d600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.451+0000 7fa93d600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40972/0 (socket says 172.21.15.22:40972) 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.451+0000 7fa93d600640 1 -- 172.21.15.22:0/2176127153 learned_addr learned my addr 172.21.15.22:0/2176127153 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.451+0000 7fa93cc00640 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9b9c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.451+0000 7fa93d600640 1 -- 172.21.15.22:0/2176127153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 msgr2=0x562df9b9c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9b9c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 -- 172.21.15.22:0/2176127153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 msgr2=0x562df9b9d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9b9d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 -- 172.21.15.22:0/2176127153 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df9ac5c20 con 0x562df9ba62d0 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 0x7fa9200a47f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa930006010 tx=0x7fa93002c9c0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93cc00640 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9b9c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.452+0000 7fa93d600640 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa930009c10 tx=0x7fa930049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31577b18077d6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.453+0000 7fa929c00640 1 -- 172.21.15.22:0/2176127153 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930003970 con 0x562df9ba62d0 2024-03-24T06:41:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.453+0000 7fa929c00640 1 -- 172.21.15.22:0/2176127153 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa930004040 con 0x562df9ba62d0 2024-03-24T06:41:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.453+0000 7fa929c00640 1 -- 172.21.15.22:0/2176127153 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930034610 con 0x562df9ba62d0 2024-03-24T06:41:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 -- 172.21.15.22:0/2176127153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 msgr2=0x562df9ba86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa930009c10 tx=0x7fa930049000 comp rx=0 tx=0).stop 2024-03-24T06:41:48.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 -- 172.21.15.22:0/2176127153 shutdown_connections 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9ba86c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9b9d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2176127153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9b9c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 -- 172.21.15.22:0/2176127153 >> 172.21.15.22:0/2176127153 conn(0x562df9b63b30 msgr2=0x562df9b979b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.454+0000 7fa9423bf740 1 -- 172.21.15.22:0/2176127153 shutdown_connections 2024-03-24T06:41:48.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.455+0000 7fa9423bf740 1 -- 172.21.15.22:0/2176127153 wait complete. 2024-03-24T06:41:48.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.455+0000 7fa9423bf740 1 -- start start 2024-03-24T06:41:48.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.456+0000 7fa9423bf740 1 Processor -- start 2024-03-24T06:41:48.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.456+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.456+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9ba38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9ba3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa9423bf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562df9aa8f20 con 0x562df9b9dc40 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa9423bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562df9aa8bc0 con 0x562df9b9cf60 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa9423bf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562df98721c0 con 0x562df9ba62d0 2024-03-24T06:41:48.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa9423bf740 10 client.0 _ll_register_callbacks cb 0x562df9814be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:48.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40986/0 (socket says 172.21.15.22:40986) 2024-03-24T06:41:48.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa93cc00640 1 -- 172.21.15.22:0/576098617 learned_addr learned my addr 172.21.15.22:0/576098617 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.457+0000 7fa937e00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9ba38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93d600640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9ba3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 -- 172.21.15.22:0/576098617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 msgr2=0x562df9ba3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9ba3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 -- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 msgr2=0x562df9ba38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9ba38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 -- 172.21.15.22:0/576098617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9381152b0 con 0x562df9b9cf60 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa93cc00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa92c012860 tx=0x7fa92c012d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cca6e0788d35c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.458+0000 7fa91e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:48.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c017070 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa92c00f920 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562df9afc160 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa92c0122e0 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa9423bf740 10 client.25985 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.459+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562df9b02b20 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.460+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562df9afc160 con 0x562df9b9cf60 2024-03-24T06:41:48.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa92c01c420 con 0x562df9b9cf60 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 10 client.25985 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 20 client.25985 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9bc79d0 0x562df9bc9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 4 client.25985 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562df98037e0 con 0x562df9bc79d0 2024-03-24T06:41:48.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.461+0000 7fa937e00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9bc79d0 0x562df9bc9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.462+0000 7fa937e00640 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9bc79d0 0x562df9bc9df0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45a557e6bdc7aa2a server_cookie=dc2f34da4db48901 in_seq=0 out_seq=0 2024-03-24T06:41:48.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.462+0000 7fa91e000640 10 client.25985 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.463+0000 7fa91e000640 1 -- 172.21.15.22:0/576098617 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562df98037e0 con 0x562df9bc79d0 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.463+0000 7fa91e000640 10 client.25985 handle_command_reply: tid=0 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.463+0000 7fa9423bf740 1 client.25985 shutdown 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 msgr2=0x562df9ba3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa92c012860 tx=0x7fa92c012d20 comp rx=0 tx=0).stop 2024-03-24T06:41:48.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 shutdown_connections 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9ba3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9ba38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9bc79d0 0x562df9bc9df0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 --2- 172.21.15.22:0/576098617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba3380 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 >> 172.21.15.22:0/576098617 conn(0x562df9b63b30 msgr2=0x562df9b99700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa91e000640 0 client.25985 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 shutdown_connections 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 1 -- 172.21.15.22:0/576098617 wait complete. 2024-03-24T06:41:48.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.464+0000 7fa9423bf740 20 client.25985 trim_cache size 0 max 16384 2024-03-24T06:41:48.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.486+0000 7fa9423bf740 1 Processor -- start 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.486+0000 7fa9423bf740 1 -- start start 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9b96bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9bcf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562df9b9c1e0 con 0x562df9b9dc40 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562df9b97f10 con 0x562df9b9cf60 2024-03-24T06:41:48.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa9423bf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562df9b93270 con 0x562df9ba62d0 2024-03-24T06:41:48.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93d600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9bcf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.487+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40994/0 (socket says 172.21.15.22:40994) 2024-03-24T06:41:48.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93cc00640 1 -- 172.21.15.22:0/770501521 learned_addr learned my addr 172.21.15.22:0/770501521 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa937e00640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9b96bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93cc00640 1 -- 172.21.15.22:0/770501521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 msgr2=0x562df9bcf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93cc00640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9bcf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.488+0000 7fa93cc00640 1 -- 172.21.15.22:0/770501521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 msgr2=0x562df9b96bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa93cc00640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9b96bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa93cc00640 1 -- 172.21.15.22:0/770501521 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df9b97250 con 0x562df9b9cf60 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa937e00640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9b96bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa93d600640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9bcf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa93cc00640 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009740 tx=0x7fa92c01df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e8974d66aad893 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa929200640 1 -- 172.21.15.22:0/770501521 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c017040 con 0x562df9b9cf60 2024-03-24T06:41:48.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa929200640 1 -- 172.21.15.22:0/770501521 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa92c005970 con 0x562df9b9cf60 2024-03-24T06:41:48.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.489+0000 7fa929200640 1 -- 172.21.15.22:0/770501521 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c005290 con 0x562df9b9cf60 2024-03-24T06:41:48.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 -- 172.21.15.22:0/770501521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 msgr2=0x562df9ba5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009740 tx=0x7fa92c01df20 comp rx=0 tx=0).stop 2024-03-24T06:41:48.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 -- 172.21.15.22:0/770501521 shutdown_connections 2024-03-24T06:41:48.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9ba62d0 0x562df9bcf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9dc40 0x562df9b96bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 --2- 172.21.15.22:0/770501521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9b9cf60 0x562df9ba5ab0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 -- 172.21.15.22:0/770501521 >> 172.21.15.22:0/770501521 conn(0x562df9b63b30 msgr2=0x562df9b98d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 -- 172.21.15.22:0/770501521 shutdown_connections 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.490+0000 7fa9423bf740 1 -- 172.21.15.22:0/770501521 wait complete. 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.491+0000 7fa9423bf740 1 -- start start 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.491+0000 7fa9423bf740 1 Processor -- start 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.491+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9bbf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9bbc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562df9bbfaa0 con 0x562df9b9cf60 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562df9bbcce0 con 0x562df9ba62d0 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562df9bbce70 con 0x562df9b9dc40 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa9423bf740 10 client.0 _ll_register_callbacks cb 0x7fa92c009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa937e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa937e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59464/0 (socket says 172.21.15.22:59464) 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa937e00640 1 -- 172.21.15.22:0/2585782853 learned_addr learned my addr 172.21.15.22:0/2585782853 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa93d600640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9bbc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.492+0000 7fa93cc00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9bbf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa937e00640 1 -- 172.21.15.22:0/2585782853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 msgr2=0x562df9bbc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa937e00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9bbc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa937e00640 1 -- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 msgr2=0x562df9bbf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa937e00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9bbf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa937e00640 1 -- 172.21.15.22:0/2585782853 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562df9ac5c20 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.493+0000 7fa93d600640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9bbc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa937e00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa924000c00 tx=0x7fa9240bb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=516a9a0395a1a99f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa91e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9240a3c50 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fa9240cc380 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa930004040 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa9240cc540 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa9423bf740 10 client.26032 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.494+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562df9b02b20 con 0x562df9b9dc40 2024-03-24T06:41:48.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.495+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa930004040 con 0x562df9b9dc40 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa92409aac0 con 0x562df9b9dc40 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 10 client.26032 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 10 client.26032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 20 client.26032 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9c2f270 0x562df9c31690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 4 client.26032 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562df9840720 con 0x562df9c2f270 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.497+0000 7fa93cc00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9c2f270 0x562df9c31690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.498+0000 7fa93cc00640 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9c2f270 0x562df9c31690 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a5ad96e3b87d922 server_cookie=1dadd7ed89bf94e5 in_seq=0 out_seq=0 2024-03-24T06:41:48.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.498+0000 7fa91e000640 10 client.26032 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.498+0000 7fa91e000640 1 -- 172.21.15.22:0/2585782853 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562df9840720 con 0x562df9c2f270 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.498+0000 7fa91e000640 10 client.26032 handle_command_reply: tid=0 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.498+0000 7fa9423bf740 1 client.26032 shutdown 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 msgr2=0x562df9bbc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa924000c00 tx=0x7fa9240bb040 comp rx=0 tx=0).stop 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 shutdown_connections 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562df9c2f270 0x562df9c31690 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562df9ba62d0 0x562df9bbc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562df9b9dc40 0x562df9bbc260 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 --2- 172.21.15.22:0/2585782853 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562df9b9cf60 0x562df9bbf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 >> 172.21.15.22:0/2585782853 conn(0x562df9b63b30 msgr2=0x562df9b99550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa91e000640 0 client.26032 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 shutdown_connections 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 1 -- 172.21.15.22:0/2585782853 wait complete. 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.499+0000 7fa9423bf740 20 client.26032 trim_cache size 0 max 16384 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:41:48.502 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.500+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 msgr2=0x7fa9200a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.500+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 0x7fa9200a47f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa930006010 tx=0x7fa93002c9c0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.500+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 msgr2=0x7fa93819e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.500+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 0x7fa93819e850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa924009fd0 tx=0x7fa92400b870 comp rx=0 tx=0).stop 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 shutdown_connections 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9200a2330 0x7fa9200a47f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa938112430 0x7fa9381a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa938076560 0x7fa93819e850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 --2- 172.21.15.22:0/3975760378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa938075c40 0x7fa93819e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 >> 172.21.15.22:0/3975760378 conn(0x7fa9380ff220 msgr2=0x7fa938101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 shutdown_connections 2024-03-24T06:41:48.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.501+0000 7fa93e000640 1 -- 172.21.15.22:0/3975760378 wait complete. 2024-03-24T06:41:48.527 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:41:48.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.819+0000 7f39b1e00640 1 -- 172.21.15.22:0/1466039339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac076560 msgr2=0x7f39ac112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.819+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1466039339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac076560 0x7f39ac112000 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f399c009930 tx=0x7f399c02bc20 comp rx=0 tx=0).stop 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.820+0000 7f39b1e00640 1 -- 172.21.15.22:0/1466039339 shutdown_connections 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.820+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1466039339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac112540 0x7f39ac114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.820+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1466039339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac076560 0x7f39ac112000 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.820+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1466039339 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac075c40 0x7f39ac076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.820+0000 7f39b1e00640 1 -- 172.21.15.22:0/1466039339 >> 172.21.15.22:0/1466039339 conn(0x7f39ac0ff370 msgr2=0x7f39ac101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:48.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.822+0000 7f39b1e00640 1 -- 172.21.15.22:0/1466039339 shutdown_connections 2024-03-24T06:41:48.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.822+0000 7f39b1e00640 1 -- 172.21.15.22:0/1466039339 wait complete. 2024-03-24T06:41:48.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.824+0000 7f39b1e00640 1 Processor -- start 2024-03-24T06:41:48.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.825+0000 7f39b1e00640 1 -- start start 2024-03-24T06:41:48.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.826+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac075c40 0x7f39ac1119a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.826+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.826+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 0x7f39ac10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39aa000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39aa000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41026/0 (socket says 172.21.15.22:41026) 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39aa000640 1 -- 172.21.15.22:0/1158858037 learned_addr learned my addr 172.21.15.22:0/1158858037 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac10b320 con 0x7f39ac112540 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac10b490 con 0x7f39ac076560 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac10b650 con 0x7f39ac075c40 2024-03-24T06:41:48.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.827+0000 7f39aaa00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac075c40 0x7f39ac1119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39ab400640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 0x7f39ac10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39aa000640 1 -- 172.21.15.22:0/1158858037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac075c40 msgr2=0x7f39ac1119a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39aa000640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac075c40 0x7f39ac1119a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39aa000640 1 -- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 msgr2=0x7f39ac10ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:48.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39aa000640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 0x7f39ac10ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:48.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39aa000640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c009560 con 0x7f39ac076560 2024-03-24T06:41:48.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.828+0000 7f39ab400640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 0x7f39ac10ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:48.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39aa000640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f399c000c00 tx=0x7f399c02cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c02dea0 con 0x7f39ac076560 2024-03-24T06:41:48.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f399c004350 con 0x7f39ac076560 2024-03-24T06:41:48.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac10b870 con 0x7f39ac076560 2024-03-24T06:41:48.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ac10f830 con 0x7f39ac076560 2024-03-24T06:41:48.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.829+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c003b50 con 0x7f39ac076560 2024-03-24T06:41:48.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.831+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c005290 con 0x7f39ac076560 2024-03-24T06:41:48.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.835+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f399c003cb0 con 0x7f39ac076560 2024-03-24T06:41:48.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.837+0000 7f39b0a00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 0x7f39900a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:48.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.837+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f399c0f0a90 con 0x7f39ac076560 2024-03-24T06:41:48.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.838+0000 7f39aaa00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 0x7f39900a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:48.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.839+0000 7f39aaa00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 0x7f39900a47f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3998009890 tx=0x7f3998008040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:48.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:48.839+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f399c0b9620 con 0x7f39ac076560 2024-03-24T06:41:49.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.222+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f398c0058f0 con 0x7f39ac076560 2024-03-24T06:41:49.227 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:41:49.227 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:41:27.278670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24544,24565,24532,24601,24584]},"id":1}]} 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 31 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.224+0000 7f39b0a00640 1 -- 172.21.15.22:0/1158858037 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11946 (secure 0 0 0) 0x7f398c0058f0 con 0x7f39ac076560 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 msgr2=0x7f39900a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 0x7f39900a47f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3998009890 tx=0x7f3998008040 comp rx=0 tx=0).stop 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 msgr2=0x7f39ac111ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f399c000c00 tx=0x7f399c02cd70 comp rx=0 tx=0).stop 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 shutdown_connections 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39900a2330 0x7f39900a47f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ac112540 0x7f39ac10ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39ac076560 0x7f39ac111ee0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 --2- 172.21.15.22:0/1158858037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39ac075c40 0x7f39ac1119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.227+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 >> 172.21.15.22:0/1158858037 conn(0x7f39ac0ff370 msgr2=0x7f39ac113b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.228+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 shutdown_connections 2024-03-24T06:41:49.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.228+0000 7f39b1e00640 1 -- 172.21.15.22:0/1158858037 wait complete. 2024-03-24T06:41:49.248 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:41:49.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 -- 172.21.15.22:0/3216395419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 msgr2=0x7fdbf8100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 --2- 172.21.15.22:0/3216395419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8100d70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdbe400d940 tx=0x7fdbe402dbb0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 -- 172.21.15.22:0/3216395419 shutdown_connections 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 --2- 172.21.15.22:0/3216395419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81012b0 0x7fdbf810d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 --2- 172.21.15.22:0/3216395419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8100d70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 --2- 172.21.15.22:0/3216395419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81066e0 0x7fdbf8106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 -- 172.21.15.22:0/3216395419 >> 172.21.15.22:0/3216395419 conn(0x7fdbf80775f0 msgr2=0x7fdbf8077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 -- 172.21.15.22:0/3216395419 shutdown_connections 2024-03-24T06:41:49.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.501+0000 7fdbfec00640 1 -- 172.21.15.22:0/3216395419 wait complete. 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.502+0000 7fdbfec00640 1 Processor -- start 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 -- start start 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81012b0 0x7fdbf819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81066e0 0x7fdbf819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf819ed60 con 0x7fdbf81012b0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf819eed0 con 0x7fdbf81066e0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbfec00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf819f090 con 0x7fdbf81008f0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59500/0 (socket says 172.21.15.22:59500) 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.503+0000 7fdbf7e00640 1 -- 172.21.15.22:0/2795270778 learned_addr learned my addr 172.21.15.22:0/2795270778 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81066e0 0x7fdbf819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7400640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81012b0 0x7fdbf819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7e00640 1 -- 172.21.15.22:0/2795270778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81066e0 msgr2=0x7fdbf819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7e00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81066e0 0x7fdbf819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7e00640 1 -- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81012b0 msgr2=0x7fdbf819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7e00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81012b0 0x7fdbf819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.504+0000 7fdbf7e00640 1 -- 172.21.15.22:0/2795270778 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe400d570 con 0x7fdbf81008f0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbf7e00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800d950 tx=0x7fdbe800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbf4c00640 1 -- 172.21.15.22:0/2795270778 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8009c30 con 0x7fdbf81008f0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbf4c00640 1 -- 172.21.15.22:0/2795270778 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe8009d90 con 0x7fdbf81008f0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf819f370 con 0x7fdbf81008f0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbf4c00640 1 -- 172.21.15.22:0/2795270778 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe800f7e0 con 0x7fdbf81008f0 2024-03-24T06:41:49.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.505+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf81a6bf0 con 0x7fdbf81008f0 2024-03-24T06:41:49.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.510+0000 7fdc01a15740 1 Processor -- start 2024-03-24T06:41:49.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.510+0000 7fdc01a15740 1 -- start start 2024-03-24T06:41:49.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.510+0000 7fdbf4c00640 1 -- 172.21.15.22:0/2795270778 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fdbe8010ad0 con 0x7fdbf81008f0 2024-03-24T06:41:49.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28c9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5640e282e6e0 con 0x5640e28c0280 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5640e282bc10 con 0x5640e28c0ba0 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdc01a15740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5640e28273f0 con 0x5640e28c9780 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdbf7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdbf7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59514/0 (socket says 172.21.15.22:59514) 2024-03-24T06:41:49.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.511+0000 7fdbf7400640 1 -- 172.21.15.22:0/1715036455 learned_addr learned my addr 172.21.15.22:0/1715036455 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.512+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.512+0000 7fdbf7400640 1 -- 172.21.15.22:0/1715036455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 msgr2=0x5640e28c9240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.512+0000 7fdbf7400640 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28c9240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.512+0000 7fdbf7400640 1 -- 172.21.15.22:0/1715036455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 msgr2=0x5640e28c0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.512+0000 7fdbf4c00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 0x7fdbdc0a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbf7400640 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbf7400640 1 -- 172.21.15.22:0/1715036455 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e27e8e40 con 0x5640e28c9780 2024-03-24T06:41:49.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbf4c00640 1 -- 172.21.15.22:0/2795270778 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbe80ce120 con 0x7fdbf81008f0 2024-03-24T06:41:49.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:49.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbf7e00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 0x7fdbdc0a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.513+0000 7fdbf7400640 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009230 tx=0x7fdbe4005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ee6807b6283d97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.514+0000 7fdbed200640 1 -- 172.21.15.22:0/1715036455 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe400d290 con 0x5640e28c9780 2024-03-24T06:41:49.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.514+0000 7fdbed200640 1 -- 172.21.15.22:0/1715036455 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe400d3f0 con 0x5640e28c9780 2024-03-24T06:41:49.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.514+0000 7fdbed200640 1 -- 172.21.15.22:0/1715036455 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4004fe0 con 0x5640e28c9780 2024-03-24T06:41:49.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdbf7e00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 0x7fdbdc0a47f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5640e28bfd50 tx=0x7fdbe80962d0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 -- 172.21.15.22:0/1715036455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 msgr2=0x5640e28cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009230 tx=0x7fdbe4005e60 comp rx=0 tx=0).stop 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 -- 172.21.15.22:0/1715036455 shutdown_connections 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28cbb70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 --2- 172.21.15.22:0/1715036455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.515+0000 7fdc01a15740 1 -- 172.21.15.22:0/1715036455 >> 172.21.15.22:0/1715036455 conn(0x5640e28ba9b0 msgr2=0x5640e28bad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.516+0000 7fdc01a15740 1 -- 172.21.15.22:0/1715036455 shutdown_connections 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.516+0000 7fdc01a15740 1 -- 172.21.15.22:0/1715036455 wait complete. 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.516+0000 7fdc01a15740 1 -- start start 2024-03-24T06:41:49.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.517+0000 7fdc01a15740 1 Processor -- start 2024-03-24T06:41:49.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0ba0 0x5640e28c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c9780 0x5640e28c7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5640e27cc060 con 0x5640e28c0280 2024-03-24T06:41:49.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5640e27cbd00 con 0x5640e28c9780 2024-03-24T06:41:49.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5640e25951c0 con 0x5640e28c0ba0 2024-03-24T06:41:49.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdc01a15740 10 client.0 _ll_register_callbacks cb 0x5640e2537be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:49.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.518+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.519+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44774/0 (socket says 172.21.15.22:44774) 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.519+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3436938883 learned_addr learned my addr 172.21.15.22:0/3436938883 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.519+0000 7fdbf7400640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0ba0 0x5640e28c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c9780 0x5640e28c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0ba0 msgr2=0x5640e28c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0ba0 0x5640e28c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3436938883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c9780 msgr2=0x5640e28c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c9780 0x5640e28c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.520+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3436938883 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe403a040 con 0x5640e28c0280 2024-03-24T06:41:49.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.521+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0015fd0 tx=0x7fdbf0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cfae592a413de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.521+0000 7fdbe1600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:41:49.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.521+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf001b070 con 0x5640e28c0280 2024-03-24T06:41:49.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.522+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbf0003970 con 0x5640e28c0280 2024-03-24T06:41:49.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.522+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5640e2563720 con 0x5640e28c0280 2024-03-24T06:41:49.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.522+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbf00152e0 con 0x5640e28c0280 2024-03-24T06:41:49.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.522+0000 7fdc01a15740 10 client.15819 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:49.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.522+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640e2825a70 con 0x5640e28c0280 2024-03-24T06:41:49.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.523+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0024030 con 0x5640e28c0280 2024-03-24T06:41:49.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.523+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fdbf000f640 con 0x5640e28c0280 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.523+0000 7fdc01a15740 10 client.15819 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdc01a15740 10 client.15819 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdc01a15740 20 client.15819 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e28f33f0 0x5640e28f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdc01a15740 4 client.15819 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:49.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640e25b2890 con 0x5640e28f33f0 2024-03-24T06:41:49.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.524+0000 7fdbf7400640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e28f33f0 0x5640e28f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.525+0000 7fdbf7400640 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e28f33f0 0x5640e28f5830 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a0d930d7a2dcd5f server_cookie=b3aa4edf1599aef1 in_seq=0 out_seq=0 2024-03-24T06:41:49.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.525+0000 7fdbe1600640 10 client.15819 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:49.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.526+0000 7fdbe1600640 1 -- 172.21.15.22:0/3436938883 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640e25b2890 con 0x5640e28f33f0 2024-03-24T06:41:49.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.526+0000 7fdbe1600640 10 client.15819 handle_command_reply: tid=0 2024-03-24T06:41:49.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.526+0000 7fdc01a15740 1 client.15819 shutdown 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.526+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 msgr2=0x5640e28c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.526+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0015fd0 tx=0x7fdbf0009f90 comp rx=0 tx=0).stop 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 shutdown_connections 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e28f33f0 0x5640e28f5830 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c9780 0x5640e28c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0ba0 0x5640e28c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3436938883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28c67f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 >> 172.21.15.22:0/3436938883 conn(0x5640e28ba9b0 msgr2=0x5640e28bc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdbe1600640 0 client.15819 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 shutdown_connections 2024-03-24T06:41:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.527+0000 7fdc01a15740 1 -- 172.21.15.22:0/3436938883 wait complete. 2024-03-24T06:41:49.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.528+0000 7fdc01a15740 20 client.15819 trim_cache size 0 max 16384 2024-03-24T06:41:49.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.545+0000 7fdc01a15740 1 Processor -- start 2024-03-24T06:41:49.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.545+0000 7fdc01a15740 1 -- start start 2024-03-24T06:41:49.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.545+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.545+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.545+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c9780 0x5640e28c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdc01a15740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5640e28f1c00 con 0x5640e28c9780 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdc01a15740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5640e28bb9b0 con 0x5640e28c0ba0 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdc01a15740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5640e28ba810 con 0x5640e28c0280 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbf7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbf7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28e5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41072/0 (socket says 172.21.15.22:41072) 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59536/0 (socket says 172.21.15.22:59536) 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbf7400640 1 -- 172.21.15.22:0/3264649046 learned_addr learned my addr 172.21.15.22:0/3264649046 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c9780 0x5640e28c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3264649046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 msgr2=0x5640e28e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28e5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3264649046 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c9780 msgr2=0x5640e28c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c9780 0x5640e28c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.546+0000 7fdbfcc00640 1 -- 172.21.15.22:0/3264649046 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640e27c1b00 con 0x5640e28c0280 2024-03-24T06:41:49.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.547+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0010010 tx=0x7fdbf0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75ecd291c6375569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.547+0000 7fdbe3e00640 1 -- 172.21.15.22:0/3264649046 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf001b070 con 0x5640e28c0280 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.547+0000 7fdbe3e00640 1 -- 172.21.15.22:0/3264649046 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0024070 con 0x5640e28c0280 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.547+0000 7fdbe3e00640 1 -- 172.21.15.22:0/3264649046 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0003bd0 con 0x5640e28c0280 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 -- 172.21.15.22:0/3264649046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 msgr2=0x5640e28c0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0010010 tx=0x7fdbf0000f40 comp rx=0 tx=0).stop 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 -- 172.21.15.22:0/3264649046 shutdown_connections 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c9780 0x5640e28c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 --2- 172.21.15.22:0/3264649046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c0280 0x5640e28c0f80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 -- 172.21.15.22:0/3264649046 >> 172.21.15.22:0/3264649046 conn(0x5640e28ba9b0 msgr2=0x5640e28bbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 -- 172.21.15.22:0/3264649046 shutdown_connections 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.548+0000 7fdc01a15740 1 -- 172.21.15.22:0/3264649046 wait complete. 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.549+0000 7fdc01a15740 1 -- start start 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.549+0000 7fdc01a15740 1 Processor -- start 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.549+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28dded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.549+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28de410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdc01a15740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdc01a15740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5640e2958770 con 0x5640e28c0280 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdc01a15740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5640e29588b0 con 0x5640e28c0ba0 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdc01a15740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5640e29589f0 con 0x5640e28c9780 2024-03-24T06:41:49.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdc01a15740 10 client.0 _ll_register_callbacks cb 0x7fdbf0009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59540/0 (socket says 172.21.15.22:59540) 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdbfcc00640 1 -- 172.21.15.22:0/246415050 learned_addr learned my addr 172.21.15.22:0/246415050 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28dded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.550+0000 7fdbf7400640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 -- 172.21.15.22:0/246415050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 msgr2=0x5640e28de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28de410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 -- 172.21.15.22:0/246415050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 msgr2=0x5640e28dded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28dded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 -- 172.21.15.22:0/246415050 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640e2825a70 con 0x5640e28c9780 2024-03-24T06:41:49.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbfcc00640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000fd90 tx=0x7fdbf000fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3a417b7a5952b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbe1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf001b070 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.551+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5640e25267e0 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdbf000fe30 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdbf000f860 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdc01a15740 10 client.26056 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5640e27e76a0 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf003d020 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.552+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fdbf002a460 con 0x5640e28c9780 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 10 client.26056 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 10 client.26056 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:41:49.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 20 client.26056 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e2959e00 0x5640e295c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 4 client.26056 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640e24d2700 con 0x5640e2959e00 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.553+0000 7fdbf7400640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e2959e00 0x5640e295c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdbf7400640 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e2959e00 0x5640e295c1f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=59700b5fb036cc17 server_cookie=c463edaa67840b13 in_seq=0 out_seq=0 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdbe1600640 10 client.26056 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdbe1600640 1 -- 172.21.15.22:0/246415050 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640e24d2700 con 0x5640e2959e00 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdbe1600640 10 client.26056 handle_command_reply: tid=0 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 client.26056 shutdown 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 msgr2=0x5640e28de950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000fd90 tx=0x7fdbf000fc90 comp rx=0 tx=0).stop 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 shutdown_connections 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5640e2959e00 0x5640e295c1f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5640e28c9780 0x5640e28de950 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5640e28c0ba0 0x5640e28de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 --2- 172.21.15.22:0/246415050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5640e28c0280 0x5640e28dded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 >> 172.21.15.22:0/246415050 conn(0x5640e28ba9b0 msgr2=0x5640e28b9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdbe1600640 0 client.26056 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.554+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 shutdown_connections 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.555+0000 7fdc01a15740 1 -- 172.21.15.22:0/246415050 wait complete. 2024-03-24T06:41:49.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.555+0000 7fdc01a15740 20 client.26056 trim_cache size 0 max 16384 2024-03-24T06:41:49.559 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 msgr2=0x7fdbdc0a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 0x7fdbdc0a47f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5640e28bfd50 tx=0x7fdbe80962d0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 msgr2=0x7fdbf8199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800d950 tx=0x7fdbe800de10 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 shutdown_connections 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdbdc0a2330 0x7fdbdc0a47f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbf81066e0 0x7fdbf819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbf81012b0 0x7fdbf819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 --2- 172.21.15.22:0/2795270778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf81008f0 0x7fdbf8199ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 >> 172.21.15.22:0/2795270778 conn(0x7fdbf80775f0 msgr2=0x7fdbf80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 shutdown_connections 2024-03-24T06:41:49.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.556+0000 7fdbfec00640 1 -- 172.21.15.22:0/2795270778 wait complete. 2024-03-24T06:41:49.581 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:49 smithi022 ceph-mon[32003]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:49.581 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:49 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1158858037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:49.585 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:41:49.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:49 smithi155 ceph-mon[36262]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:49.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:49 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1158858037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 -- 172.21.15.22:0/3013335878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f976410bb60 msgr2=0x7f976410e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 --2- 172.21.15.22:0/3013335878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f976410bb60 0x7f976410e000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f975c009ca0 tx=0x7f975c02c180 comp rx=0 tx=0).stop 2024-03-24T06:41:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 -- 172.21.15.22:0/3013335878 shutdown_connections 2024-03-24T06:41:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 --2- 172.21.15.22:0/3013335878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f976410bb60 0x7f976410e000 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 --2- 172.21.15.22:0/3013335878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f976410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 --2- 172.21.15.22:0/3013335878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9764102740 0x7f9764102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 -- 172.21.15.22:0/3013335878 >> 172.21.15.22:0/3013335878 conn(0x7f97640fbf60 msgr2=0x7f97640fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 -- 172.21.15.22:0/3013335878 shutdown_connections 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.834+0000 7f976a800640 1 -- 172.21.15.22:0/3013335878 wait complete. 2024-03-24T06:41:49.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.836+0000 7f976a800640 1 Processor -- start 2024-03-24T06:41:49.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.836+0000 7f976a800640 1 -- start start 2024-03-24T06:41:49.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.836+0000 7f976a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.836+0000 7f976a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f97641008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f9769400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f9769400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41084/0 (socket says 172.21.15.22:41084) 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f976a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f976410bb60 0x7f9764100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f976a800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f97641a5f90 con 0x7f9764103060 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f976a800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f97641a6100 con 0x7f9764102740 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f976a800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f97641a62c0 con 0x7f976410bb60 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f9768a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f97641008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f9769400640 1 -- 172.21.15.22:0/858067525 learned_addr learned my addr 172.21.15.22:0/858067525 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.837+0000 7f9769e00640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f976410bb60 0x7f9764100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 -- 172.21.15.22:0/858067525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f976410bb60 msgr2=0x7f9764100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f976410bb60 0x7f9764100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 -- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 msgr2=0x7f97641008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f97641008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 -- 172.21.15.22:0/858067525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c009900 con 0x7f9764102740 2024-03-24T06:41:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9768a00640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f97641008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.838+0000 7f9769400640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f975000d950 tx=0x7f975000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.839+0000 7f9762000640 1 -- 172.21.15.22:0/858067525 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750009a30 con 0x7f9764102740 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.839+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97641a65a0 con 0x7f9764102740 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.839+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97641a6b00 con 0x7f9764102740 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.840+0000 7f9762000640 1 -- 172.21.15.22:0/858067525 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9750009b90 con 0x7f9764102740 2024-03-24T06:41:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.840+0000 7f9762000640 1 -- 172.21.15.22:0/858067525 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750004a00 con 0x7f9764102740 2024-03-24T06:41:49.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.842+0000 7f976eb69740 1 Processor -- start 2024-03-24T06:41:49.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 -- start start 2024-03-24T06:41:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2d70 0x55ab065db410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab065ddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ab065406e0 con 0x55ab065db950 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab0653dc10 con 0x55ab065d2d70 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f976eb69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab065393f0 con 0x55ab065d2450 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f9769e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f9769e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59552/0 (socket says 172.21.15.22:59552) 2024-03-24T06:41:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.843+0000 7f9769e00640 1 -- 172.21.15.22:0/2328176614 learned_addr learned my addr 172.21.15.22:0/2328176614 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 -- 172.21.15.22:0/2328176614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2d70 msgr2=0x55ab065db410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2d70 0x55ab065db410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 -- 172.21.15.22:0/2328176614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 msgr2=0x55ab065ddd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab065ddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 -- 172.21.15.22:0/2328176614 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab064fae40 con 0x55ab065d2450 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.844+0000 7f9769e00640 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f975c02c790 tx=0x7f975c0093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=235d41977185e238 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.845+0000 7f9756600640 1 -- 172.21.15.22:0/2328176614 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c034440 con 0x55ab065d2450 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.845+0000 7f9756600640 1 -- 172.21.15.22:0/2328176614 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f975c003970 con 0x55ab065d2450 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.845+0000 7f9756600640 1 -- 172.21.15.22:0/2328176614 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c032df0 con 0x55ab065d2450 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 -- 172.21.15.22:0/2328176614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 msgr2=0x55ab065d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f975c02c790 tx=0x7f975c0093e0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 -- 172.21.15.22:0/2328176614 shutdown_connections 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab065ddd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2d70 0x55ab065db410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 --2- 172.21.15.22:0/2328176614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065d2830 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f976eb69740 1 -- 172.21.15.22:0/2328176614 >> 172.21.15.22:0/2328176614 conn(0x55ab065ccb80 msgr2=0x55ab065ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.846+0000 7f9762000640 1 -- 172.21.15.22:0/858067525 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f9750004b60 con 0x7f9764102740 2024-03-24T06:41:49.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.847+0000 7f976eb69740 1 -- 172.21.15.22:0/2328176614 shutdown_connections 2024-03-24T06:41:49.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.848+0000 7f9762000640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 0x7f974c0a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.848+0000 7f9762000640 1 -- 172.21.15.22:0/858067525 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f97500cdd00 con 0x7f9764102740 2024-03-24T06:41:49.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.849+0000 7f976eb69740 1 -- 172.21.15.22:0/2328176614 wait complete. 2024-03-24T06:41:49.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.850+0000 7f976eb69740 1 -- start start 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.852+0000 7f976eb69740 1 Processor -- start 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.852+0000 7f9768a00640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 0x7f974c0a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f9768a00640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 0x7f974c0a4680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55ab065cdd10 tx=0x7f97580046c0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 0x55ab065d89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 0x55ab065d8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d9430 0x55ab065e78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ab064de060 con 0x55ab065d9430 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab064ddd00 con 0x55ab065d2450 2024-03-24T06:41:49.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab062a71c0 con 0x55ab065db950 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f976eb69740 10 client.0 _ll_register_callbacks cb 0x55ab06249be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f9768a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 0x55ab065d8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f9768a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 0x55ab065d8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59556/0 (socket says 172.21.15.22:59556) 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.853+0000 7f9768a00640 1 -- 172.21.15.22:0/2227371892 learned_addr learned my addr 172.21.15.22:0/2227371892 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.854+0000 7f9769400640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 0x55ab065d89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.854+0000 7f9769e00640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d9430 0x55ab065e78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 -- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 msgr2=0x55ab065d8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 0x55ab065d8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 -- 172.21.15.22:0/2227371892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d9430 msgr2=0x55ab065e78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d9430 0x55ab065e78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 -- 172.21.15.22:0/2227371892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976410ec80 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f9769400640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 0x55ab065d89b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f97500cb090 tx=0x7f975009ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=862398237d2679e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f974aa00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750094ea0 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.855+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f975009e160 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ab062387e0 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f97500cb310 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f976eb69740 10 client.26021 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ab06537a70 con 0x55ab065d2450 2024-03-24T06:41:49.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ab062387e0 con 0x55ab065d2450 2024-03-24T06:41:49.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.856+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f9750099810 con 0x55ab065d2450 2024-03-24T06:41:49.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 10 client.26021 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:49.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 10 client.26021 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:41:49.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 20 client.26021 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab065fd2a0 0x55ab065ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 4 client.26021 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab06275720 con 0x55ab065fd2a0 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.857+0000 7f9768a00640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab065fd2a0 0x55ab065ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.858+0000 7f974aa00640 10 client.26021 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:49.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.858+0000 7f9768a00640 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab065fd2a0 0x55ab065ff6c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c0983b4992726bd server_cookie=2c370460ef2597b1 in_seq=0 out_seq=0 2024-03-24T06:41:49.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f974aa00640 1 -- 172.21.15.22:0/2227371892 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab06275720 con 0x55ab065fd2a0 2024-03-24T06:41:49.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f974aa00640 10 client.26021 handle_command_reply: tid=0 2024-03-24T06:41:49.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 client.26021 shutdown 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 msgr2=0x55ab065d89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 0x55ab065d89b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f97500cb090 tx=0x7f975009ab70 comp rx=0 tx=0).stop 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 shutdown_connections 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab065fd2a0 0x55ab065ff6c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.859+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d9430 0x55ab065e78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065db950 0x55ab065d8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 1 --2- 172.21.15.22:0/2227371892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d2450 0x55ab065d89b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 >> 172.21.15.22:0/2227371892 conn(0x55ab065ccb80 msgr2=0x55ab065ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f974aa00640 0 client.26021 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 shutdown_connections 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 1 -- 172.21.15.22:0/2227371892 wait complete. 2024-03-24T06:41:49.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.860+0000 7f976eb69740 20 client.26021 trim_cache size 0 max 16384 2024-03-24T06:41:49.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.899+0000 7f976eb69740 1 Processor -- start 2024-03-24T06:41:49.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 -- start start 2024-03-24T06:41:49.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d2450 0x55ab065d72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d9430 0x55ab06604f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ab065cd980 con 0x55ab065d2450 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab065cdb80 con 0x55ab065db950 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f976eb69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab065d7800 con 0x55ab065d9430 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.900+0000 7f9769e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41092/0 (socket says 172.21.15.22:41092) 2024-03-24T06:41:49.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 -- 172.21.15.22:0/1270473038 learned_addr learned my addr 172.21.15.22:0/1270473038 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 -- 172.21.15.22:0/1270473038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d9430 msgr2=0x55ab06604f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d9430 0x55ab06604f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 -- 172.21.15.22:0/1270473038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d2450 msgr2=0x55ab065d72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d2450 0x55ab065d72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 -- 172.21.15.22:0/1270473038 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab064d3b00 con 0x55ab065db950 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.901+0000 7f9769e00640 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f975c0329a0 tx=0x7f975c032a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbc0eaed1dfe90ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.902+0000 7f9755c00640 1 -- 172.21.15.22:0/1270473038 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c032ce0 con 0x55ab065db950 2024-03-24T06:41:49.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.902+0000 7f9755c00640 1 -- 172.21.15.22:0/1270473038 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f975c032580 con 0x55ab065db950 2024-03-24T06:41:49.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.902+0000 7f9755c00640 1 -- 172.21.15.22:0/1270473038 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c04ad20 con 0x55ab065db950 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 -- 172.21.15.22:0/1270473038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 msgr2=0x55ab065e7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f975c0329a0 tx=0x7f975c032a70 comp rx=0 tx=0).stop 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 -- 172.21.15.22:0/1270473038 shutdown_connections 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065db950 0x55ab065e7210 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d9430 0x55ab06604f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 --2- 172.21.15.22:0/1270473038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065d2450 0x55ab065d72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.903+0000 7f976eb69740 1 -- 172.21.15.22:0/1270473038 >> 172.21.15.22:0/1270473038 conn(0x55ab065ccb80 msgr2=0x55ab065cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.904+0000 7f976eb69740 1 -- 172.21.15.22:0/1270473038 shutdown_connections 2024-03-24T06:41:49.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.904+0000 7f976eb69740 1 -- 172.21.15.22:0/1270473038 wait complete. 2024-03-24T06:41:49.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.904+0000 7f976eb69740 1 -- start start 2024-03-24T06:41:49.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.905+0000 7f976eb69740 1 Processor -- start 2024-03-24T06:41:49.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f976eb69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f976eb69740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d9430 0x55ab065dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f9769400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f9769400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59566/0 (socket says 172.21.15.22:59566) 2024-03-24T06:41:49.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f9768a00640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d9430 0x55ab065dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.906+0000 7f9769400640 1 -- 172.21.15.22:0/1469868290 learned_addr learned my addr 172.21.15.22:0/1469868290 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:49.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab066625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ab06662ae0 con 0x55ab065db950 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab06662c50 con 0x55ab065d9430 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab06662e10 con 0x55ab065d2450 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f976eb69740 10 client.0 _ll_register_callbacks cb 0x7f9750099c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f9769e00640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab066625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f9769400640 1 -- 172.21.15.22:0/1469868290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d9430 msgr2=0x55ab065dfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.907+0000 7f9769400640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d9430 0x55ab065dfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.908+0000 7f9769400640 1 -- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 msgr2=0x55ab066625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.908+0000 7f9769400640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab066625a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.908+0000 7f9769400640 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab064fae40 con 0x55ab065d2450 2024-03-24T06:41:49.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.908+0000 7f9769e00640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab066625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:49.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.909+0000 7f9769400640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9750096d00 tx=0x7f9750009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4da90caf745ccf7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:49.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.909+0000 7f974aa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:49.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.910+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9750094ea0 con 0x55ab065d2450 2024-03-24T06:41:49.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.910+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9750099780 con 0x55ab065d2450 2024-03-24T06:41:49.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.911+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f975c003970 con 0x55ab065d2450 2024-03-24T06:41:49.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.912+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9750099e70 con 0x55ab065d2450 2024-03-24T06:41:49.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.912+0000 7f976eb69740 10 client.26071 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:49.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.912+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ab064f96a0 con 0x55ab065d2450 2024-03-24T06:41:49.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.912+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f975c003970 con 0x55ab065d2450 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f97500a2d20 con 0x55ab065d2450 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 10 client.26071 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 10 client.26071 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 20 client.26071 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab06664100 0x55ab06666520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:49.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 4 client.26071 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:49.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.913+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab062c4890 con 0x55ab06664100 2024-03-24T06:41:49.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.915+0000 7f9768a00640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab06664100 0x55ab06666520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:49.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.917+0000 7f9768a00640 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab06664100 0x55ab06666520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8fc4b8543932a34b server_cookie=f50b44b1996c7d4c in_seq=0 out_seq=0 2024-03-24T06:41:49.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.917+0000 7f974aa00640 10 client.26071 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:49.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.917+0000 7f974aa00640 1 -- 172.21.15.22:0/1469868290 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab062c4890 con 0x55ab06664100 2024-03-24T06:41:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.917+0000 7f974aa00640 10 client.26071 handle_command_reply: tid=0 2024-03-24T06:41:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 client.26071 shutdown 2024-03-24T06:41:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 msgr2=0x55ab065df8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9750096d00 tx=0x7f9750009f30 comp rx=0 tx=0).stop 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 shutdown_connections 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55ab06664100 0x55ab06666520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ab065db950 0x55ab066625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab065d9430 0x55ab065dfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 --2- 172.21.15.22:0/1469868290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab065d2450 0x55ab065df8a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 >> 172.21.15.22:0/1469868290 conn(0x55ab065ccb80 msgr2=0x55ab065cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.918+0000 7f974aa00640 0 client.26071 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:41:49.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.919+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 shutdown_connections 2024-03-24T06:41:49.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.919+0000 7f976eb69740 1 -- 172.21.15.22:0/1469868290 wait complete. 2024-03-24T06:41:49.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.919+0000 7f976eb69740 20 client.26071 trim_cache size 0 max 16384 2024-03-24T06:41:49.920 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:49.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.921+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 msgr2=0x7f974c0a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.921+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 0x7f974c0a4680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55ab065cdd10 tx=0x7f97580046c0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.921+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 msgr2=0x7f9764100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.921+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f975000d950 tx=0x7f975000de10 comp rx=0 tx=0).stop 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 shutdown_connections 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f974c0a21c0 0x7f974c0a4680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f976410bb60 0x7f9764100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9764103060 0x7f97641008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 --2- 172.21.15.22:0/858067525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9764102740 0x7f9764100390 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f975000d950 tx=0x7f975000de10 comp rx=0 tx=0).stop 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.922+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 >> 172.21.15.22:0/858067525 conn(0x7f97640fbf60 msgr2=0x7f97640fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.923+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 shutdown_connections 2024-03-24T06:41:49.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:49.923+0000 7f976a800640 1 -- 172.21.15.22:0/858067525 wait complete. 2024-03-24T06:41:49.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:49 smithi143 ceph-mon[36520]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:49.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:49 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1158858037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:41:49.959 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:41:50.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2332570715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0104150 msgr2=0x7fb0e0110660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2332570715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0104150 0x7fb0e0110660 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009cd0 tx=0x7fb0d802c0e0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2332570715 shutdown_connections 2024-03-24T06:41:50.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2332570715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0104150 0x7fb0e0110660 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2332570715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0103790 0x7fb0e0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2332570715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e0109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2332570715 >> 172.21.15.22:0/2332570715 conn(0x7fb0e00ff220 msgr2=0x7fb0e0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2332570715 shutdown_connections 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.275+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2332570715 wait complete. 2024-03-24T06:41:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.277+0000 7fb0dfe00640 1 Processor -- start 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.277+0000 7fb0dfe00640 1 -- start start 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.277+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.277+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0104150 0x7fb0e019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e019ecb0 con 0x7fb0e0104150 2024-03-24T06:41:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e019ee20 con 0x7fb0e01097a0 2024-03-24T06:41:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dfe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e019efe0 con 0x7fb0e0103790 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59582/0 (socket says 172.21.15.22:59582) 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0dea00640 1 -- 172.21.15.22:0/2397970641 learned_addr learned my addr 172.21.15.22:0/2397970641 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0df400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.278+0000 7fb0de000640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0104150 0x7fb0e019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 -- 172.21.15.22:0/2397970641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 msgr2=0x7fb0e019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 -- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0104150 msgr2=0x7fb0e019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0104150 0x7fb0e019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 -- 172.21.15.22:0/2397970641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d0009490 con 0x7fb0e0103790 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0df400640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000ea60 tx=0x7fb0d000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.279+0000 7fb0e4a00640 1 -- 172.21.15.22:0/2397970641 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000cde0 con 0x7fb0e0103790 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.280+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d8009900 con 0x7fb0e0103790 2024-03-24T06:41:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.280+0000 7fb0e4a00640 1 -- 172.21.15.22:0/2397970641 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d00035f0 con 0x7fb0e0103790 2024-03-24T06:41:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.280+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a6960 con 0x7fb0e0103790 2024-03-24T06:41:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.280+0000 7fb0e4a00640 1 -- 172.21.15.22:0/2397970641 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0010690 con 0x7fb0e0103790 2024-03-24T06:41:50.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.286+0000 7fb0e4a00640 1 -- 172.21.15.22:0/2397970641 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb0d00107f0 con 0x7fb0e0103790 2024-03-24T06:41:50.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.288+0000 7fb0e8b75740 1 Processor -- start 2024-03-24T06:41:50.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.289+0000 7fb0e4a00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 0x7fb0c40a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.289+0000 7fb0e4a00640 1 -- 172.21.15.22:0/2397970641 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb0d0014070 con 0x7fb0e0103790 2024-03-24T06:41:50.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.291+0000 7fb0e8b75740 1 -- start start 2024-03-24T06:41:50.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 0x55e87684b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 0x55e876853c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e8767b06e0 con 0x55e87684b580 2024-03-24T06:41:50.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8767adc10 con 0x55e87684ac60 2024-03-24T06:41:50.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.292+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e8767a93f0 con 0x55e876854160 2024-03-24T06:41:50.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.293+0000 7fb0df400640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 0x7fb0c40a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.294+0000 7fb0df400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.294+0000 7fb0df400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59596/0 (socket says 172.21.15.22:59596) 2024-03-24T06:41:50.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.294+0000 7fb0df400640 1 -- 172.21.15.22:0/1801477146 learned_addr learned my addr 172.21.15.22:0/1801477146 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.296+0000 7fb0dea00640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 0x55e87684b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.297+0000 7fb0dea00640 1 -- 172.21.15.22:0/1801477146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 msgr2=0x55e876856550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0de000640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 0x55e876853c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0dea00640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0dea00640 1 -- 172.21.15.22:0/1801477146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 msgr2=0x55e876853c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0dea00640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 0x55e876853c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0dea00640 1 -- 172.21.15.22:0/1801477146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e87676ae40 con 0x55e87684ac60 2024-03-24T06:41:50.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.298+0000 7fb0de000640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 0x55e876853c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:50.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.299+0000 7fb0df400640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:50.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.300+0000 7fb0df400640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 0x7fb0c40a47f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8006010 tx=0x7fb0d80347f0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.311+0000 7fb0dea00640 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 0x55e87684b040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00cccb0 tx=0x7fb0d009c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f35a08176e3ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.312+0000 7fb0cbe00640 1 -- 172.21.15.22:0/1801477146 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00a00a0 con 0x55e87684ac60 2024-03-24T06:41:50.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.313+0000 7fb0cbe00640 1 -- 172.21.15.22:0/1801477146 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d00a46f0 con 0x55e87684ac60 2024-03-24T06:41:50.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.313+0000 7fb0cbe00640 1 -- 172.21.15.22:0/1801477146 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d009b690 con 0x55e87684ac60 2024-03-24T06:41:50.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1801477146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 msgr2=0x55e87684b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 0x55e87684b040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00cccb0 tx=0x7fb0d009c650 comp rx=0 tx=0).stop 2024-03-24T06:41:50.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1801477146 shutdown_connections 2024-03-24T06:41:50.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876854160 0x55e876856550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684b580 0x55e876853c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1801477146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684ac60 0x55e87684b040 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.314+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1801477146 >> 172.21.15.22:0/1801477146 conn(0x55e876845390 msgr2=0x55e876845770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.315+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1801477146 shutdown_connections 2024-03-24T06:41:50.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.315+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1801477146 wait complete. 2024-03-24T06:41:50.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.316+0000 7fb0e8b75740 1 -- start start 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.316+0000 7fb0e8b75740 1 Processor -- start 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.317+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e876851730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44886/0 (socket says 172.21.15.22:44886) 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0de000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e876851730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0dea00640 1 -- 172.21.15.22:0/2235705843 learned_addr learned my addr 172.21.15.22:0/2235705843 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.318+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876851c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.319+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e87674e060 con 0x55e87684ac60 2024-03-24T06:41:50.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.319+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e87674dd00 con 0x55e876854160 2024-03-24T06:41:50.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.319+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e8765171c0 con 0x55e87684b580 2024-03-24T06:41:50.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.319+0000 7fb0df400640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876851c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.320+0000 7fb0dea00640 1 -- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 msgr2=0x55e876851730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.320+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e876851730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.320+0000 7fb0dea00640 1 -- 172.21.15.22:0/2235705843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 msgr2=0x55e876851c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.320+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876851c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.320+0000 7fb0dea00640 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a6960 con 0x55e87684ac60 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.321+0000 7fb0e8b75740 10 client.0 _ll_register_callbacks cb 0x55e8764b9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:50.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.321+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e8764a87e0 con 0x55e87684ac60 2024-03-24T06:41:50.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.322+0000 7fb0dea00640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00a41c0 tx=0x7fb0d00cd160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d9d528d10eb6be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.322+0000 7fb0c8c00640 10 client.15843 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:41:50.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.322+0000 7fb0c8c00640 1 -- 172.21.15.22:0/2235705843 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00ba400 con 0x55e87684ac60 2024-03-24T06:41:50.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.323+0000 7fb0c8c00640 1 -- 172.21.15.22:0/2235705843 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb0d009b3a0 con 0x55e87684ac60 2024-03-24T06:41:50.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.324+0000 7fb0c8c00640 1 -- 172.21.15.22:0/2235705843 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb0d009b5a0 con 0x55e87684ac60 2024-03-24T06:41:50.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.325+0000 7fb0e8b75740 10 client.15843 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:50.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.325+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55e8767a7a70 con 0x55e87684ac60 2024-03-24T06:41:50.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0c8c00640 1 -- 172.21.15.22:0/2235705843 <== mon.0 v2:172.21.15.22:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fb0d0097ca0 con 0x55e87684ac60 2024-03-24T06:41:50.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 10 client.15843 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:50.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 10 client.15843 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:41:50.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 20 client.15843 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:50.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e876866260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 4 client.15843 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:50.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.326+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8764e5720 con 0x55e876863e00 2024-03-24T06:41:50.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.327+0000 7fb0de000640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e876866260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.329+0000 7fb0de000640 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e876866260 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb2b3c3b5a553c97 server_cookie=9d3a8ec769ffc8fd in_seq=0 out_seq=0 2024-03-24T06:41:50.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.330+0000 7fb0c8c00640 10 client.15843 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:50.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.331+0000 7fb0c8c00640 1 -- 172.21.15.22:0/2235705843 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8764e5720 con 0x55e876863e00 2024-03-24T06:41:50.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.331+0000 7fb0c8c00640 10 client.15843 handle_command_reply: tid=0 2024-03-24T06:41:50.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.333+0000 7fb0e8b75740 1 client.15843 shutdown 2024-03-24T06:41:50.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.336+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 msgr2=0x55e8768511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.336+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00a41c0 tx=0x7fb0d00cd160 comp rx=0 tx=0).stop 2024-03-24T06:41:50.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.338+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 shutdown_connections 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e876866260 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0c8c00640 0 client.15843 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876851c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e876851730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/2235705843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e8768511f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.339+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 >> 172.21.15.22:0/2235705843 conn(0x55e876845390 msgr2=0x55e8768470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.340+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 shutdown_connections 2024-03-24T06:41:50.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.341+0000 7fb0e8b75740 1 -- 172.21.15.22:0/2235705843 wait complete. 2024-03-24T06:41:50.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.342+0000 7fb0e8b75740 20 client.15843 trim_cache size 0 max 16384 2024-03-24T06:41:50.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.405+0000 7fb0e8b75740 1 Processor -- start 2024-03-24T06:41:50.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.406+0000 7fb0e8b75740 1 -- start start 2024-03-24T06:41:50.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.407+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.407+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e87685f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.407+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44888/0 (socket says 172.21.15.22:44888) 2024-03-24T06:41:50.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0dea00640 1 -- 172.21.15.22:0/1168437517 learned_addr learned my addr 172.21.15.22:0/1168437517 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876837b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e876838090 con 0x55e87684ac60 2024-03-24T06:41:50.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e876856b20 con 0x55e876854160 2024-03-24T06:41:50.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e8768370d0 con 0x55e87684b580 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876837b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 -- 172.21.15.22:0/1168437517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 msgr2=0x55e87685f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.408+0000 7fb0de000640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e87685f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e87685f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 -- 172.21.15.22:0/1168437517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 msgr2=0x55e876853920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0df400640 1 -- 172.21.15.22:0/1168437517 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e876743b00 con 0x55e876854160 2024-03-24T06:41:50.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0de000640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e87685f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:50.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.409+0000 7fb0dea00640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:50.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.410+0000 7fb0df400640 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876837b50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55e876846520 tx=0x7fb0d803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=165a13d98e3261f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.410+0000 7fb0cb400640 1 -- 172.21.15.22:0/1168437517 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d80326b0 con 0x55e876854160 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.410+0000 7fb0cb400640 1 -- 172.21.15.22:0/1168437517 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8003e50 con 0x55e876854160 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.410+0000 7fb0cb400640 1 -- 172.21.15.22:0/1168437517 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004640 con 0x55e876854160 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 msgr2=0x55e876837b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876837b50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55e876846520 tx=0x7fb0d803d000 comp rx=0 tx=0).stop 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 shutdown_connections 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e876854160 0x55e876837b50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684b580 0x55e87685f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/1168437517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e87684ac60 0x55e876853920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.411+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 >> 172.21.15.22:0/1168437517 conn(0x55e876845390 msgr2=0x55e87683bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.412+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 shutdown_connections 2024-03-24T06:41:50.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.412+0000 7fb0e8b75740 1 -- 172.21.15.22:0/1168437517 wait complete. 2024-03-24T06:41:50.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.413+0000 7fb0e8b75740 1 -- start start 2024-03-24T06:41:50.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.413+0000 7fb0e8b75740 1 Processor -- start 2024-03-24T06:41:50.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.414+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0dea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59632/0 (socket says 172.21.15.22:59632) 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684b580 0x55e876858e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e876854160 0x55e876859360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e8768ca170 con 0x55e876854160 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8768ca2b0 con 0x55e87684b580 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e8768ca3f0 con 0x55e87684ac60 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.415+0000 7fb0e8b75740 10 client.0 _ll_register_callbacks cb 0x7fb0d009bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:50.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.416+0000 7fb0dea00640 1 -- 172.21.15.22:0/3692647531 learned_addr learned my addr 172.21.15.22:0/3692647531 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.416+0000 7fb0de000640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684b580 0x55e876858e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 -- 172.21.15.22:0/3692647531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684b580 msgr2=0x55e876858e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684b580 0x55e876858e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 -- 172.21.15.22:0/3692647531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e876854160 msgr2=0x55e876859360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e876854160 0x55e876859360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 -- 172.21.15.22:0/3692647531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8767a7a70 con 0x55e87684ac60 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0dea00640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0098920 tx=0x7fb0d00ce000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe747eaeaa8402bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d009c8e0 con 0x55e87684ac60 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.417+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb0d00ab020 con 0x55e87684ac60 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d00a46f0 con 0x55e87684ac60 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb0d00046c0 con 0x55e87684ac60 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0e8b75740 10 client.26086 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:50.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7fb0e0110f60 con 0x55e87684ac60 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d00a46f0 con 0x55e87684ac60 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.418+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fb0d0097ca0 con 0x55e87684ac60 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 10 client.26086 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 10 client.26086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 20 client.26086 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e8768cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 4 client.26086 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e876534890 con 0x55e876863e00 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.419+0000 7fb0de000640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e8768cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0de000640 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e8768cd710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef1555d5071a95f2 server_cookie=c87df72ee9e9debb in_seq=0 out_seq=0 2024-03-24T06:41:50.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0c8c00640 10 client.26086 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0c8c00640 1 -- 172.21.15.22:0/3692647531 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e876534890 con 0x55e876863e00 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0c8c00640 10 client.26086 handle_command_reply: tid=0 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 client.26086 shutdown 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 msgr2=0x55e8768588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0098920 tx=0x7fb0d00ce000 comp rx=0 tx=0).stop 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 shutdown_connections 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55e876863e00 0x55e8768cd710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e876854160 0x55e876859360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e87684b580 0x55e876858e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 --2- 172.21.15.22:0/3692647531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e87684ac60 0x55e8768588e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0c8c00640 0 client.26086 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:41:50.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 >> 172.21.15.22:0/3692647531 conn(0x55e876845390 msgr2=0x55e876846550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.420+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 shutdown_connections 2024-03-24T06:41:50.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.421+0000 7fb0e8b75740 1 -- 172.21.15.22:0/3692647531 wait complete. 2024-03-24T06:41:50.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.421+0000 7fb0e8b75740 20 client.26086 trim_cache size 0 max 16384 2024-03-24T06:41:50.422 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 msgr2=0x7fb0c40a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 0x7fb0c40a47f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8006010 tx=0x7fb0d80347f0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 msgr2=0x7fb0e0199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000ea60 tx=0x7fb0d000ef20 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 shutdown_connections 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0c40a2330 0x7fb0c40a47f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e01097a0 0x7fb0e019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e0104150 0x7fb0e019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 --2- 172.21.15.22:0/2397970641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0e0103790 0x7fb0e0199df0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 >> 172.21.15.22:0/2397970641 conn(0x7fb0e00ff220 msgr2=0x7fb0e00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 shutdown_connections 2024-03-24T06:41:50.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.422+0000 7fb0dfe00640 1 -- 172.21.15.22:0/2397970641 wait complete. 2024-03-24T06:41:50.448 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:41:50.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 -- 172.21.15.22:0/3914749878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac100530 msgr2=0x7f8cac100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 --2- 172.21.15.22:0/3914749878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac100530 0x7f8cac100910 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009900 tx=0x7f8c9c02bd40 comp rx=0 tx=0).stop 2024-03-24T06:41:50.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 -- 172.21.15.22:0/3914749878 shutdown_connections 2024-03-24T06:41:50.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 --2- 172.21.15.22:0/3914749878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac105930 0x7f8cac105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 --2- 172.21.15.22:0/3914749878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac100e50 0x7f8cac105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 --2- 172.21.15.22:0/3914749878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac100530 0x7f8cac100910 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 -- 172.21.15.22:0/3914749878 >> 172.21.15.22:0/3914749878 conn(0x7f8cac0fbf60 msgr2=0x7f8cac0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 -- 172.21.15.22:0/3914749878 shutdown_connections 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.772+0000 7f8cb1000640 1 -- 172.21.15.22:0/3914749878 wait complete. 2024-03-24T06:41:50.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 Processor -- start 2024-03-24T06:41:50.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 -- start start 2024-03-24T06:41:50.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac100e50 0x7f8cac19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 0x7f8cac1a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.774+0000 7f8cb1000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1a30a0 con 0x7f8cac100530 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cb1000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1a31e0 con 0x7f8cac105930 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cb1000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1a33a0 con 0x7f8cac100e50 2024-03-24T06:41:50.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cab400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cab400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44900/0 (socket says 172.21.15.22:44900) 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cab400640 1 -- 172.21.15.22:0/645710770 learned_addr learned my addr 172.21.15.22:0/645710770 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cabe00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 0x7f8cac1a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cabe00640 1 -- 172.21.15.22:0/645710770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac100e50 msgr2=0x7f8cac19e720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cabe00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac100e50 0x7f8cac19e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.775+0000 7f8cabe00640 1 -- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 msgr2=0x7f8cac19e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8cabe00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8cabe00640 1 -- 172.21.15.22:0/645710770 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c009560 con 0x7f8cac105930 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8cab400640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8cabe00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 0x7f8cac1a2aa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400ea80 tx=0x7f8ca400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8ca3e00640 1 -- 172.21.15.22:0/645710770 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca400cea0 con 0x7f8cac105930 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cac1a3680 con 0x7f8cac105930 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.776+0000 7f8ca3e00640 1 -- 172.21.15.22:0/645710770 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca4005020 con 0x7f8cac105930 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.777+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cac106ac0 con 0x7f8cac105930 2024-03-24T06:41:50.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.777+0000 7f8ca3e00640 1 -- 172.21.15.22:0/645710770 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4004d20 con 0x7f8cac105930 2024-03-24T06:41:50.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.781+0000 7f8cb53d6740 1 Processor -- start 2024-03-24T06:41:50.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.782+0000 7f8ca3e00640 1 -- 172.21.15.22:0/645710770 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8ca4005190 con 0x7f8cac105930 2024-03-24T06:41:50.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.785+0000 7f8cb53d6740 1 -- start start 2024-03-24T06:41:50.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05bd70 0x55ff6e064410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8ca3e00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 0x7f8c900a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff6dfc96e0 con 0x55ff6e05bd70 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff6dfc6c10 con 0x55ff6e05b450 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff6dfc23f0 con 0x55ff6e064950 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8ca3e00640 1 -- 172.21.15.22:0/645710770 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ca4014070 con 0x7f8cac105930 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8caaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8caaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41174/0 (socket says 172.21.15.22:41174) 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.786+0000 7f8caaa00640 1 -- 172.21.15.22:0/2978751465 learned_addr learned my addr 172.21.15.22:0/2978751465 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8cab400640 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05bd70 0x55ff6e064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 -- 172.21.15.22:0/2978751465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 msgr2=0x55ff6e066d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e066d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 -- 172.21.15.22:0/2978751465 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05bd70 msgr2=0x55ff6e064410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05bd70 0x55ff6e064410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 -- 172.21.15.22:0/2978751465 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff6df83e40 con 0x55ff6e05b450 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.788+0000 7f8caaa00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 0x7f8c900a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.789+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800ab50 tx=0x7f8c9800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d384c40b94b5f747 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.790+0000 7f8ca0c00640 1 -- 172.21.15.22:0/2978751465 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98011640 con 0x55ff6e05b450 2024-03-24T06:41:50.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.791+0000 7f8ca0c00640 1 -- 172.21.15.22:0/2978751465 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c98011d00 con 0x55ff6e05b450 2024-03-24T06:41:50.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.791+0000 7f8ca0c00640 1 -- 172.21.15.22:0/2978751465 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9801a370 con 0x55ff6e05b450 2024-03-24T06:41:50.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.791+0000 7f8caaa00640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 0x7f8c900a4750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c9801e000 tx=0x7f8c9801aac0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.792+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2978751465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 msgr2=0x55ff6e05b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.792+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800ab50 tx=0x7f8c9800f730 comp rx=0 tx=0).stop 2024-03-24T06:41:50.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.795+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2978751465 shutdown_connections 2024-03-24T06:41:50.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.795+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.795+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05bd70 0x55ff6e064410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.795+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2978751465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05b830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.795+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2978751465 >> 172.21.15.22:0/2978751465 conn(0x55ff6e055b80 msgr2=0x55ff6e055f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.796+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2978751465 shutdown_connections 2024-03-24T06:41:50.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.796+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2978751465 wait complete. 2024-03-24T06:41:50.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.798+0000 7f8cb53d6740 1 -- start start 2024-03-24T06:41:50.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.798+0000 7f8cb53d6740 1 Processor -- start 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05b450 0x55ff6e061ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e064950 0x55ff6e062530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff6df67060 con 0x55ff6e05b450 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff6df66d00 con 0x55ff6e064950 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff6dd301c0 con 0x55ff6e05bd70 2024-03-24T06:41:50.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.799+0000 7f8cb53d6740 10 client.0 _ll_register_callbacks cb 0x55ff6dcd2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.800+0000 7f8caaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.800+0000 7f8cab400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05b450 0x55ff6e061ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.800+0000 7f8caaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59672/0 (socket says 172.21.15.22:59672) 2024-03-24T06:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.800+0000 7f8caaa00640 1 -- 172.21.15.22:0/464138252 learned_addr learned my addr 172.21.15.22:0/464138252 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.800+0000 7f8cabe00640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e064950 0x55ff6e062530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 -- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e064950 msgr2=0x55ff6e062530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e064950 0x55ff6e062530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 -- 172.21.15.22:0/464138252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05b450 msgr2=0x55ff6e061ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05b450 0x55ff6e061ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 -- 172.21.15.22:0/464138252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9800a780 con 0x55ff6e05bd70 2024-03-24T06:41:50.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.801+0000 7f8caaa00640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8c98010e70 tx=0x7f8c98009000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f94ca0cc489cdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.802+0000 7f8c95600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:50.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.802+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9801a370 con 0x55ff6e05bd70 2024-03-24T06:41:50.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.802+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ff6dcc17e0 con 0x55ff6e05bd70 2024-03-24T06:41:50.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.803+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8c980047a0 con 0x55ff6e05bd70 2024-03-24T06:41:50.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.804+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8c9800ac40 con 0x55ff6e05bd70 2024-03-24T06:41:50.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.804+0000 7f8cb53d6740 10 client.26092 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:50.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.804+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff6dfc0a70 con 0x55ff6e05bd70 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ff6dcc17e0 con 0x55ff6e05bd70 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f8c98023070 con 0x55ff6e05bd70 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 10 client.26092 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 10 client.26092 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 20 client.26092 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e085ed0 0x55ff6e0882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 4 client.26092 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:50.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.805+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff6dcfe720 con 0x55ff6e085ed0 2024-03-24T06:41:50.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.806+0000 7f8cab400640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e085ed0 0x55ff6e0882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.807+0000 7f8cab400640 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e085ed0 0x55ff6e0882f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66480a21ee0404fd server_cookie=26b5469373848c24 in_seq=0 out_seq=0 2024-03-24T06:41:50.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.807+0000 7f8c95600640 10 client.26092 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:50.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.808+0000 7f8c95600640 1 -- 172.21.15.22:0/464138252 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff6dcfe720 con 0x55ff6e085ed0 2024-03-24T06:41:50.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.808+0000 7f8c95600640 10 client.26092 handle_command_reply: tid=0 2024-03-24T06:41:50.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.809+0000 7f8cb53d6740 1 client.26092 shutdown 2024-03-24T06:41:50.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.809+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 msgr2=0x55ff6e061ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.809+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8c98010e70 tx=0x7f8c98009000 comp rx=0 tx=0).stop 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 shutdown_connections 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e085ed0 0x55ff6e0882f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e064950 0x55ff6e062530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e05bd70 0x55ff6e061ff0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/464138252 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e05b450 0x55ff6e061ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 >> 172.21.15.22:0/464138252 conn(0x55ff6e055b80 msgr2=0x55ff6e0578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8c95600640 0 client.26092 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:50.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 shutdown_connections 2024-03-24T06:41:50.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.810+0000 7f8cb53d6740 1 -- 172.21.15.22:0/464138252 wait complete. 2024-03-24T06:41:50.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.813+0000 7f8cb53d6740 20 client.26092 trim_cache size 0 max 16384 2024-03-24T06:41:50.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.840+0000 7f8cb53d6740 1 Processor -- start 2024-03-24T06:41:50.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.840+0000 7f8cb53d6740 1 -- start start 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e050b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e08dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e02b1c0 con 0x55ff6e062350 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e05b120 con 0x55ff6e05b450 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e05ae90 con 0x55ff6e064950 2024-03-24T06:41:50.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41196/0 (socket says 172.21.15.22:41196) 2024-03-24T06:41:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8caaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e050b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.841+0000 7f8cab400640 1 -- 172.21.15.22:0/2483054330 learned_addr learned my addr 172.21.15.22:0/2483054330 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.842+0000 7f8cabe00640 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e08dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.843+0000 7f8caaa00640 1 -- 172.21.15.22:0/2483054330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 msgr2=0x55ff6e0811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.843+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.843+0000 7f8caaa00640 1 -- 172.21.15.22:0/2483054330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 msgr2=0x55ff6e08dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.843+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e08dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.843+0000 7f8caaa00640 1 -- 172.21.15.22:0/2483054330 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff6df5cb00 con 0x55ff6e064950 2024-03-24T06:41:50.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.844+0000 7f8cab400640 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:50.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.844+0000 7f8caaa00640 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e050b50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ff6e062b40 tx=0x7f8c9800a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3780e7ff01fb490 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.844+0000 7f8c97e00640 1 -- 172.21.15.22:0/2483054330 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9801a590 con 0x55ff6e064950 2024-03-24T06:41:50.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.844+0000 7f8c97e00640 1 -- 172.21.15.22:0/2483054330 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c98010ac0 con 0x55ff6e064950 2024-03-24T06:41:50.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.844+0000 7f8c97e00640 1 -- 172.21.15.22:0/2483054330 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98010ea0 con 0x55ff6e064950 2024-03-24T06:41:50.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.845+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2483054330 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 msgr2=0x55ff6e050b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.845+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e050b50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55ff6e062b40 tx=0x7f8c9800a460 comp rx=0 tx=0).stop 2024-03-24T06:41:50.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2483054330 shutdown_connections 2024-03-24T06:41:50.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e08dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e050b50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/2483054330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e0811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2483054330 >> 172.21.15.22:0/2483054330 conn(0x55ff6e055b80 msgr2=0x55ff6e057860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2483054330 shutdown_connections 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 -- 172.21.15.22:0/2483054330 wait complete. 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 -- start start 2024-03-24T06:41:50.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.846+0000 7f8cb53d6740 1 Processor -- start 2024-03-24T06:41:50.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.847+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.847+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e05f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.847+0000 7f8cb53d6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cb53d6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e0eb730 con 0x55ff6e062350 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59688/0 (socket says 172.21.15.22:59688) 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cabe00640 1 -- 172.21.15.22:0/1007057838 learned_addr learned my addr 172.21.15.22:0/1007057838 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e0eb870 con 0x55ff6e05b450 2024-03-24T06:41:50.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8caaa00640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e05f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff6e0eb9e0 con 0x55ff6e064950 2024-03-24T06:41:50.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.848+0000 7f8cb53d6740 10 client.0 _ll_register_callbacks cb 0x7f8c9801f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 -- 172.21.15.22:0/1007057838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 msgr2=0x55ff6e0eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 -- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 msgr2=0x55ff6e05ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cac115290 con 0x55ff6e062350 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8cabe00640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8caaa00640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e05f390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800f770 tx=0x7f8c9800a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e6c6b5f466eb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:50.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.849+0000 7f8c95600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98010ea0 con 0x55ff6e062350 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8c98000e40 con 0x55ff6e062350 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ff6dd4d890 con 0x55ff6e062350 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8c9800a9a0 con 0x55ff6e062350 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8cb53d6740 10 client.15858 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:50.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.850+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x7f8c98011e70 con 0x55ff6e062350 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ff6dd4d890 con 0x55ff6e062350 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f8c98023070 con 0x55ff6e062350 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 10 client.15858 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 10 client.15858 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 20 client.15858 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e0eccb0 0x55ff6e0ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 4 client.15858 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.851+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff6dc6d700 con 0x55ff6e0eccb0 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8cab400640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e0eccb0 0x55ff6e0ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:50.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8cab400640 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e0eccb0 0x55ff6e0ef0a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7d79460d540febf server_cookie=5c7b9a3d7ba973bd in_seq=0 out_seq=0 2024-03-24T06:41:50.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8c95600640 10 client.15858 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8c95600640 1 -- 172.21.15.22:0/1007057838 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff6dc6d700 con 0x55ff6e0eccb0 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8c95600640 10 client.15858 handle_command_reply: tid=0 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.852+0000 7f8cb53d6740 1 client.15858 shutdown 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 msgr2=0x55ff6e05f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e05f390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800f770 tx=0x7f8c9800a550 comp rx=0 tx=0).stop 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 shutdown_connections 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff6e064950 0x55ff6e0eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff6e062350 0x55ff6e05f390 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55ff6e0eccb0 0x55ff6e0ef0a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 --2- 172.21.15.22:0/1007057838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff6e05b450 0x55ff6e05ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 >> 172.21.15.22:0/1007057838 conn(0x55ff6e055b80 msgr2=0x55ff6e061310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8c95600640 0 client.15858 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:41:50.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 shutdown_connections 2024-03-24T06:41:50.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 1 -- 172.21.15.22:0/1007057838 wait complete. 2024-03-24T06:41:50.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.853+0000 7f8cb53d6740 20 client.15858 trim_cache size 0 max 16384 2024-03-24T06:41:50.855 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 msgr2=0x7f8c900a4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 0x7f8c900a4750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c9801e000 tx=0x7f8c9801aac0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 msgr2=0x7f8cac1a2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 0x7f8cac1a2aa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ca400ea80 tx=0x7f8ca400ef40 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 shutdown_connections 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8c900a2290 0x7f8c900a4750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cac105930 0x7f8cac1a2aa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac100e50 0x7f8cac19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 --2- 172.21.15.22:0/645710770 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cac100530 0x7f8cac19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 >> 172.21.15.22:0/645710770 conn(0x7f8cac0fbf60 msgr2=0x7f8cac1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 shutdown_connections 2024-03-24T06:41:50.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:50.854+0000 7f8cb1000640 1 -- 172.21.15.22:0/645710770 wait complete. 2024-03-24T06:41:50.870 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 -- 172.21.15.22:0/2145301108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024109950 msgr2=0x7fe024109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 --2- 172.21.15.22:0/2145301108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024109950 0x7fe024109d30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe010009900 tx=0x7fe01002bc30 comp rx=0 tx=0).stop 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 -- 172.21.15.22:0/2145301108 shutdown_connections 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 --2- 172.21.15.22:0/2145301108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024104820 0x7fe024110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 --2- 172.21.15.22:0/2145301108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe0241042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 --2- 172.21.15.22:0/2145301108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024109950 0x7fe024109d30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.249+0000 7fe02b600640 1 -- 172.21.15.22:0/2145301108 >> 172.21.15.22:0/2145301108 conn(0x7fe024076150 msgr2=0x7fe024078570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.250+0000 7fe02b600640 1 -- 172.21.15.22:0/2145301108 shutdown_connections 2024-03-24T06:41:51.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.250+0000 7fe02b600640 1 -- 172.21.15.22:0/2145301108 wait complete. 2024-03-24T06:41:51.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.251+0000 7fe02b600640 1 Processor -- start 2024-03-24T06:41:51.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.251+0000 7fe02b600640 1 -- start start 2024-03-24T06:41:51.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe024199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024104820 0x7fe02419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe02419ee40 con 0x7fe024103e60 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe02419efb0 con 0x7fe024104820 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe02b600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe02419f170 con 0x7fe024109950 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe029600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe029600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59692/0 (socket says 172.21.15.22:59692) 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe029600640 1 -- 172.21.15.22:0/1249023611 learned_addr learned my addr 172.21.15.22:0/1249023611 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.252+0000 7fe028c00640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe024199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe023e00640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024104820 0x7fe02419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe029600640 1 -- 172.21.15.22:0/1249023611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024104820 msgr2=0x7fe02419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe029600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024104820 0x7fe02419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe029600640 1 -- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 msgr2=0x7fe024199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe029600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe024199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe029600640 1 -- 172.21.15.22:0/1249023611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01c009840 con 0x7fe024109950 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.253+0000 7fe028c00640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe024199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe029600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00efd0 tx=0x7fe01c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe021600640 1 -- 172.21.15.22:0/1249023611 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c003b70 con 0x7fe024109950 2024-03-24T06:41:51.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe021600640 1 -- 172.21.15.22:0/1249023611 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01c003cd0 con 0x7fe024109950 2024-03-24T06:41:51.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe021600640 1 -- 172.21.15.22:0/1249023611 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c010960 con 0x7fe024109950 2024-03-24T06:41:51.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe010009560 con 0x7fe024109950 2024-03-24T06:41:51.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.254+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a6af0 con 0x7fe024109950 2024-03-24T06:41:51.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.259+0000 7fe02e3c2740 1 Processor -- start 2024-03-24T06:41:51.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.263+0000 7fe02e3c2740 1 -- start start 2024-03-24T06:41:51.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.264+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.264+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 0x563351dd8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.265+0000 7fe023e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.265+0000 7fe023e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44966/0 (socket says 172.21.15.22:44966) 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.265+0000 7fe023e00640 1 -- 172.21.15.22:0/397653119 learned_addr learned my addr 172.21.15.22:0/397653119 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.265+0000 7fe028c00640 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 0x563351dd8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.265+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dd8780 0x563351ddab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.266+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563351d3d6e0 con 0x563351dcf280 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.266+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563351d3ac10 con 0x563351dcfba0 2024-03-24T06:41:51.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.266+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563351d363f0 con 0x563351dd8780 2024-03-24T06:41:51.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe021600640 1 -- 172.21.15.22:0/1249023611 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe01c004590 con 0x7fe024109950 2024-03-24T06:41:51.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe028c00640 1 -- 172.21.15.22:0/397653119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dd8780 msgr2=0x563351ddab70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:41:51.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe028c00640 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dd8780 0x563351ddab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe028c00640 1 -- 172.21.15.22:0/397653119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 msgr2=0x563351dcf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:51.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe028c00640 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.267+0000 7fe028c00640 1 -- 172.21.15.22:0/397653119 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563351cf7e40 con 0x563351dcfba0 2024-03-24T06:41:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.268+0000 7fe023e00640 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.268+0000 7fe028c00640 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 0x563351dd8240 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe010036040 tx=0x7fe01002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a939071e19232371 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.269+0000 7fe015c00640 1 -- 172.21.15.22:0/397653119 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01002dea0 con 0x563351dcfba0 2024-03-24T06:41:51.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.269+0000 7fe015c00640 1 -- 172.21.15.22:0/397653119 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe010004040 con 0x563351dcfba0 2024-03-24T06:41:51.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.271+0000 7fe021600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 0x7fe00c0a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.271+0000 7fe021600640 1 -- 172.21.15.22:0/1249023611 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe01c014070 con 0x7fe024109950 2024-03-24T06:41:51.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.272+0000 7fe023e00640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 0x7fe00c0a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.270+0000 7fe015c00640 1 -- 172.21.15.22:0/397653119 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01002dea0 con 0x563351dcfba0 2024-03-24T06:41:51.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.273+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 msgr2=0x563351dd8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.273+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 0x563351dd8240 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe010036040 tx=0x7fe01002c650 comp rx=0 tx=0).stop 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.273+0000 7fe023e00640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 0x7fe00c0a48c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x563351dcab40 tx=0x7fe018005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 shutdown_connections 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dd8780 0x563351ddab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcfba0 0x563351dd8240 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/397653119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcf280 0x563351dcf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 >> 172.21.15.22:0/397653119 conn(0x563351dc99b0 msgr2=0x563351dc9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 shutdown_connections 2024-03-24T06:41:51.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.274+0000 7fe02e3c2740 1 -- 172.21.15.22:0/397653119 wait complete. 2024-03-24T06:41:51.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.275+0000 7fe02e3c2740 1 -- start start 2024-03-24T06:41:51.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.275+0000 7fe02e3c2740 1 Processor -- start 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.275+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 0x563351dd5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe028c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe028c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41254/0 (socket says 172.21.15.22:41254) 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe023e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 0x563351dd5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe023e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 0x563351dd5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59698/0 (socket says 172.21.15.22:59698) 2024-03-24T06:41:51.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.276+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe029600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe029600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:44968/0 (socket says 172.21.15.22:44968) 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe023e00640 1 -- 172.21.15.22:0/747860260 learned_addr learned my addr 172.21.15.22:0/747860260 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563351cdb060 con 0x563351dd8780 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563351cdad00 con 0x563351dcf280 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563351aa41c0 con 0x563351dcfba0 2024-03-24T06:41:51.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.277+0000 7fe02e3c2740 10 client.0 _ll_register_callbacks cb 0x563351a46be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:51.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 -- 172.21.15.22:0/747860260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 msgr2=0x563351dd5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:51.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 0x563351dd5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 -- 172.21.15.22:0/747860260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 msgr2=0x563351dd5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:51.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe024111080 con 0x563351dd8780 2024-03-24T06:41:51.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.278+0000 7fe029600640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0cbce0 tx=0x7fe01c09b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b754a7df506603d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe028c00640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:51.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe00a000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c0cc160 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe01c09f0d0 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563351a357e0 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe01c0a3720 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe02e3c2740 10 client.15870 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.279+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563351d34a70 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.280+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563351a357e0 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.280+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fe01c09d070 con 0x563351dd8780 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 10 client.15870 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 10 client.15870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 20 client.15870 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351df1380 0x563351df37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 4 client.15870 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.281+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563351a72720 con 0x563351df1380 2024-03-24T06:41:51.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.282+0000 7fe028c00640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351df1380 0x563351df37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.283+0000 7fe028c00640 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351df1380 0x563351df37e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db4d7d318b79f63f server_cookie=175abfbb3058b84f in_seq=0 out_seq=0 2024-03-24T06:41:51.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.283+0000 7fe00a000640 10 client.15870 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:51.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.284+0000 7fe00a000640 1 -- 172.21.15.22:0/747860260 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563351a72720 con 0x563351df1380 2024-03-24T06:41:51.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.284+0000 7fe00a000640 10 client.15870 handle_command_reply: tid=0 2024-03-24T06:41:51.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.284+0000 7fe02e3c2740 1 client.15870 shutdown 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 msgr2=0x563351dd6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe01c0cbce0 tx=0x7fe01c09b680 comp rx=0 tx=0).stop 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 shutdown_connections 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dd8780 0x563351dd6110 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcfba0 0x563351dd5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351df1380 0x563351df37e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/747860260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dcf280 0x563351dd5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 >> 172.21.15.22:0/747860260 conn(0x563351dc99b0 msgr2=0x563351dcb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.285+0000 7fe00a000640 0 client.15870 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:51.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.286+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 shutdown_connections 2024-03-24T06:41:51.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.286+0000 7fe02e3c2740 1 -- 172.21.15.22:0/747860260 wait complete. 2024-03-24T06:41:51.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.289+0000 7fe02e3c2740 20 client.15870 trim_cache size 0 max 16384 2024-03-24T06:41:51.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.338+0000 7fe02e3c2740 1 Processor -- start 2024-03-24T06:41:51.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.339+0000 7fe02e3c2740 1 -- start start 2024-03-24T06:41:51.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351df90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563351a72750 con 0x563351dcfba0 2024-03-24T06:41:51.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563351a35810 con 0x563351dd8780 2024-03-24T06:41:51.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.341+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563351dca9b0 con 0x563351dcf280 2024-03-24T06:41:51.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.342+0000 7fe028c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.343+0000 7fe029600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351df90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.343+0000 7fe028c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59710/0 (socket says 172.21.15.22:59710) 2024-03-24T06:41:51.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.343+0000 7fe028c00640 1 -- 172.21.15.22:0/2487018209 learned_addr learned my addr 172.21.15.22:0/2487018209 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:51.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.343+0000 7fe023e00640 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe028c00640 1 -- 172.21.15.22:0/2487018209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 msgr2=0x563351df90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe028c00640 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351df90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe028c00640 1 -- 172.21.15.22:0/2487018209 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 msgr2=0x563351dec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe028c00640 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe028c00640 1 -- 172.21.15.22:0/2487018209 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563351cd0b00 con 0x563351dcf280 2024-03-24T06:41:51.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.344+0000 7fe023e00640 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:51.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.345+0000 7fe028c00640 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe01002c170 tx=0x7fe010041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e675ed6976975b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.345+0000 7fe015200640 1 -- 172.21.15.22:0/2487018209 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01002dea0 con 0x563351dcf280 2024-03-24T06:41:51.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.345+0000 7fe015200640 1 -- 172.21.15.22:0/2487018209 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0100513f0 con 0x563351dcf280 2024-03-24T06:41:51.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.345+0000 7fe015200640 1 -- 172.21.15.22:0/2487018209 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010004f10 con 0x563351dcf280 2024-03-24T06:41:51.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.347+0000 7fe02e3c2740 1 -- 172.21.15.22:0/2487018209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 msgr2=0x563351dd7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.347+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe01002c170 tx=0x7fe010041040 comp rx=0 tx=0).stop 2024-03-24T06:41:51.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.347+0000 7fe02e3c2740 1 -- 172.21.15.22:0/2487018209 shutdown_connections 2024-03-24T06:41:51.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.347+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351df90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.347+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.348+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/2487018209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351dd7dc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.348+0000 7fe02e3c2740 1 -- 172.21.15.22:0/2487018209 >> 172.21.15.22:0/2487018209 conn(0x563351dc99b0 msgr2=0x563351dd5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.348+0000 7fe02e3c2740 1 -- 172.21.15.22:0/2487018209 shutdown_connections 2024-03-24T06:41:51.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.348+0000 7fe02e3c2740 1 -- 172.21.15.22:0/2487018209 wait complete. 2024-03-24T06:41:51.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.349+0000 7fe02e3c2740 1 -- start start 2024-03-24T06:41:51.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.350+0000 7fe02e3c2740 1 Processor -- start 2024-03-24T06:41:51.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dd1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351dd1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563351de95e0 con 0x563351dcfba0 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563351de9750 con 0x563351dd8780 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563351dd24c0 con 0x563351dcf280 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe028c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe02e3c2740 10 client.0 _ll_register_callbacks cb 0x7fe01c009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe028c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59714/0 (socket says 172.21.15.22:59714) 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.354+0000 7fe028c00640 1 -- 172.21.15.22:0/1776231473 learned_addr learned my addr 172.21.15.22:0/1776231473 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.355+0000 7fe028c00640 1 -- 172.21.15.22:0/1776231473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 msgr2=0x563351dd1f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.355+0000 7fe028c00640 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351dd1f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.355+0000 7fe028c00640 1 -- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 msgr2=0x563351dd1a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.355+0000 7fe028c00640 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dd1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.355+0000 7fe028c00640 1 -- 172.21.15.22:0/1776231473 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563351cf7e40 con 0x563351dcf280 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.356+0000 7fe028c00640 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe01002c690 tx=0x7fe010004bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9919179d9858d49c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.356+0000 7fe00a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.356+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100044f0 con 0x563351dcf280 2024-03-24T06:41:51.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.356+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe01002cbd0 con 0x563351dcf280 2024-03-24T06:41:51.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.357+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe010004040 con 0x563351dcf280 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.357+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe010004f10 con 0x563351dcf280 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.357+0000 7fe02e3c2740 10 client.26113 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.357+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563351cf66a0 con 0x563351dcf280 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe010004040 con 0x563351dcf280 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fe010047020 con 0x563351dcf280 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 10 client.26113 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 10 client.26113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 20 client.26113 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:51.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351e575a0 0x563351e599c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:51.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 4 client.26113 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:41:51.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.358+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563351ac1890 con 0x563351e575a0 2024-03-24T06:41:51.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.360+0000 7fe023e00640 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351e575a0 0x563351e599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:51.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.362+0000 7fe023e00640 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351e575a0 0x563351e599c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e1a4cc588e4984d server_cookie=65c8696d79e332cd in_seq=0 out_seq=0 2024-03-24T06:41:51.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.362+0000 7fe00a000640 10 client.26113 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:51.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.364+0000 7fe00a000640 1 -- 172.21.15.22:0/1776231473 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563351ac1890 con 0x563351e575a0 2024-03-24T06:41:51.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.364+0000 7fe00a000640 10 client.26113 handle_command_reply: tid=0 2024-03-24T06:41:51.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.364+0000 7fe02e3c2740 1 client.26113 shutdown 2024-03-24T06:41:51.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.365+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 msgr2=0x563351de8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.365+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe01002c690 tx=0x7fe010004bb0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.365+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 shutdown_connections 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.365+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563351e575a0 0x563351e599c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.366+0000 7fe00a000640 0 client.26113 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.366+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563351dd8780 0x563351dd1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.366+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563351dcfba0 0x563351dd1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.366+0000 7fe02e3c2740 1 --2- 172.21.15.22:0/1776231473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563351dcf280 0x563351de8f10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.366+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 >> 172.21.15.22:0/1776231473 conn(0x563351dc99b0 msgr2=0x563351dcb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.368+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 shutdown_connections 2024-03-24T06:41:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.368+0000 7fe02e3c2740 1 -- 172.21.15.22:0/1776231473 wait complete. 2024-03-24T06:41:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.369+0000 7fe02e3c2740 20 client.26113 trim_cache size 0 max 16384 2024-03-24T06:41:51.369 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:41:51.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 msgr2=0x7fe00c0a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 0x7fe00c0a48c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x563351dcab40 tx=0x7fe018005f90 comp rx=0 tx=0).stop 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 msgr2=0x7fe02419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00efd0 tx=0x7fe01c00c6a0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 shutdown_connections 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe00c0a2400 0x7fe00c0a48c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe024109950 0x7fe02419e840 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe024104820 0x7fe02419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 --2- 172.21.15.22:0/1249023611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe024103e60 0x7fe024199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:51.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 >> 172.21.15.22:0/1249023611 conn(0x7fe024076150 msgr2=0x7fe024077890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:51.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.371+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 shutdown_connections 2024-03-24T06:41:51.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:51.372+0000 7fe02b600640 1 -- 172.21.15.22:0/1249023611 wait complete. 2024-03-24T06:41:51.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:51 smithi022 ceph-mon[32003]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:51.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:51 smithi155 ceph-mon[36262]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:51.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:51 smithi143 ceph-mon[36520]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 139 MiB/s wr, 71 op/s 2024-03-24T06:41:52.399 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:41:52.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:41:52.399 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:41:52.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 -- 172.21.15.22:0/3818532273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd6300968e0 msgr2=0x7fd6300a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 --2- 172.21.15.22:0/3818532273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd6300968e0 0x7fd6300a2b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6380673e0 tx=0x7fd638097310 comp rx=0 tx=0).stop 2024-03-24T06:41:52.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 -- 172.21.15.22:0/3818532273 shutdown_connections 2024-03-24T06:41:52.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 --2- 172.21.15.22:0/3818532273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd6300968e0 0x7fd6300a2b70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 --2- 172.21.15.22:0/3818532273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 0x7fd6300963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 --2- 172.21.15.22:0/3818532273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd63009ace0 0x7fd63009b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 -- 172.21.15.22:0/3818532273 >> 172.21.15.22:0/3818532273 conn(0x7fd63000cb30 msgr2=0x7fd63000cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 -- 172.21.15.22:0/3818532273 shutdown_connections 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.660+0000 7fd63ce00640 1 -- 172.21.15.22:0/3818532273 wait complete. 2024-03-24T06:41:52.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.662+0000 7fd63ce00640 1 Processor -- start 2024-03-24T06:41:52.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.662+0000 7fd63ce00640 1 -- start start 2024-03-24T06:41:52.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.662+0000 7fd63ce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 0x7fd63012a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.662+0000 7fd63ce00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd63ce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd63009ace0 0x7fd63012e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd63ce00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd63012ef50 con 0x7fd63009ace0 2024-03-24T06:41:52.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd63ce00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd63012f0c0 con 0x7fd6300968e0 2024-03-24T06:41:52.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd63ce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd63012f280 con 0x7fd630095f20 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41274/0 (socket says 172.21.15.22:41274) 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 -- 172.21.15.22:0/826853615 learned_addr learned my addr 172.21.15.22:0/826853615 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd637e00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd63009ace0 0x7fd63012e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd637400640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 0x7fd63012a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 -- 172.21.15.22:0/826853615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 msgr2=0x7fd63012a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 0x7fd63012a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 -- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd63009ace0 msgr2=0x7fd63012e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.663+0000 7fd636a00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd63009ace0 0x7fd63012e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd636a00640 1 -- 172.21.15.22:0/826853615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd62c00d970 con 0x7fd6300968e0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd636a00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d770 tx=0x7fd62400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd62be00640 1 -- 172.21.15.22:0/826853615 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624009c30 con 0x7fd6300968e0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd62be00640 1 -- 172.21.15.22:0/826853615 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd62400de80 con 0x7fd6300968e0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63804fcc0 con 0x7fd6300968e0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd62be00640 1 -- 172.21.15.22:0/826853615 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624004e80 con 0x7fd6300968e0 2024-03-24T06:41:52.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.664+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd630136c00 con 0x7fd6300968e0 2024-03-24T06:41:52.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.669+0000 7fd641069740 1 Processor -- start 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.669+0000 7fd641069740 1 -- start start 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.669+0000 7fd62be00640 1 -- 172.21.15.22:0/826853615 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd6240104c0 con 0x7fd6300968e0 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae8fbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae904800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae907130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5576ae8612d0 con 0x5576ae904d40 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5576ae85e800 con 0x5576ae8fb840 2024-03-24T06:41:52.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.670+0000 7fd641069740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5576ae859fe0 con 0x5576ae8fc160 2024-03-24T06:41:52.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.672+0000 7fd62be00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 0x7fd6200a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.672+0000 7fd62be00640 1 -- 172.21.15.22:0/826853615 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd6240cdf50 con 0x7fd6300968e0 2024-03-24T06:41:52.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.673+0000 7fd637e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae8fbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.673+0000 7fd637e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae8fbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41290/0 (socket says 172.21.15.22:41290) 2024-03-24T06:41:52.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.673+0000 7fd637e00640 1 -- 172.21.15.22:0/1721677562 learned_addr learned my addr 172.21.15.22:0/1721677562 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:52.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.673+0000 7fd637400640 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae904800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.673+0000 7fd636a00640 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae907130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 -- 172.21.15.22:0/1721677562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 msgr2=0x5576ae8fbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae8fbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 -- 172.21.15.22:0/1721677562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 msgr2=0x5576ae907130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae907130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 -- 172.21.15.22:0/1721677562 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576ae81be90 con 0x5576ae8fc160 2024-03-24T06:41:52.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.674+0000 7fd637400640 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae904800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00dde0 tx=0x7fd62c00d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68e057211d70c65c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.675+0000 7fd628c00640 1 -- 172.21.15.22:0/1721677562 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c01a070 con 0x5576ae8fc160 2024-03-24T06:41:52.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.675+0000 7fd628c00640 1 -- 172.21.15.22:0/1721677562 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd62c005500 con 0x5576ae8fc160 2024-03-24T06:41:52.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.675+0000 7fd628c00640 1 -- 172.21.15.22:0/1721677562 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c012040 con 0x5576ae8fc160 2024-03-24T06:41:52.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.676+0000 7fd637e00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 0x7fd6200a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.676+0000 7fd641069740 1 -- 172.21.15.22:0/1721677562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 msgr2=0x5576ae904800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.676+0000 7fd641069740 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae904800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00dde0 tx=0x7fd62c00d670 comp rx=0 tx=0).stop 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd637e00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 0x7fd6200a4770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd638064c00 tx=0x7fd638097310 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 -- 172.21.15.22:0/1721677562 shutdown_connections 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae907130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae904800 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 --2- 172.21.15.22:0/1721677562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae8fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 -- 172.21.15.22:0/1721677562 >> 172.21.15.22:0/1721677562 conn(0x5576ae8f5f70 msgr2=0x5576ae8f6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 -- 172.21.15.22:0/1721677562 shutdown_connections 2024-03-24T06:41:52.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.677+0000 7fd641069740 1 -- 172.21.15.22:0/1721677562 wait complete. 2024-03-24T06:41:52.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.679+0000 7fd641069740 1 -- start start 2024-03-24T06:41:52.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.679+0000 7fd641069740 1 Processor -- start 2024-03-24T06:41:52.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 0x5576ae901c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5576ae7fefc0 con 0x5576ae8fb840 2024-03-24T06:41:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5576ae7fec60 con 0x5576ae904d40 2024-03-24T06:41:52.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5576ae5a20c0 con 0x5576ae8fc160 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd641069740 10 client.0 _ll_register_callbacks cb 0x5576ae816e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.681+0000 7fd637e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.682+0000 7fd637e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41300/0 (socket says 172.21.15.22:41300) 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.682+0000 7fd637e00640 1 -- 172.21.15.22:0/3520393045 learned_addr learned my addr 172.21.15.22:0/3520393045 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.682+0000 7fd637e00640 1 -- 172.21.15.22:0/3520393045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 msgr2=0x5576ae9021a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.682+0000 7fd637e00640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9021a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.682+0000 7fd637e00640 1 -- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 msgr2=0x5576ae901c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:52.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.684+0000 7fd637400640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 0x5576ae901c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.684+0000 7fd637e00640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 0x5576ae901c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.684+0000 7fd637e00640 1 -- 172.21.15.22:0/3520393045 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd62c00d460 con 0x5576ae904d40 2024-03-24T06:41:52.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.684+0000 7fd637400640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 0x5576ae901c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:52.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.687+0000 7fd61d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:52.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.687+0000 7fd637e00640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd6380673e0 tx=0x7fd638099040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0caefbe535b3fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.687+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380ad040 con 0x5576ae904d40 2024-03-24T06:41:52.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.688+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd63806f070 con 0x5576ae904d40 2024-03-24T06:41:52.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.688+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576ae87ee90 con 0x5576ae904d40 2024-03-24T06:41:52.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.688+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd6380a8800 con 0x5576ae904d40 2024-03-24T06:41:52.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.688+0000 7fd641069740 10 client.26090 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:52.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.688+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5576ae858660 con 0x5576ae904d40 2024-03-24T06:41:52.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.689+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5576ae87ee90 con 0x5576ae904d40 2024-03-24T06:41:52.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.689+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd6380a7070 con 0x5576ae904d40 2024-03-24T06:41:52.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 10 client.26090 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:52.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:52.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 20 client.26090 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:52.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae9261e0 0x5576ae928620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 4 client.26090 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:52.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.690+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576ae850210 con 0x5576ae9261e0 2024-03-24T06:41:52.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.691+0000 7fd637400640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae9261e0 0x5576ae928620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.692+0000 7fd637400640 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae9261e0 0x5576ae928620 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca54f6c7f6db0dd8 server_cookie=d9b0fcf67e9eb28a in_seq=0 out_seq=0 2024-03-24T06:41:52.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd61d600640 10 client.26090 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:52.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd61d600640 1 -- 172.21.15.22:0/3520393045 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576ae850210 con 0x5576ae9261e0 2024-03-24T06:41:52.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd61d600640 10 client.26090 handle_command_reply: tid=0 2024-03-24T06:41:52.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 client.26090 shutdown 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 msgr2=0x5576ae9026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd6380673e0 tx=0x7fd638099040 comp rx=0 tx=0).stop 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 shutdown_connections 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae9261e0 0x5576ae928620 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae904d40 0x5576ae9026e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 --2- 172.21.15.22:0/3520393045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fb840 0x5576ae901c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 >> 172.21.15.22:0/3520393045 conn(0x5576ae8f5f70 msgr2=0x5576ae8f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.695+0000 7fd61d600640 0 client.26090 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.696+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 shutdown_connections 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.696+0000 7fd641069740 1 -- 172.21.15.22:0/3520393045 wait complete. 2024-03-24T06:41:52.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.696+0000 7fd641069740 20 client.26090 trim_cache size 0 max 16384 2024-03-24T06:41:52.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.739+0000 7fd641069740 1 Processor -- start 2024-03-24T06:41:52.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.739+0000 7fd641069740 1 -- start start 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae904390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 0x5576ae921640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5576ae918a20 con 0x5576ae8fc160 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5576ae921b80 con 0x5576ae8fb840 2024-03-24T06:41:52.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.740+0000 7fd641069740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5576ae8ebd60 con 0x5576ae904d40 2024-03-24T06:41:52.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.741+0000 7fd637e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.741+0000 7fd637e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59750/0 (socket says 172.21.15.22:59750) 2024-03-24T06:41:52.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.741+0000 7fd637e00640 1 -- 172.21.15.22:0/3621300118 learned_addr learned my addr 172.21.15.22:0/3621300118 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:52.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.741+0000 7fd637400640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae904390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.742+0000 7fd636a00640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 0x5576ae921640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.743+0000 7fd637e00640 1 -- 172.21.15.22:0/3621300118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 msgr2=0x5576ae904390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.743+0000 7fd637e00640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae904390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.743+0000 7fd637e00640 1 -- 172.21.15.22:0/3621300118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 msgr2=0x5576ae921640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.743+0000 7fd637e00640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 0x5576ae921640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.743+0000 7fd637e00640 1 -- 172.21.15.22:0/3621300118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576ae7f4ae0 con 0x5576ae904d40 2024-03-24T06:41:52.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.744+0000 7fd636a00640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 0x5576ae921640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:52.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.744+0000 7fd637e00640 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd638066fd0 tx=0x7fd638062b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=208539ea1326204c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.745+0000 7fd61fe00640 1 -- 172.21.15.22:0/3621300118 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380ad070 con 0x5576ae904d40 2024-03-24T06:41:52.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.745+0000 7fd61fe00640 1 -- 172.21.15.22:0/3621300118 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6380733e0 con 0x5576ae904d40 2024-03-24T06:41:52.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.745+0000 7fd61fe00640 1 -- 172.21.15.22:0/3621300118 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380ad070 con 0x5576ae904d40 2024-03-24T06:41:52.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.746+0000 7fd641069740 1 -- 172.21.15.22:0/3621300118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 msgr2=0x5576ae900dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.746+0000 7fd641069740 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd638066fd0 tx=0x7fd638062b80 comp rx=0 tx=0).stop 2024-03-24T06:41:52.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 -- 172.21.15.22:0/3621300118 shutdown_connections 2024-03-24T06:41:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae904d40 0x5576ae900dc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae8fc160 0x5576ae921640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 --2- 172.21.15.22:0/3621300118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae904390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 -- 172.21.15.22:0/3621300118 >> 172.21.15.22:0/3621300118 conn(0x5576ae8f5f70 msgr2=0x5576ae8ebf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 -- 172.21.15.22:0/3621300118 shutdown_connections 2024-03-24T06:41:52.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.747+0000 7fd641069740 1 -- 172.21.15.22:0/3621300118 wait complete. 2024-03-24T06:41:52.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.748+0000 7fd641069740 1 -- start start 2024-03-24T06:41:52.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.748+0000 7fd641069740 1 Processor -- start 2024-03-24T06:41:52.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.750+0000 7fd641069740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae91df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.750+0000 7fd641069740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9192d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.750+0000 7fd641069740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.751+0000 7fd637400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae91df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.751+0000 7fd637e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd637e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45020/0 (socket says 172.21.15.22:45020) 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd637e00640 1 -- 172.21.15.22:0/867326896 learned_addr learned my addr 172.21.15.22:0/867326896 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5576ae919f10 con 0x5576ae904d40 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5576ae91a080 con 0x5576ae8fb840 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5576ae98b560 con 0x5576ae8fc160 2024-03-24T06:41:52.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.752+0000 7fd636a00640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.753+0000 7fd641069740 10 client.0 _ll_register_callbacks cb 0x7fd63804fb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:52.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.753+0000 7fd637400640 1 -- 172.21.15.22:0/867326896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 msgr2=0x5576ae9192d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.754+0000 7fd637400640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9192d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.754+0000 7fd637400640 1 -- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 msgr2=0x5576ae919810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:52.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.754+0000 7fd637400640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.754+0000 7fd637400640 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd630136c00 con 0x5576ae8fb840 2024-03-24T06:41:52.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.755+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c005500 con 0x5576ae8fb840 2024-03-24T06:41:52.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.756+0000 7fd637e00640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:41:52.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.758+0000 7fd636a00640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:41:52.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.760+0000 7fd637400640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae91df60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd62c004cf0 tx=0x7fd62c004680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4c67bcf52bec5b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:52.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.760+0000 7fd61d600640 10 client.26096 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:52.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.762+0000 7fd61d600640 1 -- 172.21.15.22:0/867326896 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c028040 con 0x5576ae8fb840 2024-03-24T06:41:52.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.763+0000 7fd61d600640 1 -- 172.21.15.22:0/867326896 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd62c004450 con 0x5576ae8fb840 2024-03-24T06:41:52.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.763+0000 7fd61d600640 1 -- 172.21.15.22:0/867326896 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd62c012040 con 0x5576ae8fb840 2024-03-24T06:41:52.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.763+0000 7fd641069740 10 client.26096 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:52.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.763+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5576ae858660 con 0x5576ae8fb840 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.764+0000 7fd61d600640 1 -- 172.21.15.22:0/867326896 <== mon.2 v2:172.21.15.143:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fd62c004dc0 con 0x5576ae8fb840 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.764+0000 7fd641069740 10 client.26096 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.764+0000 7fd641069740 10 client.26096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.764+0000 7fd641069740 20 client.26096 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.764+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae98cb10 0x5576ae98ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.765+0000 7fd641069740 4 client.26096 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:41:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.765+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5576ae559860 con 0x5576ae98cb10 2024-03-24T06:41:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.765+0000 7fd636a00640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae98cb10 0x5576ae98ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:52.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.767+0000 7fd636a00640 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae98cb10 0x5576ae98ef50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c923af436719e24 server_cookie=7d4509ea8a09a606 in_seq=0 out_seq=0 2024-03-24T06:41:52.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.767+0000 7fd61d600640 10 client.26096 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:52.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.772+0000 7fd61d600640 1 -- 172.21.15.22:0/867326896 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5576ae559860 con 0x5576ae98cb10 2024-03-24T06:41:52.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.772+0000 7fd61d600640 10 client.26096 handle_command_reply: tid=0 2024-03-24T06:41:52.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.773+0000 7fd641069740 1 client.26096 shutdown 2024-03-24T06:41:52.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.773+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 msgr2=0x5576ae91df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.773+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae91df60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd62c004cf0 tx=0x7fd62c004680 comp rx=0 tx=0).stop 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 shutdown_connections 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5576ae904d40 0x5576ae919810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576ae8fc160 0x5576ae9192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5576ae98cb10 0x5576ae98ef50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 --2- 172.21.15.22:0/867326896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5576ae8fb840 0x5576ae91df60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 >> 172.21.15.22:0/867326896 conn(0x5576ae8f5f70 msgr2=0x5576ae8ebf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.774+0000 7fd61d600640 0 client.26096 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:52.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.775+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 shutdown_connections 2024-03-24T06:41:52.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.776+0000 7fd641069740 1 -- 172.21.15.22:0/867326896 wait complete. 2024-03-24T06:41:52.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.776+0000 7fd641069740 20 client.26096 trim_cache size 0 max 16384 2024-03-24T06:41:52.779 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:41:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "a1e7f812-67aa-4337-ad08-517c085caed8", 2024-03-24T06:41:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:41:52.779 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:52.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.780+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 msgr2=0x7fd6200a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.780+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 0x7fd6200a4770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd638064c00 tx=0x7fd638097310 comp rx=0 tx=0).stop 2024-03-24T06:41:52.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.780+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 msgr2=0x7fd63012a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:52.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.780+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d770 tx=0x7fd62400dc30 comp rx=0 tx=0).stop 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 shutdown_connections 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd6200a22b0 0x7fd6200a4770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd63009ace0 0x7fd63012e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6300968e0 0x7fd63012a5d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 --2- 172.21.15.22:0/826853615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd630095f20 0x7fd63012a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:52.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 >> 172.21.15.22:0/826853615 conn(0x7fd63000cb30 msgr2=0x7fd630094a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:52.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.781+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 shutdown_connections 2024-03-24T06:41:52.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:52.782+0000 7fd63ce00640 1 -- 172.21.15.22:0/826853615 wait complete. 2024-03-24T06:41:52.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:41:52.819 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:41:53.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.176+0000 7f6381600640 1 -- 172.21.15.22:0/1131760896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c1088d0 msgr2=0x7f637c10c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.176+0000 7f6381600640 1 --2- 172.21.15.22:0/1131760896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c1088d0 0x7f637c10c910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6368009930 tx=0x7f636802bc40 comp rx=0 tx=0).stop 2024-03-24T06:41:53.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.177+0000 7f6381600640 1 -- 172.21.15.22:0/1131760896 shutdown_connections 2024-03-24T06:41:53.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.177+0000 7f6381600640 1 --2- 172.21.15.22:0/1131760896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c114bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.177+0000 7f6381600640 1 --2- 172.21.15.22:0/1131760896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c1088d0 0x7f637c10c910 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.177+0000 7f6381600640 1 --2- 172.21.15.22:0/1131760896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f637c107f20 0x7f637c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.177+0000 7f6381600640 1 -- 172.21.15.22:0/1131760896 >> 172.21.15.22:0/1131760896 conn(0x7f637c075c50 msgr2=0x7f637c078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.178+0000 7f6381600640 1 -- 172.21.15.22:0/1131760896 shutdown_connections 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.178+0000 7f6381600640 1 -- 172.21.15.22:0/1131760896 wait complete. 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.179+0000 7f6381600640 1 Processor -- start 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.181+0000 7f6381600640 1 -- start start 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.182+0000 7f6381600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f637c107f20 0x7f637c0819f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.182+0000 7f6381600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f6381600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c07aa70 0x7f637c07af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f6381600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f637c07b520 con 0x7f637c107f20 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f6381600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f637c07b690 con 0x7f637c07aa70 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f6381600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f637c07b850 con 0x7f637c10cff0 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f637a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f637a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59780/0 (socket says 172.21.15.22:59780) 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.183+0000 7f637a000640 1 -- 172.21.15.22:0/3533403003 learned_addr learned my addr 172.21.15.22:0/3533403003 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.184+0000 7f637a000640 1 -- 172.21.15.22:0/3533403003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c07aa70 msgr2=0x7f637c07af20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.184+0000 7f637a000640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c07aa70 0x7f637c07af20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.185+0000 7f637a000640 1 -- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f637c107f20 msgr2=0x7f637c0819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.185+0000 7f637a000640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f637c107f20 0x7f637c0819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.185+0000 7f637a000640 1 -- 172.21.15.22:0/3533403003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6368009560 con 0x7f637c10cff0 2024-03-24T06:41:53.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.187+0000 7f637a000640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6368009850 tx=0x7f6368004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.189+0000 7f6373400640 1 -- 172.21.15.22:0/3533403003 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368004df0 con 0x7f637c10cff0 2024-03-24T06:41:53.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.189+0000 7f6373400640 1 -- 172.21.15.22:0/3533403003 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6368005430 con 0x7f637c10cff0 2024-03-24T06:41:53.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.189+0000 7f6373400640 1 -- 172.21.15.22:0/3533403003 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636803e450 con 0x7f637c10cff0 2024-03-24T06:41:53.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.190+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c07f570 con 0x7f637c10cff0 2024-03-24T06:41:53.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.190+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c07fa30 con 0x7f637c10cff0 2024-03-24T06:41:53.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.230+0000 7f6373400640 1 -- 172.21.15.22:0/3533403003 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f63680055a0 con 0x7f637c10cff0 2024-03-24T06:41:53.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.239+0000 7f63843be740 1 Processor -- start 2024-03-24T06:41:53.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.243+0000 7f63843be740 1 -- start start 2024-03-24T06:41:53.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.253+0000 7f6373400640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 0x7f63600a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.254+0000 7f6373400640 1 -- 172.21.15.22:0/3533403003 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f63680f09e0 con 0x7f637c10cff0 2024-03-24T06:41:53.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d337184200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d33718d0f0 0x55d33718f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3370f2790 con 0x55d33718d0f0 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d3370efcc0 con 0x55d337183d80 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f63843be740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d3370eb4a0 con 0x55d337184a60 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f637aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f637aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59784/0 (socket says 172.21.15.22:59784) 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.255+0000 7f637aa00640 1 -- 172.21.15.22:0/3797198879 learned_addr learned my addr 172.21.15.22:0/3797198879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 -- 172.21.15.22:0/3797198879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 msgr2=0x55d337184200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d337184200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 -- 172.21.15.22:0/3797198879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d33718d0f0 msgr2=0x55d33718f4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d33718d0f0 0x55d33718f4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 -- 172.21.15.22:0/3797198879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3370acc20 con 0x55d337184a60 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.256+0000 7f637aa00640 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f636c0097c0 tx=0x7f636c009c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76fd69f31279bb8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.257+0000 7f6367400640 1 -- 172.21.15.22:0/3797198879 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c00bad0 con 0x55d337184a60 2024-03-24T06:41:53.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.257+0000 7f6367400640 1 -- 172.21.15.22:0/3797198879 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636c00bc30 con 0x55d337184a60 2024-03-24T06:41:53.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.257+0000 7f6367400640 1 -- 172.21.15.22:0/3797198879 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0105c0 con 0x55d337184a60 2024-03-24T06:41:53.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.257+0000 7f63843be740 1 -- 172.21.15.22:0/3797198879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 msgr2=0x55d3371837b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.257+0000 7f63843be740 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f636c0097c0 tx=0x7f636c009c80 comp rx=0 tx=0).stop 2024-03-24T06:41:53.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.262+0000 7f637b400640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 0x7f63600a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.266+0000 7f63843be740 1 -- 172.21.15.22:0/3797198879 shutdown_connections 2024-03-24T06:41:53.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.266+0000 7f63843be740 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d33718d0f0 0x55d33718f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.266+0000 7f63843be740 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d337184200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.266+0000 7f63843be740 1 --2- 172.21.15.22:0/3797198879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d3371837b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.266+0000 7f63843be740 1 -- 172.21.15.22:0/3797198879 >> 172.21.15.22:0/3797198879 conn(0x55d33714ab30 msgr2=0x55d33717e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.267+0000 7f63843be740 1 -- 172.21.15.22:0/3797198879 shutdown_connections 2024-03-24T06:41:53.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.267+0000 7f63843be740 1 -- 172.21.15.22:0/3797198879 wait complete. 2024-03-24T06:41:53.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.268+0000 7f63843be740 1 -- start start 2024-03-24T06:41:53.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.268+0000 7f63843be740 1 Processor -- start 2024-03-24T06:41:53.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 0x55d33718a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d33708ff20 con 0x55d337184a60 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d33708fbc0 con 0x55d337183d80 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d336e591c0 con 0x55d33718d0f0 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f63843be740 10 client.0 _ll_register_callbacks cb 0x55d336dfbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f637aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f637aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41352/0 (socket says 172.21.15.22:41352) 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.269+0000 7f637aa00640 1 -- 172.21.15.22:0/3884746839 learned_addr learned my addr 172.21.15.22:0/3884746839 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637b400640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637a000640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 0x55d33718a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 -- 172.21.15.22:0/3884746839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 msgr2=0x55d33718ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 -- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 msgr2=0x55d33718a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 0x55d33718a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 -- 172.21.15.22:0/3884746839 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636c009420 con 0x55d337183d80 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637a000640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 0x55d33718a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:53.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637b400640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 0x7f63600a4900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6374005fd0 tx=0x7f6374005f60 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f637aa00640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f636c013000 tx=0x7f636c00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7204a538d5e5aaad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.270+0000 7f635be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:53.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c011620 con 0x55d337183d80 2024-03-24T06:41:53.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d3370e3160 con 0x55d337183d80 2024-03-24T06:41:53.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f636c021070 con 0x55d337183d80 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f636c011780 con 0x55d337183d80 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f63843be740 10 client.26102 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d3370e9b20 con 0x55d337183d80 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d3370e3160 con 0x55d337183d80 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.271+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f636c01b550 con 0x55d337183d80 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 10 client.26102 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 20 client.26102 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d33719d4d0 0x55d33719f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 4 client.26102 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:41:53.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.272+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d336dea7e0 con 0x55d33719d4d0 2024-03-24T06:41:53.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.273+0000 7f637a000640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d33719d4d0 0x55d33719f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.274+0000 7f637a000640 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d33719d4d0 0x55d33719f930 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8f468a0c8341092 server_cookie=c34782e70aab6492 in_seq=0 out_seq=0 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.274+0000 7f635be00640 10 client.26102 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.274+0000 7f635be00640 1 -- 172.21.15.22:0/3884746839 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d336dea7e0 con 0x55d33719d4d0 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.274+0000 7f635be00640 10 client.26102 handle_command_reply: tid=0 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 client.26102 shutdown 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 msgr2=0x55d33718a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f636c013000 tx=0x7f636c00bfd0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 shutdown_connections 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337184a60 0x55d33718a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d33719d4d0 0x55d33719f930 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 --2- 172.21.15.22:0/3884746839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337183d80 0x55d33718a0a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f635be00640 0 client.26102 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:53.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 >> 172.21.15.22:0/3884746839 conn(0x55d33714ab30 msgr2=0x55d337180520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 shutdown_connections 2024-03-24T06:41:53.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 1 -- 172.21.15.22:0/3884746839 wait complete. 2024-03-24T06:41:53.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.275+0000 7f63843be740 20 client.26102 trim_cache size 0 max 16384 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.315+0000 7f63843be740 1 Processor -- start 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.315+0000 7f63843be740 1 -- start start 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d33718c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 0x55d3371989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d3371846c0 con 0x55d337183d80 2024-03-24T06:41:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d337185cc0 con 0x55d337184a60 2024-03-24T06:41:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f63843be740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d33717fa00 con 0x55d33718d0f0 2024-03-24T06:41:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 0x55d3371989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 0x55d3371989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41354/0 (socket says 172.21.15.22:41354) 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637a000640 1 -- 172.21.15.22:0/896620013 learned_addr learned my addr 172.21.15.22:0/896620013 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637aa00640 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d33718c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637b400640 1 -- 172.21.15.22:0/896620013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 msgr2=0x55d3371989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637b400640 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 0x55d3371989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.316+0000 7f637b400640 1 -- 172.21.15.22:0/896620013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 msgr2=0x55d33718c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.317+0000 7f637b400640 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d33718c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.317+0000 7f637b400640 1 -- 172.21.15.22:0/896620013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3370859a0 con 0x55d33718d0f0 2024-03-24T06:41:53.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.317+0000 7f637b400640 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718a940 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6374007d40 tx=0x7f6374017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e2134ef33b142e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.317+0000 7f6366a00640 1 -- 172.21.15.22:0/896620013 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374025670 con 0x55d33718d0f0 2024-03-24T06:41:53.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.318+0000 7f6366a00640 1 -- 172.21.15.22:0/896620013 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6374025cb0 con 0x55d33718d0f0 2024-03-24T06:41:53.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.318+0000 7f6366a00640 1 -- 172.21.15.22:0/896620013 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637401cc60 con 0x55d33718d0f0 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.318+0000 7f63843be740 1 -- 172.21.15.22:0/896620013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 msgr2=0x55d33718a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.318+0000 7f63843be740 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718a940 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6374007d40 tx=0x7f6374017000 comp rx=0 tx=0).stop 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 -- 172.21.15.22:0/896620013 shutdown_connections 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d33718d0f0 0x55d33718a940 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d337184a60 0x55d3371989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 --2- 172.21.15.22:0/896620013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d33718c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 -- 172.21.15.22:0/896620013 >> 172.21.15.22:0/896620013 conn(0x55d33714ab30 msgr2=0x55d33717e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 -- 172.21.15.22:0/896620013 shutdown_connections 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 -- 172.21.15.22:0/896620013 wait complete. 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.319+0000 7f63843be740 1 -- start start 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 Processor -- start 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d337194ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d337187320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d337195600 con 0x55d337183d80 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d337195770 con 0x55d33718d0f0 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d337187da0 con 0x55d337184a60 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f63843be740 10 client.0 _ll_register_callbacks cb 0x7f636c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f637b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f637aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d337194ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f637b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41358/0 (socket says 172.21.15.22:41358) 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.320+0000 7f637b400640 1 -- 172.21.15.22:0/1116997784 learned_addr learned my addr 172.21.15.22:0/1116997784 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637a000640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d337187320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 -- 172.21.15.22:0/1116997784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 msgr2=0x55d337187320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d337187320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 -- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 msgr2=0x55d337194ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d337194ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 -- 172.21.15.22:0/1116997784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6374017070 con 0x55d33718d0f0 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637a000640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d337187320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f637b400640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6374007a20 tx=0x7f63740256a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c51b265de4464 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.321+0000 7f635be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374025ad0 con 0x55d33718d0f0 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6374030460 con 0x55d33718d0f0 2024-03-24T06:41:53.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636c00bc30 con 0x55d33718d0f0 2024-03-24T06:41:53.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6374030620 con 0x55d33718d0f0 2024-03-24T06:41:53.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f63843be740 10 client.26108 fetch_fsmap learned FSMap version 31 2024-03-24T06:41:53.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.322+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d3370ab480 con 0x55d33718d0f0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636c00bc30 con 0x55d33718d0f0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f637403a540 con 0x55d33718d0f0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 10 client.26108 fetch_fsmap finished waiting for FSMap version 31 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 10 client.26108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 20 client.26108 populate_metadata read hostname 'smithi022' 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d3372265b0 0x55d3372289d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 4 client.26108 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.323+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d336e27720 con 0x55d3372265b0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.324+0000 7f637aa00640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d3372265b0 0x55d3372289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.324+0000 7f635be00640 10 client.26108 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.324+0000 7f637aa00640 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d3372265b0 0x55d3372289d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=937b68fd4868749a server_cookie=7dd6642a953cbec1 in_seq=0 out_seq=0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f635be00640 1 -- 172.21.15.22:0/1116997784 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d336e27720 con 0x55d3372265b0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f635be00640 10 client.26108 handle_command_reply: tid=0 2024-03-24T06:41:53.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 client.26108 shutdown 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 msgr2=0x55d337187860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6374007a20 tx=0x7f63740256a0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 shutdown_connections 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d3372265b0 0x55d3372289d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d33718d0f0 0x55d337187860 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d337184a60 0x55d337187320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 --2- 172.21.15.22:0/1116997784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d337183d80 0x55d337194ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 >> 172.21.15.22:0/1116997784 conn(0x55d33714ab30 msgr2=0x55d33717fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f635be00640 0 client.26108 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 shutdown_connections 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 1 -- 172.21.15.22:0/1116997784 wait complete. 2024-03-24T06:41:53.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.325+0000 7f63843be740 20 client.26108 trim_cache size 0 max 16384 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "a1e7f812-67aa-4337-ad08-517c085caed8": { 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "a1e7f812-67aa-4337-ad08-517c085caed8", 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:41:53.329 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 msgr2=0x7f63600a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 0x7f63600a4900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6374005fd0 tx=0x7f6374005f60 comp rx=0 tx=0).stop 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 msgr2=0x7f637c081f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6368009850 tx=0x7f6368004270 comp rx=0 tx=0).stop 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 shutdown_connections 2024-03-24T06:41:53.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f63600a2440 0x7f63600a4900 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f637c07aa70 0x7f637c07af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f637c10cff0 0x7f637c081f30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 --2- 172.21.15.22:0/3533403003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f637c107f20 0x7f637c0819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.326+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 >> 172.21.15.22:0/3533403003 conn(0x7f637c075c50 msgr2=0x7f637c0765b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.327+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 shutdown_connections 2024-03-24T06:41:53.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:41:53.327+0000 7f6381600640 1 -- 172.21.15.22:0/3533403003 wait complete. 2024-03-24T06:41:53.347 INFO:tasks.cephfs.filesystem:scrub status for tag:a1e7f812-67aa-4337-ad08-517c085caed8 - {'path': '/', 'tag': 'a1e7f812-67aa-4337-ad08-517c085caed8', 'options': 'recursive,force'} 2024-03-24T06:41:53.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 223 MiB/s wr, 106 op/s 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:53 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 223 MiB/s wr, 106 op/s 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:41:53.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:53 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: pgmap v59: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 223 MiB/s wr, 106 op/s 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:41:53.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:41:53.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:41:53.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:53.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:53 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:41:55.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:55 smithi155 ceph-mon[36262]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 195 MiB/s wr, 76 op/s 2024-03-24T06:41:55.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:55 smithi022 ceph-mon[32003]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 195 MiB/s wr, 76 op/s 2024-03-24T06:41:55.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:55 smithi143 ceph-mon[36520]: pgmap v60: 129 pgs: 129 active+clean; 2.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 195 MiB/s wr, 76 op/s 2024-03-24T06:41:57.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:57 smithi155 ceph-mon[36262]: pgmap v61: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 292 MiB/s wr, 114 op/s 2024-03-24T06:41:57.904 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:57 smithi022 ceph-mon[32003]: pgmap v61: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 292 MiB/s wr, 114 op/s 2024-03-24T06:41:57.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:57 smithi143 ceph-mon[36520]: pgmap v61: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 292 MiB/s wr, 114 op/s 2024-03-24T06:41:59.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:41:59 smithi155 ceph-mon[36262]: pgmap v62: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:41:59.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:41:59 smithi022 ceph-mon[32003]: pgmap v62: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:41:59.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:41:59 smithi143 ceph-mon[36520]: pgmap v62: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:42:01.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:01 smithi155 ceph-mon[36262]: pgmap v63: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:42:01.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:01 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:01.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:01 smithi022 ceph-mon[32003]: pgmap v63: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:42:01.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:01 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:01.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:01 smithi143 ceph-mon[36520]: pgmap v63: 129 pgs: 129 active+clean; 3.8 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 180 MiB/s wr, 71 op/s 2024-03-24T06:42:01.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:01 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:02 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:02.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:02 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:02.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:02 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:03 smithi155 ceph-mon[36262]: pgmap v64: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 269 MiB/s wr, 123 op/s 2024-03-24T06:42:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:03 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:03.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:03 smithi022 ceph-mon[32003]: pgmap v64: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 269 MiB/s wr, 123 op/s 2024-03-24T06:42:03.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:03 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:03 smithi143 ceph-mon[36520]: pgmap v64: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 269 MiB/s wr, 123 op/s 2024-03-24T06:42:03.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:03 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:05.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:05 smithi022 ceph-mon[32003]: pgmap v65: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 186 MiB/s wr, 88 op/s 2024-03-24T06:42:05.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:05 smithi143 ceph-mon[36520]: pgmap v65: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 186 MiB/s wr, 88 op/s 2024-03-24T06:42:06.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:05 smithi155 ceph-mon[36262]: pgmap v65: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 186 MiB/s wr, 88 op/s 2024-03-24T06:42:06.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:06 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:06.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:06 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:07.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:06 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:42:07.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:07 smithi022 ceph-mon[32003]: pgmap v66: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 278 MiB/s wr, 124 op/s 2024-03-24T06:42:07.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:07 smithi143 ceph-mon[36520]: pgmap v66: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 278 MiB/s wr, 124 op/s 2024-03-24T06:42:08.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:07 smithi155 ceph-mon[36262]: pgmap v66: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 278 MiB/s wr, 124 op/s 2024-03-24T06:42:09.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:09 smithi022 ceph-mon[32003]: pgmap v67: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:09.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:09 smithi143 ceph-mon[36520]: pgmap v67: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:10.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:09 smithi155 ceph-mon[36262]: pgmap v67: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:11.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:11 smithi022 ceph-mon[32003]: pgmap v68: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:11.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:11 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:42:11.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:11 smithi143 ceph-mon[36520]: pgmap v68: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:11.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:11 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:42:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:11 smithi155 ceph-mon[36262]: pgmap v68: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 181 MiB/s wr, 87 op/s 2024-03-24T06:42:12.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:11 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:42:13.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:13 smithi022 ceph-mon[32003]: pgmap v69: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 278 MiB/s wr, 127 op/s 2024-03-24T06:42:13.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:13 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:42:13.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:13 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:42:13.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:13 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:42:13.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:13 smithi143 ceph-mon[36520]: pgmap v69: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 278 MiB/s wr, 127 op/s 2024-03-24T06:42:13.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:13 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:42:13.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:13 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:42:13.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:13 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:42:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:13 smithi155 ceph-mon[36262]: pgmap v69: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 278 MiB/s wr, 127 op/s 2024-03-24T06:42:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:13 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:42:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:13 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:42:14.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:13 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:42:15.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:15 smithi022 ceph-mon[32003]: pgmap v70: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 75 op/s 2024-03-24T06:42:15.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:15 smithi143 ceph-mon[36520]: pgmap v70: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 75 op/s 2024-03-24T06:42:16.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:15 smithi155 ceph-mon[36262]: pgmap v70: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 189 MiB/s wr, 75 op/s 2024-03-24T06:42:17.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:17 smithi022 ceph-mon[32003]: pgmap v71: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 269 MiB/s wr, 113 op/s 2024-03-24T06:42:17.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:17 smithi143 ceph-mon[36520]: pgmap v71: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 269 MiB/s wr, 113 op/s 2024-03-24T06:42:18.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:17 smithi155 ceph-mon[36262]: pgmap v71: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 269 MiB/s wr, 113 op/s 2024-03-24T06:42:19.900 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:19 smithi022 ceph-mon[32003]: pgmap v72: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:19.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:19 smithi143 ceph-mon[36520]: pgmap v72: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:20.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:19 smithi155 ceph-mon[36262]: pgmap v72: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:21 smithi022 ceph-mon[32003]: pgmap v73: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:21 smithi143 ceph-mon[36520]: pgmap v73: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:22.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:21 smithi155 ceph-mon[36262]: pgmap v73: 129 pgs: 129 active+clean; 8.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 182 MiB/s wr, 78 op/s 2024-03-24T06:42:22.953 INFO:tasks.workunit.client.0.smithi022.stdout:fs setup took 51 secs 2024-03-24T06:42:23.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:42:23.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:42:23.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.647+0000 7fd42a800640 1 -- 172.21.15.22:0/2557666995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42410d050 msgr2=0x7fd424079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.647+0000 7fd42a800640 1 --2- 172.21.15.22:0/2557666995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42410d050 0x7fd424079a20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd4140074e0 tx=0x7fd41402c8d0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.648+0000 7fd42a800640 1 -- 172.21.15.22:0/2557666995 shutdown_connections 2024-03-24T06:42:23.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.648+0000 7fd42a800640 1 --2- 172.21.15.22:0/2557666995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42410d050 0x7fd424079a20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.648+0000 7fd42a800640 1 --2- 172.21.15.22:0/2557666995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd424109330 0x7fd42410c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.648+0000 7fd42a800640 1 --2- 172.21.15.22:0/2557666995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.648+0000 7fd42a800640 1 -- 172.21.15.22:0/2557666995 >> 172.21.15.22:0/2557666995 conn(0x7fd42406c7f0 msgr2=0x7fd42406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.651+0000 7fd42a800640 1 -- 172.21.15.22:0/2557666995 shutdown_connections 2024-03-24T06:42:23.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.651+0000 7fd42a800640 1 -- 172.21.15.22:0/2557666995 wait complete. 2024-03-24T06:42:23.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.652+0000 7fd42a800640 1 Processor -- start 2024-03-24T06:42:23.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 -- start start 2024-03-24T06:42:23.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 0x7fd42407e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 0x7fd42407ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241c40b0 con 0x7fd42407e700 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241c41f0 con 0x7fd42410d050 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd42a800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd4241c43b0 con 0x7fd424108980 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37740/0 (socket says 172.21.15.22:37740) 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.653+0000 7fd423400640 1 -- 172.21.15.22:0/2820433727 learned_addr learned my addr 172.21.15.22:0/2820433727 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:23.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423e00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 0x7fd42407ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 -- 172.21.15.22:0/2820433727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 msgr2=0x7fd42407e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd422a00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 0x7fd42407e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 0x7fd42407e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 -- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 msgr2=0x7fd42407ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 0x7fd42407ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 -- 172.21.15.22:0/2820433727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4140071c0 con 0x7fd424108980 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423e00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 0x7fd42407ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd422a00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 0x7fd42407e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.654+0000 7fd423400640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00ba80 tx=0x7fd41c00bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.655+0000 7fd41be00640 1 -- 172.21.15.22:0/2820433727 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c00cc00 con 0x7fd424108980 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.655+0000 7fd41be00640 1 -- 172.21.15.22:0/2820433727 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd41c00cd60 con 0x7fd424108980 2024-03-24T06:42:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.655+0000 7fd41be00640 1 -- 172.21.15.22:0/2820433727 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0077b0 con 0x7fd424108980 2024-03-24T06:42:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.655+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4241c4690 con 0x7fd424108980 2024-03-24T06:42:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.655+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4241c4b50 con 0x7fd424108980 2024-03-24T06:42:23.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.660+0000 7fd41be00640 1 -- 172.21.15.22:0/2820433727 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd41c007990 con 0x7fd424108980 2024-03-24T06:42:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.663+0000 7fd41be00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 0x7fd4080a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.663+0000 7fd41be00640 1 -- 172.21.15.22:0/2820433727 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd41c0cecb0 con 0x7fd424108980 2024-03-24T06:42:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.664+0000 7fd422a00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 0x7fd4080a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.665+0000 7fd422a00640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 0x7fd4080a47b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd42407f980 tx=0x7fd410005d30 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.671+0000 7fd42d5c7740 1 Processor -- start 2024-03-24T06:42:23.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.672+0000 7fd42d5c7740 1 -- start start 2024-03-24T06:42:23.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.673+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.673+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.673+0000 7fd423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.673+0000 7fd423400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37750/0 (socket says 172.21.15.22:37750) 2024-03-24T06:42:23.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.674+0000 7fd423400640 1 -- 172.21.15.22:0/3533323265 learned_addr learned my addr 172.21.15.22:0/3533323265 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:23.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.675+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397dbc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.675+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560397d1f790 con 0x560397db0d80 2024-03-24T06:42:23.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.675+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560397d1ccc0 con 0x560397dba0f0 2024-03-24T06:42:23.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.675+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560397d184a0 con 0x560397db1a60 2024-03-24T06:42:23.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.676+0000 7fd423400640 1 -- 172.21.15.22:0/3533323265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 msgr2=0x560397dbc4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:42:23.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.676+0000 7fd423400640 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397dbc4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.676+0000 7fd423400640 1 -- 172.21.15.22:0/3533323265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 msgr2=0x560397db1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.676+0000 7fd423400640 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.676+0000 7fd423400640 1 -- 172.21.15.22:0/3533323265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560397cd9c20 con 0x560397db1a60 2024-03-24T06:42:23.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.677+0000 7fd423400640 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00ba50 tx=0x7fd41c09bc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbe1e5bf67e035c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.679+0000 7fd418c00640 1 -- 172.21.15.22:0/3533323265 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0a3bf0 con 0x560397db1a60 2024-03-24T06:42:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.679+0000 7fd418c00640 1 -- 172.21.15.22:0/3533323265 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd41c099750 con 0x560397db1a60 2024-03-24T06:42:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.679+0000 7fd418c00640 1 -- 172.21.15.22:0/3533323265 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c007ea0 con 0x560397db1a60 2024-03-24T06:42:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.680+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 msgr2=0x560397db07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.681+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00ba50 tx=0x7fd41c09bc10 comp rx=0 tx=0).stop 2024-03-24T06:42:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.682+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 shutdown_connections 2024-03-24T06:42:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.683+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397dbc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.683+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.683+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3533323265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db1a60 0x560397db07b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.683+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 >> 172.21.15.22:0/3533323265 conn(0x560397d77b30 msgr2=0x560397dab7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.683+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 shutdown_connections 2024-03-24T06:42:23.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.684+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3533323265 wait complete. 2024-03-24T06:42:23.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.684+0000 7fd42d5c7740 1 -- start start 2024-03-24T06:42:23.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.685+0000 7fd42d5c7740 1 Processor -- start 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.685+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db0d80 0x560397db71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.685+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 0x560397db7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.685+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.686+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560397cbcf20 con 0x560397db1a60 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.686+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560397cbcbc0 con 0x560397dba0f0 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.686+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560397a861c0 con 0x560397db0d80 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.686+0000 7fd42d5c7740 10 client.0 _ll_register_callbacks cb 0x560397a28be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.686+0000 7fd423e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd423e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49526/0 (socket says 172.21.15.22:49526) 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd423e00640 1 -- 172.21.15.22:0/2104204595 learned_addr learned my addr 172.21.15.22:0/2104204595 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:23.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 0x560397db7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 -- 172.21.15.22:0/2104204595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db0d80 msgr2=0x560397db71f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db0d80 0x560397db71f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 -- 172.21.15.22:0/2104204595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 msgr2=0x560397db7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.687+0000 7fd422a00640 1 -- 172.21.15.22:0/2104204595 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4241c4b50 con 0x560397db1a60 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.688+0000 7fd422a00640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 0x560397db7730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd42407fa10 tx=0x7fd41000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56eb8deba9ea6d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.688+0000 7fd423e00640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:23.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.688+0000 7fd40d600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:42:23.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.689+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41000ea80 con 0x560397db1a60 2024-03-24T06:42:23.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.689+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd41000ebe0 con 0x560397db1a60 2024-03-24T06:42:23.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.689+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560397d10160 con 0x560397db1a60 2024-03-24T06:42:23.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.691+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd41000eda0 con 0x560397db1a60 2024-03-24T06:42:23.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.691+0000 7fd42d5c7740 10 client.15900 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:23.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.691+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560397d16b20 con 0x560397db1a60 2024-03-24T06:42:23.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.692+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560397d10160 con 0x560397db1a60 2024-03-24T06:42:23.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.692+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fd41000b050 con 0x560397db1a60 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 10 client.15900 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 20 client.15900 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397dca8f0 0x560397dccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 4 client.15900 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.693+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560397a177e0 con 0x560397dca8f0 2024-03-24T06:42:23.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.694+0000 7fd423400640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397dca8f0 0x560397dccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.696+0000 7fd423400640 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397dca8f0 0x560397dccd30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e592a3b0e62a6023 server_cookie=c1a55a1e96ecff70 in_seq=0 out_seq=0 2024-03-24T06:42:23.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.697+0000 7fd40d600640 10 client.15900 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:23.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.697+0000 7fd40d600640 1 -- 172.21.15.22:0/2104204595 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560397a177e0 con 0x560397dca8f0 2024-03-24T06:42:23.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.697+0000 7fd40d600640 10 client.15900 handle_command_reply: tid=0 2024-03-24T06:42:23.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.697+0000 7fd42d5c7740 1 client.15900 shutdown 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 msgr2=0x560397db7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 0x560397db7730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd42407fa10 tx=0x7fd41000fc00 comp rx=0 tx=0).stop 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 shutdown_connections 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397dca8f0 0x560397dccd30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db1a60 0x560397db7730 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd42407fa10 tx=0x7fd41000fc00 comp rx=0 tx=0).stop 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/2104204595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db0d80 0x560397db71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 >> 172.21.15.22:0/2104204595 conn(0x560397d77b30 msgr2=0x560397dad520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.698+0000 7fd40d600640 0 client.15900 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.699+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 shutdown_connections 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.699+0000 7fd42d5c7740 1 -- 172.21.15.22:0/2104204595 wait complete. 2024-03-24T06:42:23.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.699+0000 7fd42d5c7740 20 client.15900 trim_cache size 0 max 16384 2024-03-24T06:42:23.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.739+0000 7fd42d5c7740 1 Processor -- start 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.739+0000 7fd42d5c7740 1 -- start start 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397dc5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db9560 0x560397dd2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560397db1730 con 0x560397db0d80 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560397a177e0 con 0x560397dba0f0 2024-03-24T06:42:23.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560397d10160 con 0x560397db9560 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd422a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd422a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49538/0 (socket says 172.21.15.22:49538) 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd422a00640 1 -- 172.21.15.22:0/3511489182 learned_addr learned my addr 172.21.15.22:0/3511489182 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.740+0000 7fd423400640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397dc5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd423e00640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db9560 0x560397dd2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 -- 172.21.15.22:0/3511489182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db9560 msgr2=0x560397dd2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db9560 0x560397dd2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 -- 172.21.15.22:0/3511489182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 msgr2=0x560397dc5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397dc5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 -- 172.21.15.22:0/3511489182 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560397cb29a0 con 0x560397dba0f0 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd423400640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397dc5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:23.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.741+0000 7fd422a00640 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd410012040 tx=0x7fd41000f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa81f7cecb630947 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.742+0000 7fd40fe00640 1 -- 172.21.15.22:0/3511489182 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd410025640 con 0x560397dba0f0 2024-03-24T06:42:23.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.742+0000 7fd40fe00640 1 -- 172.21.15.22:0/3511489182 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd41000ece0 con 0x560397dba0f0 2024-03-24T06:42:23.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.742+0000 7fd40fe00640 1 -- 172.21.15.22:0/3511489182 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41001ca60 con 0x560397dba0f0 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3511489182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 msgr2=0x560397db9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd410012040 tx=0x7fd41000f420 comp rx=0 tx=0).stop 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3511489182 shutdown_connections 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397db9560 0x560397dd2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397dba0f0 0x560397db9020 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/3511489182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397dc5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3511489182 >> 172.21.15.22:0/3511489182 conn(0x560397d77b30 msgr2=0x560397db8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3511489182 shutdown_connections 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.743+0000 7fd42d5c7740 1 -- 172.21.15.22:0/3511489182 wait complete. 2024-03-24T06:42:23.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.744+0000 7fd42d5c7740 1 -- start start 2024-03-24T06:42:23.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.744+0000 7fd42d5c7740 1 Processor -- start 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 0x560397db4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560397db5320 con 0x560397db0d80 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560397dc2530 con 0x560397db9560 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560397dc26f0 con 0x560397dba0f0 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd42d5c7740 10 client.0 _ll_register_callbacks cb 0x7fd410013070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd423e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd423e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37768/0 (socket says 172.21.15.22:37768) 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd423e00640 1 -- 172.21.15.22:0/1513205240 learned_addr learned my addr 172.21.15.22:0/1513205240 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.745+0000 7fd422a00640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 0x560397db4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 -- 172.21.15.22:0/1513205240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 msgr2=0x560397db4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 0x560397db4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 -- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 msgr2=0x560397db44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 -- 172.21.15.22:0/1513205240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560397d16b20 con 0x560397dba0f0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd422a00640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 0x560397db4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd423e00640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd41400d090 tx=0x7fd414004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83a3b977553dd3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.746+0000 7fd40d600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.747+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414036040 con 0x560397dba0f0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.747+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd41400a9b0 con 0x560397dba0f0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.747+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd41c099750 con 0x560397dba0f0 2024-03-24T06:42:23.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.747+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd414007a50 con 0x560397dba0f0 2024-03-24T06:42:23.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.748+0000 7fd42d5c7740 10 client.26152 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:23.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.748+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560397cd8480 con 0x560397dba0f0 2024-03-24T06:42:23.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.748+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd41c099750 con 0x560397dba0f0 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.749+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fd414003940 con 0x560397dba0f0 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 10 client.26152 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 20 client.26152 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397e31680 0x560397e33aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 4 client.26152 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560397a54720 con 0x560397e31680 2024-03-24T06:42:23.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.750+0000 7fd423400640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397e31680 0x560397e33aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:23.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.751+0000 7fd423400640 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397e31680 0x560397e33aa0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3dc75b18df7439d server_cookie=cf6ba5920adaf774 in_seq=0 out_seq=0 2024-03-24T06:42:23.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.751+0000 7fd40d600640 10 client.26152 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:23.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.752+0000 7fd40d600640 1 -- 172.21.15.22:0/1513205240 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560397a54720 con 0x560397e31680 2024-03-24T06:42:23.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.752+0000 7fd40d600640 10 client.26152 handle_command_reply: tid=0 2024-03-24T06:42:23.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.752+0000 7fd42d5c7740 1 client.26152 shutdown 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.752+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 msgr2=0x560397dbad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.752+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd41400d090 tx=0x7fd414004be0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 shutdown_connections 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560397dba0f0 0x560397dbad20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x560397e31680 0x560397e33aa0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd40d600640 0 client.26152 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560397db9560 0x560397db4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 --2- 172.21.15.22:0/1513205240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560397db0d80 0x560397db44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.753+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 >> 172.21.15.22:0/1513205240 conn(0x560397d77b30 msgr2=0x560397da9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.754+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 shutdown_connections 2024-03-24T06:42:23.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.754+0000 7fd42d5c7740 1 -- 172.21.15.22:0/1513205240 wait complete. 2024-03-24T06:42:23.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.754+0000 7fd42d5c7740 20 client.26152 trim_cache size 0 max 16384 2024-03-24T06:42:23.755 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:42:23.755 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:42:23.755 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:42:23.755 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.755+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 msgr2=0x7fd4080a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.755+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 0x7fd4080a47b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd42407f980 tx=0x7fd410005d30 comp rx=0 tx=0).stop 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.755+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 msgr2=0x7fd424084150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.755+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00ba80 tx=0x7fd41c00bf40 comp rx=0 tx=0).stop 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.755+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 shutdown_connections 2024-03-24T06:42:23.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd4080a22f0 0x7fd4080a47b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd42407e700 0x7fd42407ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd42410d050 0x7fd42407e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 --2- 172.21.15.22:0/2820433727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd424108980 0x7fd424084150 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 >> 172.21.15.22:0/2820433727 conn(0x7fd42406c7f0 msgr2=0x7fd4240711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 shutdown_connections 2024-03-24T06:42:23.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:23.756+0000 7fd42a800640 1 -- 172.21.15.22:0/2820433727 wait complete. 2024-03-24T06:42:23.778 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:23 smithi022 ceph-mon[32003]: pgmap v74: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 275 MiB/s wr, 117 op/s 2024-03-24T06:42:23.782 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:42:23.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:42:23.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:23 smithi143 ceph-mon[36520]: pgmap v74: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 275 MiB/s wr, 117 op/s 2024-03-24T06:42:24.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:23 smithi155 ceph-mon[36262]: pgmap v74: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 275 MiB/s wr, 117 op/s 2024-03-24T06:42:24.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 -- 172.21.15.22:0/761868916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a4101ee0 msgr2=0x7ff6a410e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/761868916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a4101ee0 0x7ff6a410e3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009cd0 tx=0x7ff69c02c0f0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 -- 172.21.15.22:0/761868916 shutdown_connections 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/761868916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a4101ee0 0x7ff6a410e3a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/761868916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 0x7ff6a41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/761868916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a41075e0 0x7ff6a41079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.053+0000 7ff6a8a00640 1 -- 172.21.15.22:0/761868916 >> 172.21.15.22:0/761868916 conn(0x7ff6a40fcff0 msgr2=0x7ff6a40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.054+0000 7ff6a8a00640 1 -- 172.21.15.22:0/761868916 shutdown_connections 2024-03-24T06:42:24.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.054+0000 7ff6a8a00640 1 -- 172.21.15.22:0/761868916 wait complete. 2024-03-24T06:42:24.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.055+0000 7ff6a8a00640 1 Processor -- start 2024-03-24T06:42:24.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.055+0000 7ff6a8a00640 1 -- start start 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 0x7ff6a419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a41075e0 0x7ff6a4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a419a0f0 con 0x7ff6a4101520 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a419a260 con 0x7ff6a4101ee0 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a419a420 con 0x7ff6a41075e0 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49560/0 (socket says 172.21.15.22:49560) 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a1e00640 1 -- 172.21.15.22:0/246438300 learned_addr learned my addr 172.21.15.22:0/246438300 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a3200640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a41075e0 0x7ff6a4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.056+0000 7ff6a2800640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 0x7ff6a419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 -- 172.21.15.22:0/246438300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a41075e0 msgr2=0x7ff6a4199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a41075e0 0x7ff6a4199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 -- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 msgr2=0x7ff6a419f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 0x7ff6a419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 -- 172.21.15.22:0/246438300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff69c009900 con 0x7ff6a4101ee0 2024-03-24T06:42:24.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a1e00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69000c890 tx=0x7ff69000cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690004220 con 0x7ff6a4101ee0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff690007e90 con 0x7ff6a4101ee0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690004a00 con 0x7ff6a4101ee0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.057+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a419a700 con 0x7ff6a4101ee0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.058+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41a6990 con 0x7ff6a4101ee0 2024-03-24T06:42:24.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.059+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a4102c40 con 0x7ff6a4101ee0 2024-03-24T06:42:24.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.065+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7ff6900079b0 con 0x7ff6a4101ee0 2024-03-24T06:42:24.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.067+0000 7ff69b400640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 0x7ff6880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.067+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff6900ce910 con 0x7ff6a4101ee0 2024-03-24T06:42:24.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.068+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff6900fc320 con 0x7ff6a4101ee0 2024-03-24T06:42:24.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.068+0000 7ff6a2800640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 0x7ff6880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.071+0000 7ff6a2800640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 0x7ff6880a4770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6a419abe0 tx=0x7ff694005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.443+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6a4068420 con 0x7ff6a4101ee0 2024-03-24T06:42:24.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.444+0000 7ff69b400640 1 -- 172.21.15.22:0/246438300 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11953 (secure 0 0 0) 0x7ff690097260 con 0x7ff6a4101ee0 2024-03-24T06:42:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:42:24.444 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:42:05.168986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:42:24.445 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 33 2024-03-24T06:42:24.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.447+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 msgr2=0x7ff6880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 0x7ff6880a4770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6a419abe0 tx=0x7ff694005f90 comp rx=0 tx=0).stop 2024-03-24T06:42:24.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 msgr2=0x7ff6a419fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff69000c890 tx=0x7ff69000cd50 comp rx=0 tx=0).stop 2024-03-24T06:42:24.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 shutdown_connections 2024-03-24T06:42:24.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff6880a22b0 0x7ff6880a4770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a41075e0 0x7ff6a4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a4101ee0 0x7ff6a419fe30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 --2- 172.21.15.22:0/246438300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff6a4101520 0x7ff6a419f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.448+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 >> 172.21.15.22:0/246438300 conn(0x7ff6a40fcff0 msgr2=0x7ff6a40fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.449+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 shutdown_connections 2024-03-24T06:42:24.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.449+0000 7ff6a8a00640 1 -- 172.21.15.22:0/246438300 wait complete. 2024-03-24T06:42:24.478 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:42:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 -- 172.21.15.22:0/3269182588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 msgr2=0x7f6b60102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3269182588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b60102b20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009900 tx=0x7f6b5002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 -- 172.21.15.22:0/3269182588 shutdown_connections 2024-03-24T06:42:24.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3269182588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6010bb60 0x7f6b6010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3269182588 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b60103060 0x7f6b6010b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3269182588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b60102b20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 -- 172.21.15.22:0/3269182588 >> 172.21.15.22:0/3269182588 conn(0x7f6b600fbf60 msgr2=0x7f6b600fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 -- 172.21.15.22:0/3269182588 shutdown_connections 2024-03-24T06:42:24.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.717+0000 7f6b5f200640 1 -- 172.21.15.22:0/3269182588 wait complete. 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.718+0000 7f6b5f200640 1 Processor -- start 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 -- start start 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b60103060 0x7f6b6019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b6010bb60 0x7f6b601979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6b60198030 con 0x7f6b6010bb60 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601981a0 con 0x7f6b60103060 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5f200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6b60198360 con 0x7f6b60102740 2024-03-24T06:42:24.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37808/0 (socket says 172.21.15.22:37808) 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5de00640 1 -- 172.21.15.22:0/3120973648 learned_addr learned my addr 172.21.15.22:0/3120973648 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b57e00640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b60103060 0x7f6b6019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.719+0000 7f6b5e800640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b6010bb60 0x7f6b601979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 -- 172.21.15.22:0/3120973648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b60103060 msgr2=0x7f6b6019dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b60103060 0x7f6b6019dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 -- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b6010bb60 msgr2=0x7f6b601979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b6010bb60 0x7f6b601979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 -- 172.21.15.22:0/3120973648 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b48009490 con 0x7f6b60102740 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5de00640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6b50005940 tx=0x7f6b5002c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b56a00640 1 -- 172.21.15.22:0/3120973648 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50004350 con 0x7f6b60102740 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b56a00640 1 -- 172.21.15.22:0/3120973648 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b500044b0 con 0x7f6b60102740 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b56a00640 1 -- 172.21.15.22:0/3120973648 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50003d30 con 0x7f6b60102740 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b50009560 con 0x7f6b60102740 2024-03-24T06:42:24.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.720+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b601a46f0 con 0x7f6b60102740 2024-03-24T06:42:24.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.723+0000 7f6b56a00640 1 -- 172.21.15.22:0/3120973648 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f6b50004df0 con 0x7f6b60102740 2024-03-24T06:42:24.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 Processor -- start 2024-03-24T06:42:24.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 -- start start 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62df450 0x55d9e62df830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e62e8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d9e624d6e0 con 0x55d9e62dfd70 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d9e624ac10 con 0x55d9e62df450 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.724+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d9e62463f0 con 0x55d9e62e8950 2024-03-24T06:42:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37814/0 (socket says 172.21.15.22:37814) 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 -- 172.21.15.22:0/535241232 learned_addr learned my addr 172.21.15.22:0/535241232 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b5e800640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62df450 0x55d9e62df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b5de00640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e62e8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 -- 172.21.15.22:0/535241232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62df450 msgr2=0x55d9e62df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62df450 0x55d9e62df830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 -- 172.21.15.22:0/535241232 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 msgr2=0x55d9e62e8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e62e8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b57e00640 1 -- 172.21.15.22:0/535241232 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e6207e40 con 0x55d9e62e8950 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b5de00640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e62e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.725+0000 7f6b56a00640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 0x7f6b440a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b57e00640 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009de0 tx=0x7f6b48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=914f2ea885c616cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b56a00640 1 -- 172.21.15.22:0/3120973648 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6b500f09e0 con 0x7f6b60102740 2024-03-24T06:42:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b4f400640 1 -- 172.21.15.22:0/535241232 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48004230 con 0x55d9e62e8950 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b4f400640 1 -- 172.21.15.22:0/535241232 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b48004390 con 0x55d9e62e8950 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b4f400640 1 -- 172.21.15.22:0/535241232 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480198d0 con 0x55d9e62e8950 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.726+0000 7f6b5e800640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 0x7f6b440a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 -- 172.21.15.22:0/535241232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 msgr2=0x55d9e62ead40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b5e800640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 0x7f6b440a4770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b58005fd0 tx=0x7f6b58009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009de0 tx=0x7f6b48011810 comp rx=0 tx=0).stop 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 -- 172.21.15.22:0/535241232 shutdown_connections 2024-03-24T06:42:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62ead40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e62e8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/535241232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62df450 0x55d9e62df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 -- 172.21.15.22:0/535241232 >> 172.21.15.22:0/535241232 conn(0x55d9e62d9b80 msgr2=0x55d9e62d9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 -- 172.21.15.22:0/535241232 shutdown_connections 2024-03-24T06:42:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.727+0000 7f6b67bc8740 1 -- 172.21.15.22:0/535241232 wait complete. 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.728+0000 7f6b67bc8740 1 -- start start 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.728+0000 7f6b67bc8740 1 Processor -- start 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62df450 0x55d9e62e1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62e1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d9e61eb060 con 0x55d9e62df450 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d9e61ead00 con 0x55d9e62dfd70 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d9e5fb41c0 con 0x55d9e62e8950 2024-03-24T06:42:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b67bc8740 10 client.0 _ll_register_callbacks cb 0x55d9e5f56be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b5e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b5e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37818/0 (socket says 172.21.15.22:37818) 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.729+0000 7f6b5e800640 1 -- 172.21.15.22:0/2198582159 learned_addr learned my addr 172.21.15.22:0/2198582159 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 -- 172.21.15.22:0/2198582159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 msgr2=0x55d9e62e1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62e1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 -- 172.21.15.22:0/2198582159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62df450 msgr2=0x55d9e62e1900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62df450 0x55d9e62e1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62df450 0x55d9e62e1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 -- 172.21.15.22:0/2198582159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b48009a10 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6b580099c0 tx=0x7f6b58004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f59b4cee77461682 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.730+0000 7f6b43e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58016070 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d9e5f457e0 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6b58004b20 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6b58004ce0 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b67bc8740 10 client.26170 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d9e6244a70 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.731+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d9e5f457e0 con 0x55d9e62e8950 2024-03-24T06:42:24.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f6b58005520 con 0x55d9e62e8950 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 10 client.26170 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 10 client.26170 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 20 client.26170 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e6301430 0x55d9e6303890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 4 client.26170 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.732+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9e5f82720 con 0x55d9e6301430 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.733+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e6301430 0x55d9e6303890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.733+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e6301430 0x55d9e6303890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=284f30fd2cf64cc7 server_cookie=38e3cfab051f4615 in_seq=0 out_seq=0 2024-03-24T06:42:24.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.733+0000 7f6b43e00640 10 client.26170 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b43e00640 1 -- 172.21.15.22:0/2198582159 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9e5f82720 con 0x55d9e6301430 2024-03-24T06:42:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b43e00640 10 client.26170 handle_command_reply: tid=0 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 client.26170 shutdown 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 msgr2=0x55d9e62e2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6b580099c0 tx=0x7f6b58004ab0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 shutdown_connections 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e6301430 0x55d9e6303890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62e8950 0x55d9e62e2380 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62e1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2198582159 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62df450 0x55d9e62e1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b43e00640 0 client.26170 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.735+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 >> 172.21.15.22:0/2198582159 conn(0x55d9e62d9b80 msgr2=0x55d9e62db8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.736+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 shutdown_connections 2024-03-24T06:42:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.736+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2198582159 wait complete. 2024-03-24T06:42:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.736+0000 7f6b67bc8740 20 client.26170 trim_cache size 0 max 16384 2024-03-24T06:42:24.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.768+0000 7f6b67bc8740 1 Processor -- start 2024-03-24T06:42:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.768+0000 7f6b67bc8740 1 -- start start 2024-03-24T06:42:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e63091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62e8950 0x55d9e62dafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d9e5f82750 con 0x55d9e62dfd70 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d9e5f45810 con 0x55d9e62e8950 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d9e62df0f0 con 0x55d9e62df450 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37828/0 (socket says 172.21.15.22:37828) 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 -- 172.21.15.22:0/2865193134 learned_addr learned my addr 172.21.15.22:0/2865193134 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b57e00640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e63091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62e8950 0x55d9e62dafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 -- 172.21.15.22:0/2865193134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62e8950 msgr2=0x55d9e62dafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62e8950 0x55d9e62dafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.769+0000 7f6b5de00640 1 -- 172.21.15.22:0/2865193134 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 msgr2=0x55d9e63091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e63091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b5de00640 1 -- 172.21.15.22:0/2865193134 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9e61e0b00 con 0x55d9e62df450 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b57e00640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e63091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6b500bd0b0 tx=0x7f6b500f2d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89dedfbd66ecfe3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b4ea00640 1 -- 172.21.15.22:0/2865193134 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b500bd120 con 0x55d9e62df450 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b4ea00640 1 -- 172.21.15.22:0/2865193134 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b500c1270 con 0x55d9e62df450 2024-03-24T06:42:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.770+0000 7f6b4ea00640 1 -- 172.21.15.22:0/2865193134 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b500b9790 con 0x55d9e62df450 2024-03-24T06:42:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2865193134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 msgr2=0x55d9e62e7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6b500bd0b0 tx=0x7f6b500f2d70 comp rx=0 tx=0).stop 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2865193134 shutdown_connections 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62e8950 0x55d9e62dafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62dfd70 0x55d9e63091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2865193134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62e7c80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.771+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2865193134 >> 172.21.15.22:0/2865193134 conn(0x55d9e62d9b80 msgr2=0x55d9e62e0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.772+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2865193134 shutdown_connections 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.772+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2865193134 wait complete. 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.772+0000 7f6b67bc8740 1 -- start start 2024-03-24T06:42:24.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.772+0000 7f6b67bc8740 1 Processor -- start 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 0x55d9e635f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d9e635fd40 con 0x55d9e62e8950 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d9e635feb0 con 0x55d9e62dfd70 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d9e6360070 con 0x55d9e62df450 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b67bc8740 10 client.0 _ll_register_callbacks cb 0x7f6b58009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.773+0000 7f6b5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37832/0 (socket says 172.21.15.22:37832) 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 -- 172.21.15.22:0/2313659921 learned_addr learned my addr 172.21.15.22:0/2313659921 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:24.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 0x55d9e635f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 -- 172.21.15.22:0/2313659921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 msgr2=0x55d9e62f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b57e00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62f8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 -- 172.21.15.22:0/2313659921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 msgr2=0x55d9e635f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 0x55d9e635f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 -- 172.21.15.22:0/2313659921 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4801e040 con 0x55d9e62df450 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5e800640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 0x55d9e635f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.774+0000 7f6b5de00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6b500bce20 tx=0x7f6b500c1160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d877391c94b38644 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.775+0000 7f6b43e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:24.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.775+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b500ee060 con 0x55d9e62df450 2024-03-24T06:42:24.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.775+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6b5002ce40 con 0x55d9e62df450 2024-03-24T06:42:24.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.776+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b48004390 con 0x55d9e62df450 2024-03-24T06:42:24.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.776+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6b500ee1c0 con 0x55d9e62df450 2024-03-24T06:42:24.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.776+0000 7f6b67bc8740 10 client.26176 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:24.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.776+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x7f6b601a46f0 con 0x55d9e62df450 2024-03-24T06:42:24.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b48004390 con 0x55d9e62df450 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f6b5003c070 con 0x55d9e62df450 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 10 client.26176 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 10 client.26176 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 20 client.26176 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e63674d0 0x55d9e63698c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 4 client.26176 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.777+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9e5fd1890 con 0x55d9e63674d0 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.778+0000 7f6b57e00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e63674d0 0x55d9e63698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.778+0000 7f6b57e00640 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e63674d0 0x55d9e63698c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b7408a68ff97a81 server_cookie=27931711b5534b73 in_seq=0 out_seq=0 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.778+0000 7f6b43e00640 10 client.26176 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:24.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b43e00640 1 -- 172.21.15.22:0/2313659921 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9e5fd1890 con 0x55d9e63674d0 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b43e00640 10 client.26176 handle_command_reply: tid=0 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 client.26176 shutdown 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 msgr2=0x55d9e62f8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6b500bce20 tx=0x7f6b500c1160 comp rx=0 tx=0).stop 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 shutdown_connections 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55d9e63674d0 0x55d9e63698c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d9e62e8950 0x55d9e635f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d9e62dfd70 0x55d9e62f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 --2- 172.21.15.22:0/2313659921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9e62df450 0x55d9e62f8970 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.779+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 >> 172.21.15.22:0/2313659921 conn(0x55d9e62d9b80 msgr2=0x55d9e62e18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.780+0000 7f6b43e00640 0 client.26176 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.780+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 shutdown_connections 2024-03-24T06:42:24.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.780+0000 7f6b67bc8740 1 -- 172.21.15.22:0/2313659921 wait complete. 2024-03-24T06:42:24.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.780+0000 7f6b67bc8740 20 client.26176 trim_cache size 0 max 16384 2024-03-24T06:42:24.784 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:42:24.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 msgr2=0x7f6b440a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 0x7f6b440a4770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6b58005fd0 tx=0x7f6b58009500 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 msgr2=0x7f6b6019d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6b50005940 tx=0x7f6b5002c710 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 shutdown_connections 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6b440a22b0 0x7f6b440a4770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6b6010bb60 0x7f6b601979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b60103060 0x7f6b6019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 --2- 172.21.15.22:0/3120973648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b60102740 0x7f6b6019d830 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 >> 172.21.15.22:0/3120973648 conn(0x7f6b600fbf60 msgr2=0x7f6b600fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 shutdown_connections 2024-03-24T06:42:24.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:24.782+0000 7f6b5f200640 1 -- 172.21.15.22:0/3120973648 wait complete. 2024-03-24T06:42:24.809 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:24 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/246438300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:42:24.822 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:42:24.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:24 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/246438300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:42:25.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:24 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/246438300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 -- 172.21.15.22:0/611303344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3008107f20 msgr2=0x7f3008108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 --2- 172.21.15.22:0/611303344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3008107f20 0x7f3008108300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8009900 tx=0x7f2ff802bc50 comp rx=0 tx=0).stop 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 -- 172.21.15.22:0/611303344 shutdown_connections 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 --2- 172.21.15.22:0/611303344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300810cff0 0x7f3008114bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 --2- 172.21.15.22:0/611303344 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30081088d0 0x7f300810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 --2- 172.21.15.22:0/611303344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3008107f20 0x7f3008108300 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 -- 172.21.15.22:0/611303344 >> 172.21.15.22:0/611303344 conn(0x7f3008075c50 msgr2=0x7f3008078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 -- 172.21.15.22:0/611303344 shutdown_connections 2024-03-24T06:42:25.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.095+0000 7f300cc00640 1 -- 172.21.15.22:0/611303344 wait complete. 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 Processor -- start 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 -- start start 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f300810cff0 0x7f300819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 0x7f30081a3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a3640 con 0x7f300810cff0 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a37b0 con 0x7f3008107f20 2024-03-24T06:42:25.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.096+0000 7f300cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a3920 con 0x7f30081a2c50 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59912/0 (socket says 172.21.15.22:59912) 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 -- 172.21.15.22:0/3009855814 learned_addr learned my addr 172.21.15.22:0/3009855814 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005400640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f300810cff0 0x7f300819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3006800640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 0x7f30081a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 -- 172.21.15.22:0/3009855814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 msgr2=0x7f30081a3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 0x7f30081a3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 -- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f300810cff0 msgr2=0x7f300819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f300810cff0 0x7f300819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3005e00640 1 -- 172.21.15.22:0/3009855814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff8009560 con 0x7f3008107f20 2024-03-24T06:42:25.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.097+0000 7f3006800640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 0x7f30081a3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:25.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f3005e00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8005ab0 tx=0x7f2ff80056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f2ffea00640 1 -- 172.21.15.22:0/3009855814 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8032d70 con 0x7f3008107f20 2024-03-24T06:42:25.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f2ffea00640 1 -- 172.21.15.22:0/3009855814 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff8004510 con 0x7f3008107f20 2024-03-24T06:42:25.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30081aaf00 con 0x7f3008107f20 2024-03-24T06:42:25.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f2ffea00640 1 -- 172.21.15.22:0/3009855814 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff802c850 con 0x7f3008107f20 2024-03-24T06:42:25.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.098+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30081ab270 con 0x7f3008107f20 2024-03-24T06:42:25.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.103+0000 7f300fa18740 1 Processor -- start 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.103+0000 7f300fa18740 1 -- start start 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.103+0000 7f2ffea00640 1 -- 172.21.15.22:0/3009855814 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f2ff803b040 con 0x7f3008107f20 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.103+0000 7f300fa18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.103+0000 7f300fa18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723ba0 0x563adc72c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f300fa18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f300fa18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563adc6916e0 con 0x563adc72c780 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f300fa18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563adc68ec10 con 0x563adc723ba0 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f300fa18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563adc68a3f0 con 0x563adc723280 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46818/0 (socket says 172.21.15.22:46818) 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 -- 172.21.15.22:0/1006283767 learned_addr learned my addr 172.21.15.22:0/1006283767 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3005e00640 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723ba0 0x563adc72c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 -- 172.21.15.22:0/1006283767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723ba0 msgr2=0x563adc72c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723ba0 0x563adc72c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 -- 172.21.15.22:0/1006283767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 msgr2=0x563adc72eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.104+0000 7f3006800640 1 -- 172.21.15.22:0/1006283767 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563adc64be40 con 0x563adc723280 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f2ffea00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 0x7f2fec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f3006800640 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3000009f10 tx=0x7f300000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9589aa2aac18b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f2ffea00640 1 -- 172.21.15.22:0/3009855814 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2ff80efe80 con 0x7f3008107f20 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f3005e00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 0x7f2fec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f2ff3000640 1 -- 172.21.15.22:0/1006283767 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000003b70 con 0x563adc723280 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f2ff3000640 1 -- 172.21.15.22:0/1006283767 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3000003cd0 con 0x563adc723280 2024-03-24T06:42:25.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.105+0000 7f2ff3000640 1 -- 172.21.15.22:0/1006283767 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000003b70 con 0x563adc723280 2024-03-24T06:42:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 -- 172.21.15.22:0/1006283767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 msgr2=0x563adc723660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3000009f10 tx=0x7f300000f730 comp rx=0 tx=0).stop 2024-03-24T06:42:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f3005e00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 0x7f2fec0a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80ed430 tx=0x7f2ff80f3990 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 -- 172.21.15.22:0/1006283767 shutdown_connections 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723ba0 0x563adc72c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 --2- 172.21.15.22:0/1006283767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc723660 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.106+0000 7f300fa18740 1 -- 172.21.15.22:0/1006283767 >> 172.21.15.22:0/1006283767 conn(0x563adc71d9b0 msgr2=0x563adc71dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.107+0000 7f300fa18740 1 -- 172.21.15.22:0/1006283767 shutdown_connections 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.107+0000 7f300fa18740 1 -- 172.21.15.22:0/1006283767 wait complete. 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.107+0000 7f300fa18740 1 -- start start 2024-03-24T06:42:25.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 Processor -- start 2024-03-24T06:42:25.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc729cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563adc62f060 con 0x563adc72c780 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563adc62ed00 con 0x563adc723280 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563adc3f81c0 con 0x563adc723ba0 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f300fa18740 10 client.0 _ll_register_callbacks cb 0x563adc39abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.108+0000 7f3006800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59926/0 (socket says 172.21.15.22:59926) 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 -- 172.21.15.22:0/3113926055 learned_addr learned my addr 172.21.15.22:0/3113926055 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3005400640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc729cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 -- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 msgr2=0x563adc729cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc729cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 -- 172.21.15.22:0/3113926055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 msgr2=0x563adc72a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 -- 172.21.15.22:0/3113926055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3000009bc0 con 0x563adc723280 2024-03-24T06:42:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.109+0000 7f3006800640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f300000fc30 tx=0x7f3000009aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b072e8b54e4ee726 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f2fe7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000004350 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f30000044b0 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563adc3897e0 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f30000116f0 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f300fa18740 10 client.26147 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.110+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563adc688a70 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.111+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3000011d30 con 0x563adc723280 2024-03-24T06:42:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.111+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f3000028030 con 0x563adc723280 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 10 client.26147 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 10 client.26147 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 20 client.26147 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc73ca30 0x563adc73ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 4 client.26147 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.113+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563adc3897e0 con 0x563adc73ca30 2024-03-24T06:42:25.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.114+0000 7f3005400640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc73ca30 0x563adc73ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.115+0000 7f3005400640 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc73ca30 0x563adc73ee70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=94bccc269b0d1abd server_cookie=aeb9a7820264b1fe in_seq=0 out_seq=0 2024-03-24T06:42:25.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.115+0000 7f2fe7400640 10 client.26147 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:42:25.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.116+0000 7f2fe7400640 1 -- 172.21.15.22:0/3113926055 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563adc3897e0 con 0x563adc73ca30 2024-03-24T06:42:25.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.116+0000 7f2fe7400640 10 client.26147 handle_command_reply: tid=0 2024-03-24T06:42:25.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 client.26147 shutdown 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 msgr2=0x563adc729770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f300000fc30 tx=0x7f3000009aa0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 shutdown_connections 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc73ca30 0x563adc73ee70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc72a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc729cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 --2- 172.21.15.22:0/3113926055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc729770 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 >> 172.21.15.22:0/3113926055 conn(0x563adc71d9b0 msgr2=0x563adc71f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.117+0000 7f2fe7400640 0 client.26147 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:42:25.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.118+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 shutdown_connections 2024-03-24T06:42:25.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.118+0000 7f300fa18740 1 -- 172.21.15.22:0/3113926055 wait complete. 2024-03-24T06:42:25.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.118+0000 7f300fa18740 20 client.26147 trim_cache size 0 max 16384 2024-03-24T06:42:25.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.134+0000 7f300fa18740 1 Processor -- start 2024-03-24T06:42:25.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.134+0000 7f300fa18740 1 -- start start 2024-03-24T06:42:25.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.134+0000 7f300fa18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f300fa18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc723ba0 0x563adc71ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f300fa18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc72c780 0x563adc72a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f300fa18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563adc3897e0 con 0x563adc723ba0 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f300fa18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563adc739640 con 0x563adc72c780 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f300fa18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563adc722cf0 con 0x563adc723280 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46832/0 (socket says 172.21.15.22:46832) 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 -- 172.21.15.22:0/1124702771 learned_addr learned my addr 172.21.15.22:0/1124702771 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc72c780 0x563adc72a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3005400640 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc723ba0 0x563adc71ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 -- 172.21.15.22:0/1124702771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc72c780 msgr2=0x563adc72a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc72c780 0x563adc72a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 -- 172.21.15.22:0/1124702771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc723ba0 msgr2=0x563adc71ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc723ba0 0x563adc71ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.135+0000 7f3006800640 1 -- 172.21.15.22:0/1124702771 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563adc624b00 con 0x563adc723280 2024-03-24T06:42:25.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.136+0000 7f3006800640 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3000005d70 tx=0x7f3000000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77bde6b2c9f40bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.136+0000 7f2ff2600640 1 -- 172.21.15.22:0/1124702771 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000003a20 con 0x563adc723280 2024-03-24T06:42:25.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.136+0000 7f2ff2600640 1 -- 172.21.15.22:0/1124702771 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300001bc20 con 0x563adc723280 2024-03-24T06:42:25.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.136+0000 7f2ff2600640 1 -- 172.21.15.22:0/1124702771 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300001b740 con 0x563adc723280 2024-03-24T06:42:25.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.137+0000 7f300fa18740 1 -- 172.21.15.22:0/1124702771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 msgr2=0x563adc72bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.137+0000 7f300fa18740 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3000005d70 tx=0x7f3000000f40 comp rx=0 tx=0).stop 2024-03-24T06:42:25.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 -- 172.21.15.22:0/1124702771 shutdown_connections 2024-03-24T06:42:25.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc72c780 0x563adc72a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc723ba0 0x563adc71ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 --2- 172.21.15.22:0/1124702771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723280 0x563adc72bea0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 -- 172.21.15.22:0/1124702771 >> 172.21.15.22:0/1124702771 conn(0x563adc71d9b0 msgr2=0x563adc71ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 -- 172.21.15.22:0/1124702771 shutdown_connections 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 -- 172.21.15.22:0/1124702771 wait complete. 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 -- start start 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.138+0000 7f300fa18740 1 Processor -- start 2024-03-24T06:42:25.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc730870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc79bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563adc79c330 con 0x563adc72c780 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563adc79c4a0 con 0x563adc723280 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563adc79c660 con 0x563adc723ba0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f300fa18740 10 client.0 _ll_register_callbacks cb 0x7f3000015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f3005400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f3006800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc79bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f3005400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46842/0 (socket says 172.21.15.22:46842) 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.139+0000 7f3005400640 1 -- 172.21.15.22:0/2360188587 learned_addr learned my addr 172.21.15.22:0/2360188587 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3006800640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc730870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 -- 172.21.15.22:0/2360188587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 msgr2=0x563adc730870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc730870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 -- 172.21.15.22:0/2360188587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 msgr2=0x563adc79bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc79bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 -- 172.21.15.22:0/2360188587 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563adc688a70 con 0x563adc723ba0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f3005400640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009cd0 tx=0x7f2ff4004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae8888b85179a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.140+0000 7f2fe7400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:25.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4017040 con 0x563adc723ba0 2024-03-24T06:42:25.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2ff4003d20 con 0x563adc723ba0 2024-03-24T06:42:25.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3000003cd0 con 0x563adc723ba0 2024-03-24T06:42:25.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2ff401de60 con 0x563adc723ba0 2024-03-24T06:42:25.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f300fa18740 10 client.26191 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.142+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563adc64a6a0 con 0x563adc723ba0 2024-03-24T06:42:25.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3000003cd0 con 0x563adc723ba0 2024-03-24T06:42:25.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f2ff4026ae0 con 0x563adc723ba0 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 10 client.26191 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 10 client.26191 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 20 client.26191 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc7a4020 0x563adc7a6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 4 client.26191 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.143+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563adc3c6720 con 0x563adc7a4020 2024-03-24T06:42:25.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.144+0000 7f3006800640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc7a4020 0x563adc7a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.145+0000 7f3006800640 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc7a4020 0x563adc7a6440 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=df4bc0ce4f92dcba server_cookie=5cbcff98056e0fa9 in_seq=0 out_seq=0 2024-03-24T06:42:25.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.145+0000 7f2fe7400640 10 client.26191 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f2fe7400640 1 -- 172.21.15.22:0/2360188587 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563adc3c6720 con 0x563adc7a4020 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f2fe7400640 10 client.26191 handle_command_reply: tid=0 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 client.26191 shutdown 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 msgr2=0x563adc730db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009cd0 tx=0x7f2ff4004410 comp rx=0 tx=0).stop 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 shutdown_connections 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x563adc7a4020 0x563adc7a6440 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563adc72c780 0x563adc79bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563adc723ba0 0x563adc730db0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 --2- 172.21.15.22:0/2360188587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563adc723280 0x563adc730870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f2fe7400640 0 client.26191 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:42:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 >> 172.21.15.22:0/2360188587 conn(0x563adc71d9b0 msgr2=0x563adc71c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 shutdown_connections 2024-03-24T06:42:25.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.146+0000 7f300fa18740 1 -- 172.21.15.22:0/2360188587 wait complete. 2024-03-24T06:42:25.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300fa18740 20 client.26191 trim_cache size 0 max 16384 2024-03-24T06:42:25.148 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:42:25.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 msgr2=0x7f2fec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 0x7f2fec0a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80ed430 tx=0x7f2ff80f3990 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 msgr2=0x7f300819e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8005ab0 tx=0x7f2ff80056d0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 shutdown_connections 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2fec0a2230 0x7f2fec0a46f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f30081a2c50 0x7f30081a3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f300810cff0 0x7f300819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 --2- 172.21.15.22:0/3009855814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008107f20 0x7f300819e390 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.147+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 >> 172.21.15.22:0/3009855814 conn(0x7f3008075c50 msgr2=0x7f3008077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.148+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 shutdown_connections 2024-03-24T06:42:25.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.148+0000 7f300cc00640 1 -- 172.21.15.22:0/3009855814 wait complete. 2024-03-24T06:42:25.163 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.391+0000 7f4d26a00640 1 -- 172.21.15.22:0/2445627927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20107690 msgr2=0x7f4d20107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.391+0000 7f4d26a00640 1 --2- 172.21.15.22:0/2445627927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20107690 0x7f4d20107a70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c009900 tx=0x7f4d0c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 -- 172.21.15.22:0/2445627927 shutdown_connections 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 --2- 172.21.15.22:0/2445627927 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20102040 0x7f4d2010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 --2- 172.21.15.22:0/2445627927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20101680 0x7f4d20101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 --2- 172.21.15.22:0/2445627927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20107690 0x7f4d20107a70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 -- 172.21.15.22:0/2445627927 >> 172.21.15.22:0/2445627927 conn(0x7f4d200fd170 msgr2=0x7f4d200ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 -- 172.21.15.22:0/2445627927 shutdown_connections 2024-03-24T06:42:25.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.392+0000 7f4d26a00640 1 -- 172.21.15.22:0/2445627927 wait complete. 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 Processor -- start 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 -- start start 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20102040 0x7f4d2019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 0x7f4d2019e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2019ee10 con 0x7f4d20107690 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2019ef80 con 0x7f4d20101680 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.393+0000 7f4d26a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2019f140 con 0x7f4d20102040 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59972/0 (socket says 172.21.15.22:59972) 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1037643130 learned_addr learned my addr 172.21.15.22:0/1037643130 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1f400640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20102040 0x7f4d2019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d24a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 0x7f4d2019e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1037643130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20102040 msgr2=0x7f4d2019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20102040 0x7f4d2019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 msgr2=0x7f4d2019e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 0x7f4d2019e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1037643130 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c009560 con 0x7f4d20101680 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.394+0000 7f4d24a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 0x7f4d2019e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.395+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c000c00 tx=0x7f4d0c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.395+0000 7f4d1cc00640 1 -- 172.21.15.22:0/1037643130 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c02c690 con 0x7f4d20101680 2024-03-24T06:42:25.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.395+0000 7f4d1cc00640 1 -- 172.21.15.22:0/1037643130 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0c0044e0 con 0x7f4d20101680 2024-03-24T06:42:25.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.395+0000 7f4d1cc00640 1 -- 172.21.15.22:0/1037643130 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c03e450 con 0x7f4d20101680 2024-03-24T06:42:25.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.396+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2019f3c0 con 0x7f4d20101680 2024-03-24T06:42:25.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.396+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201a6ca0 con 0x7f4d20101680 2024-03-24T06:42:25.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.400+0000 7f4d29803740 1 Processor -- start 2024-03-24T06:42:25.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 -- start start 2024-03-24T06:42:25.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d1cc00640 1 -- 172.21.15.22:0/1037643130 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f4d0c0035f0 con 0x7f4d20101680 2024-03-24T06:42:25.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460280 0x564a48460660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48469240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4846bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564a483ce6e0 con 0x564a48460280 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.403+0000 7f4d1f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4846bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.403+0000 7f4d1f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4846bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46866/0 (socket says 172.21.15.22:46866) 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.403+0000 7f4d1f400640 1 -- 172.21.15.22:0/3735789155 learned_addr learned my addr 172.21.15.22:0/3735789155 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.403+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48469240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.402+0000 7f4d29803740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a483cbc10 con 0x564a48460ba0 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a483c73f0 con 0x564a48469780 2024-03-24T06:42:25.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d1fe00640 1 -- 172.21.15.22:0/3735789155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 msgr2=0x564a4846bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4846bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d1fe00640 1 -- 172.21.15.22:0/3735789155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460280 msgr2=0x564a48460660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460280 0x564a48460660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.404+0000 7f4d1fe00640 1 -- 172.21.15.22:0/3735789155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a48388e40 con 0x564a48460ba0 2024-03-24T06:42:25.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.405+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48469240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0edcc0 tx=0x7f4d0c0f6b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ea211e04a3482c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.407+0000 7f4d1cc00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 0x7f4d040a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.407+0000 7f4d15200640 1 -- 172.21.15.22:0/3735789155 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c0b9200 con 0x564a48460ba0 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.407+0000 7f4d15200640 1 -- 172.21.15.22:0/3735789155 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0c0f5050 con 0x564a48460ba0 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.407+0000 7f4d15200640 1 -- 172.21.15.22:0/3735789155 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c043020 con 0x564a48460ba0 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.407+0000 7f4d1cc00640 1 -- 172.21.15.22:0/1037643130 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4d0c0f07c0 con 0x7f4d20101680 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.408+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 msgr2=0x564a48469240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.408+0000 7f4d29803740 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48469240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0edcc0 tx=0x7f4d0c0f6b40 comp rx=0 tx=0).stop 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 shutdown_connections 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d24a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 0x7f4d040a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d29803740 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4846bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d29803740 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48469240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d29803740 1 --2- 172.21.15.22:0/3735789155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460280 0x564a48460660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.409+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 >> 172.21.15.22:0/3735789155 conn(0x564a4845a9b0 msgr2=0x564a4845ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.410+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 shutdown_connections 2024-03-24T06:42:25.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.410+0000 7f4d29803740 1 -- 172.21.15.22:0/3735789155 wait complete. 2024-03-24T06:42:25.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.411+0000 7f4d24a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 0x7f4d040a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005fd0 tx=0x7f4d18008300 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.411+0000 7f4d29803740 1 -- start start 2024-03-24T06:42:25.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.411+0000 7f4d29803740 1 Processor -- start 2024-03-24T06:42:25.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 0x564a48466810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48466d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a48467290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564a4836c060 con 0x564a48469780 2024-03-24T06:42:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a4836bd00 con 0x564a48460ba0 2024-03-24T06:42:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a481351c0 con 0x564a48460280 2024-03-24T06:42:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.412+0000 7f4d29803740 10 client.0 _ll_register_callbacks cb 0x564a480d7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.413+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 0x564a48466810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.413+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 0x564a48466810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46874/0 (socket says 172.21.15.22:46874) 2024-03-24T06:42:25.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.413+0000 7f4d1fe00640 1 -- 172.21.15.22:0/2287908308 learned_addr learned my addr 172.21.15.22:0/2287908308 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.413+0000 7f4d1f400640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48466d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.413+0000 7f4d24a00640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a48467290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 -- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 msgr2=0x564a48466810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 0x564a48466810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 -- 172.21.15.22:0/2287908308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 msgr2=0x564a48467290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a48467290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 -- 172.21.15.22:0/2287908308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201a6ca0 con 0x564a48460ba0 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d1f400640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48466d50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006aa0 tx=0x7f4d10010960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b8024dce5e912ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.414+0000 7f4d09600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.415+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10014640 con 0x564a48460ba0 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.415+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4d100147a0 con 0x564a48460ba0 2024-03-24T06:42:25.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.415+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a480c67e0 con 0x564a48460ba0 2024-03-24T06:42:25.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.416+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4d10014960 con 0x564a48460ba0 2024-03-24T06:42:25.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.416+0000 7f4d29803740 10 client.26165 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.416+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564a483c5a70 con 0x564a48460ba0 2024-03-24T06:42:25.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.416+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564a480c67e0 con 0x564a48460ba0 2024-03-24T06:42:25.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.417+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f4d100123f0 con 0x564a48460ba0 2024-03-24T06:42:25.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.417+0000 7f4d29803740 10 client.26165 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.417+0000 7f4d29803740 10 client.26165 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.418+0000 7f4d29803740 20 client.26165 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.418+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a48493240 0x564a48495660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.418+0000 7f4d29803740 4 client.26165 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.418+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a48103720 con 0x564a48493240 2024-03-24T06:42:25.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.418+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a48493240 0x564a48495660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.420+0000 7f4d09600640 10 client.26165 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:42:25.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.420+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a48493240 0x564a48495660 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c54981d324b1dad9 server_cookie=11d86503e43e931d in_seq=0 out_seq=0 2024-03-24T06:42:25.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.421+0000 7f4d09600640 1 -- 172.21.15.22:0/2287908308 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a48103720 con 0x564a48493240 2024-03-24T06:42:25.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.421+0000 7f4d09600640 10 client.26165 handle_command_reply: tid=0 2024-03-24T06:42:25.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.421+0000 7f4d29803740 1 client.26165 shutdown 2024-03-24T06:42:25.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.422+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 msgr2=0x564a48466d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.422+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48466d50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006aa0 tx=0x7f4d10010960 comp rx=0 tx=0).stop 2024-03-24T06:42:25.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 shutdown_connections 2024-03-24T06:42:25.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a48493240 0x564a48495660 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a48467290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460ba0 0x564a48466d50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 --2- 172.21.15.22:0/2287908308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460280 0x564a48466810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 >> 172.21.15.22:0/2287908308 conn(0x564a4845a9b0 msgr2=0x564a4845c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.423+0000 7f4d09600640 0 client.26165 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:42:25.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.424+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 shutdown_connections 2024-03-24T06:42:25.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.424+0000 7f4d29803740 1 -- 172.21.15.22:0/2287908308 wait complete. 2024-03-24T06:42:25.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.424+0000 7f4d29803740 20 client.26165 trim_cache size 0 max 16384 2024-03-24T06:42:25.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.451+0000 7f4d29803740 1 Processor -- start 2024-03-24T06:42:25.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.451+0000 7f4d29803740 1 -- start start 2024-03-24T06:42:25.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a48485de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 0x564a484670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a4845c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564a48495cc0 con 0x564a48469780 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a4845a4e0 con 0x564a48460280 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a48485de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d29803740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a484873d0 con 0x564a48460ba0 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a48485de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60010/0 (socket says 172.21.15.22:60010) 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.452+0000 7f4d1fe00640 1 -- 172.21.15.22:0/230228867 learned_addr learned my addr 172.21.15.22:0/230228867 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.453+0000 7f4d1f400640 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 0x564a484670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d24a00640 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a4845c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 -- 172.21.15.22:0/230228867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 msgr2=0x564a48485de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a48485de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 -- 172.21.15.22:0/230228867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 msgr2=0x564a4845c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a4845c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 -- 172.21.15.22:0/230228867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a48361b00 con 0x564a48460ba0 2024-03-24T06:42:25.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.455+0000 7f4d1f400640 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 0x564a484670b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006a70 tx=0x7f4d10006d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad44c751c7e310fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.456+0000 7f4d0be00640 1 -- 172.21.15.22:0/230228867 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000bd60 con 0x564a48460ba0 2024-03-24T06:42:25.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.456+0000 7f4d0be00640 1 -- 172.21.15.22:0/230228867 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d10012e50 con 0x564a48460ba0 2024-03-24T06:42:25.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.456+0000 7f4d0be00640 1 -- 172.21.15.22:0/230228867 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10014d20 con 0x564a48460ba0 2024-03-24T06:42:25.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.457+0000 7f4d29803740 1 -- 172.21.15.22:0/230228867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 msgr2=0x564a484670b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.457+0000 7f4d29803740 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 0x564a484670b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4d10006a70 tx=0x7f4d10006d80 comp rx=0 tx=0).stop 2024-03-24T06:42:25.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 -- 172.21.15.22:0/230228867 shutdown_connections 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48469780 0x564a4845c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48460ba0 0x564a484670b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 --2- 172.21.15.22:0/230228867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a48485de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 -- 172.21.15.22:0/230228867 >> 172.21.15.22:0/230228867 conn(0x564a4845a9b0 msgr2=0x564a4845bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 -- 172.21.15.22:0/230228867 shutdown_connections 2024-03-24T06:42:25.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 -- 172.21.15.22:0/230228867 wait complete. 2024-03-24T06:42:25.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.458+0000 7f4d29803740 1 -- start start 2024-03-24T06:42:25.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 Processor -- start 2024-03-24T06:42:25.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460ba0 0x564a4847e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4847e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564a484f8520 con 0x564a48460ba0 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a484f8660 con 0x564a48460280 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.459+0000 7f4d29803740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a484f8820 con 0x564a48469780 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.460+0000 7f4d29803740 10 client.0 _ll_register_callbacks cb 0x7f4d1000b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.460+0000 7f4d24a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4847e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.460+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.460+0000 7f4d24a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4847e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46888/0 (socket says 172.21.15.22:46888) 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.460+0000 7f4d1fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60020/0 (socket says 172.21.15.22:60020) 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1491428511 learned_addr learned my addr 172.21.15.22:0/1491428511 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1f400640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460ba0 0x564a4847e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 msgr2=0x564a4847e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4847e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1491428511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460ba0 msgr2=0x564a4847e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.461+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460ba0 0x564a4847e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.462+0000 7f4d1fe00640 1 -- 172.21.15.22:0/1491428511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a483c5a70 con 0x564a48460280 2024-03-24T06:42:25.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.462+0000 7f4d09600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:25.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.462+0000 7f4d1fe00640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0edd70 tx=0x7f4d0c02bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771fd9273b55be5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.463+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c0bc8c0 con 0x564a48460280 2024-03-24T06:42:25.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.463+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4d0c04d250 con 0x564a48460280 2024-03-24T06:42:25.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.463+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d0c0f5050 con 0x564a48460280 2024-03-24T06:42:25.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.464+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4d0c0bd660 con 0x564a48460280 2024-03-24T06:42:25.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.464+0000 7f4d29803740 10 client.26171 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.464+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564a483876a0 con 0x564a48460280 2024-03-24T06:42:25.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0c0f5050 con 0x564a48460280 2024-03-24T06:42:25.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f4d0c03c070 con 0x564a48460280 2024-03-24T06:42:25.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 10 client.26171 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 10 client.26171 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:42:25.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 20 client.26171 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a484f9d70 0x564a484fc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 4 client.26171 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:42:25.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.465+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a48152890 con 0x564a484f9d70 2024-03-24T06:42:25.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.466+0000 7f4d1f400640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a484f9d70 0x564a484fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.467+0000 7f4d09600640 10 client.26171 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:42:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.467+0000 7f4d1f400640 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a484f9d70 0x564a484fc190 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2645214870046303 server_cookie=78000e09b734f03a in_seq=0 out_seq=0 2024-03-24T06:42:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.467+0000 7f4d09600640 1 -- 172.21.15.22:0/1491428511 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a48152890 con 0x564a484f9d70 2024-03-24T06:42:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.467+0000 7f4d09600640 10 client.26171 handle_command_reply: tid=0 2024-03-24T06:42:25.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.467+0000 7f4d29803740 1 client.26171 shutdown 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 msgr2=0x564a4847de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0edd70 tx=0x7f4d0c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 shutdown_connections 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x564a484f9d70 0x564a484fc190 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a48469780 0x564a4847e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564a48460ba0 0x564a4847e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d09600640 0 client.26171 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 --2- 172.21.15.22:0/1491428511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a48460280 0x564a4847de10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 >> 172.21.15.22:0/1491428511 conn(0x564a4845a9b0 msgr2=0x564a4845c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 shutdown_connections 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 1 -- 172.21.15.22:0/1491428511 wait complete. 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.469+0000 7f4d29803740 20 client.26171 trim_cache size 0 max 16384 2024-03-24T06:42:25.470 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.470+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 msgr2=0x7f4d040a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.470+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 0x7f4d040a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005fd0 tx=0x7f4d18008300 comp rx=0 tx=0).stop 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.470+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 msgr2=0x7f4d20199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.470+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c000c00 tx=0x7f4d0c0042f0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 shutdown_connections 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4d040a22b0 0x7f4d040a4770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d20107690 0x7f4d2019e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d20102040 0x7f4d2019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 --2- 172.21.15.22:0/1037643130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d20101680 0x7f4d20199f50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 >> 172.21.15.22:0/1037643130 conn(0x7f4d200fd170 msgr2=0x7f4d200fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 shutdown_connections 2024-03-24T06:42:25.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.471+0000 7f4d26a00640 1 -- 172.21.15.22:0/1037643130 wait complete. 2024-03-24T06:42:25.494 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:42:25.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.768+0000 7fe582a00640 1 -- 172.21.15.22:0/2067670843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 msgr2=0x7fe57c110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.768+0000 7fe582a00640 1 --2- 172.21.15.22:0/2067670843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c110230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe574009be0 tx=0x7fe57402bf80 comp rx=0 tx=0).stop 2024-03-24T06:42:25.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.770+0000 7fe582a00640 1 -- 172.21.15.22:0/2067670843 shutdown_connections 2024-03-24T06:42:25.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.771+0000 7fe582a00640 1 --2- 172.21.15.22:0/2067670843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c110230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.772+0000 7fe582a00640 1 --2- 172.21.15.22:0/2067670843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.772+0000 7fe582a00640 1 --2- 172.21.15.22:0/2067670843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.772+0000 7fe582a00640 1 -- 172.21.15.22:0/2067670843 >> 172.21.15.22:0/2067670843 conn(0x7fe57c0fd0b0 msgr2=0x7fe57c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.772+0000 7fe582a00640 1 -- 172.21.15.22:0/2067670843 shutdown_connections 2024-03-24T06:42:25.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.772+0000 7fe582a00640 1 -- 172.21.15.22:0/2067670843 wait complete. 2024-03-24T06:42:25.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.774+0000 7fe582a00640 1 Processor -- start 2024-03-24T06:42:25.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.775+0000 7fe582a00640 1 -- start start 2024-03-24T06:42:25.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.775+0000 7fe582a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c072d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.775+0000 7fe582a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c073260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.775+0000 7fe582a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c06be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe581600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe581600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c072d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60044/0 (socket says 172.21.15.22:60044) 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe580c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe580c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46910/0 (socket says 172.21.15.22:46910) 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe582a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c06c610 con 0x7fe57c10de20 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe581600640 1 -- 172.21.15.22:0/4104469417 learned_addr learned my addr 172.21.15.22:0/4104469417 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c06c780 con 0x7fe57c073ae0 2024-03-24T06:42:25.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.776+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c06c940 con 0x7fe57c074400 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 -- 172.21.15.22:0/4104469417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 msgr2=0x7fe57c073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 -- 172.21.15.22:0/4104469417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 msgr2=0x7fe57c072d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c072d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.777+0000 7fe582000640 1 -- 172.21.15.22:0/4104469417 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574009840 con 0x7fe57c10de20 2024-03-24T06:42:25.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.778+0000 7fe582000640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c06be80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe57402c590 tx=0x7fe5740042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.783+0000 7fe57a000640 1 -- 172.21.15.22:0/4104469417 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574004660 con 0x7fe57c10de20 2024-03-24T06:42:25.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.783+0000 7fe57a000640 1 -- 172.21.15.22:0/4104469417 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5740047c0 con 0x7fe57c10de20 2024-03-24T06:42:25.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.783+0000 7fe57a000640 1 -- 172.21.15.22:0/4104469417 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57403e6f0 con 0x7fe57c10de20 2024-03-24T06:42:25.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.784+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c06cbc0 con 0x7fe57c10de20 2024-03-24T06:42:25.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.784+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c070a80 con 0x7fe57c10de20 2024-03-24T06:42:25.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.790+0000 7fe586da3740 1 Processor -- start 2024-03-24T06:42:25.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.791+0000 7fe57a000640 1 -- 172.21.15.22:0/4104469417 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe574034aa0 con 0x7fe57c10de20 2024-03-24T06:42:25.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.791+0000 7fe586da3740 1 -- start start 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.794+0000 7fe57a000640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 0x7fe5640a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.794+0000 7fe57a000640 1 -- 172.21.15.22:0/4104469417 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe5740f08a0 con 0x7fe57c10de20 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a647f950 0x55b1a6481d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b1a63e46e0 con 0x55b1a647f950 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b1a63e1c10 con 0x55b1a6476d70 2024-03-24T06:42:25.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.795+0000 7fe586da3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b1a63dd3f0 con 0x55b1a6476450 2024-03-24T06:42:25.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.796+0000 7fe581600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.796+0000 7fe581600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46926/0 (socket says 172.21.15.22:46926) 2024-03-24T06:42:25.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.796+0000 7fe581600640 1 -- 172.21.15.22:0/1957744458 learned_addr learned my addr 172.21.15.22:0/1957744458 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.796+0000 7fe582000640 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a647f950 0x55b1a6481d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 -- 172.21.15.22:0/1957744458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 msgr2=0x55b1a647f410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647f410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 -- 172.21.15.22:0/1957744458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a647f950 msgr2=0x55b1a6481d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a647f950 0x55b1a6481d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 -- 172.21.15.22:0/1957744458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a639ee40 con 0x55b1a6476450 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe580c00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 0x7fe5640a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.797+0000 7fe581600640 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0096c0 tx=0x7fe56c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae49ab781447308 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.798+0000 7fe56a600640 1 -- 172.21.15.22:0/1957744458 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0077a0 con 0x55b1a6476450 2024-03-24T06:42:25.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.798+0000 7fe580c00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 0x7fe5640a4840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b1a6475f20 tx=0x7fe570009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.798+0000 7fe56a600640 1 -- 172.21.15.22:0/1957744458 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe56c019040 con 0x55b1a6476450 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.798+0000 7fe56a600640 1 -- 172.21.15.22:0/1957744458 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c01de20 con 0x55b1a6476450 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 -- 172.21.15.22:0/1957744458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 msgr2=0x55b1a6476830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0096c0 tx=0x7fe56c009b80 comp rx=0 tx=0).stop 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 -- 172.21.15.22:0/1957744458 shutdown_connections 2024-03-24T06:42:25.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a647f950 0x55b1a6481d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 --2- 172.21.15.22:0/1957744458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a6476450 0x55b1a6476830 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.799+0000 7fe586da3740 1 -- 172.21.15.22:0/1957744458 >> 172.21.15.22:0/1957744458 conn(0x55b1a6470b80 msgr2=0x55b1a6470f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.800+0000 7fe586da3740 1 -- 172.21.15.22:0/1957744458 shutdown_connections 2024-03-24T06:42:25.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.800+0000 7fe586da3740 1 -- 172.21.15.22:0/1957744458 wait complete. 2024-03-24T06:42:25.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.800+0000 7fe586da3740 1 -- start start 2024-03-24T06:42:25.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.801+0000 7fe586da3740 1 Processor -- start 2024-03-24T06:42:25.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.803+0000 7fe586da3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.803+0000 7fe586da3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.803+0000 7fe586da3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.804+0000 7fe580c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.804+0000 7fe580c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60066/0 (socket says 172.21.15.22:60066) 2024-03-24T06:42:25.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.804+0000 7fe581600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.804+0000 7fe581600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43492/0 (socket says 172.21.15.22:43492) 2024-03-24T06:42:25.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.806+0000 7fe586da3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b1a6382060 con 0x55b1a6476450 2024-03-24T06:42:25.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.806+0000 7fe586da3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b1a6381d00 con 0x55b1a6476d70 2024-03-24T06:42:25.806 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.806+0000 7fe586da3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b1a614b1c0 con 0x55b1a647f950 2024-03-24T06:42:25.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.806+0000 7fe586da3740 10 client.0 _ll_register_callbacks cb 0x55b1a60edbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.806+0000 7fe580c00640 1 -- 172.21.15.22:0/3068751488 learned_addr learned my addr 172.21.15.22:0/3068751488 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.808+0000 7fe582000640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe580c00640 1 -- 172.21.15.22:0/3068751488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 msgr2=0x55b1a647d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe580c00640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe580c00640 1 -- 172.21.15.22:0/3068751488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 msgr2=0x55b1a647ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe580c00640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe580c00640 1 -- 172.21.15.22:0/3068751488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe56c009320 con 0x55b1a6476d70 2024-03-24T06:42:25.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.809+0000 7fe581600640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:25.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.810+0000 7fe582000640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:25.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.811+0000 7fe580c00640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe570004570 tx=0x7fe570004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a203152d68bcd961 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.812+0000 7fe55ea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:25.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe570005220 con 0x55b1a6476d70 2024-03-24T06:42:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe570005380 con 0x55b1a6476d70 2024-03-24T06:42:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1a60dc7e0 con 0x55b1a6476d70 2024-03-24T06:42:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe570005540 con 0x55b1a6476d70 2024-03-24T06:42:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe586da3740 10 client.26180 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.813+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b1a63dba70 con 0x55b1a6476d70 2024-03-24T06:42:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b1a60dc7e0 con 0x55b1a6476d70 2024-03-24T06:42:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe570024030 con 0x55b1a6476d70 2024-03-24T06:42:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 10 client.26180 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 10 client.26180 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:42:25.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 20 client.26180 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a6490060 0x55b1a6492480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 4 client.26180 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:25.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.814+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1a6119720 con 0x55b1a6490060 2024-03-24T06:42:25.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.815+0000 7fe581600640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a6490060 0x55b1a6492480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.816+0000 7fe581600640 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a6490060 0x55b1a6492480 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1bb4707357ee71d server_cookie=9ae531c80ebe752 in_seq=0 out_seq=0 2024-03-24T06:42:25.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.816+0000 7fe55ea00640 10 client.26180 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:42:25.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.817+0000 7fe55ea00640 1 -- 172.21.15.22:0/3068751488 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b1a6119720 con 0x55b1a6490060 2024-03-24T06:42:25.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.817+0000 7fe55ea00640 10 client.26180 handle_command_reply: tid=0 2024-03-24T06:42:25.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.817+0000 7fe586da3740 1 client.26180 shutdown 2024-03-24T06:42:25.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 msgr2=0x55b1a647cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe570004570 tx=0x7fe570004730 comp rx=0 tx=0).stop 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 shutdown_connections 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a6490060 0x55b1a6492480 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a647cfb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 --2- 172.21.15.22:0/3068751488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 >> 172.21.15.22:0/3068751488 conn(0x55b1a6470b80 msgr2=0x55b1a64728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe55ea00640 0 client.26180 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 shutdown_connections 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.818+0000 7fe586da3740 1 -- 172.21.15.22:0/3068751488 wait complete. 2024-03-24T06:42:25.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.819+0000 7fe586da3740 20 client.26180 trim_cache size 0 max 16384 2024-03-24T06:42:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.838+0000 7fe586da3740 1 Processor -- start 2024-03-24T06:42:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.838+0000 7fe586da3740 1 -- start start 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a648b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b1a648c9a0 con 0x55b1a6476450 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b1a648b770 con 0x55b1a6476d70 2024-03-24T06:42:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe586da3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b1a64760b0 con 0x55b1a647f950 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe582000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe582000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46956/0 (socket says 172.21.15.22:46956) 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe582000640 1 -- 172.21.15.22:0/230859311 learned_addr learned my addr 172.21.15.22:0/230859311 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.839+0000 7fe581600640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe580c00640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a648b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 -- 172.21.15.22:0/230859311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 msgr2=0x55b1a648b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a648b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 -- 172.21.15.22:0/230859311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 msgr2=0x55b1a647f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 -- 172.21.15.22:0/230859311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1a6377b00 con 0x55b1a647f950 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe580c00640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a648b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe582000640 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5740bcce0 tx=0x7fe574009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0442078b052c482 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe569c00640 1 -- 172.21.15.22:0/230859311 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740bd6e0 con 0x55b1a647f950 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe569c00640 1 -- 172.21.15.22:0/230859311 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5740d4030 con 0x55b1a647f950 2024-03-24T06:42:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.840+0000 7fe569c00640 1 -- 172.21.15.22:0/230859311 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740db520 con 0x55b1a647f950 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 -- 172.21.15.22:0/230859311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 msgr2=0x55b1a647d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5740bcce0 tx=0x7fe574009280 comp rx=0 tx=0).stop 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 -- 172.21.15.22:0/230859311 shutdown_connections 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a647d310 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a648b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 --2- 172.21.15.22:0/230859311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a647f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.841+0000 7fe586da3740 1 -- 172.21.15.22:0/230859311 >> 172.21.15.22:0/230859311 conn(0x55b1a6470b80 msgr2=0x55b1a6477dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.842+0000 7fe586da3740 1 -- 172.21.15.22:0/230859311 shutdown_connections 2024-03-24T06:42:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.842+0000 7fe586da3740 1 -- 172.21.15.22:0/230859311 wait complete. 2024-03-24T06:42:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.842+0000 7fe586da3740 1 -- start start 2024-03-24T06:42:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.842+0000 7fe586da3740 1 Processor -- start 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a6487860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a6478ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b1a647a800 con 0x55b1a6476450 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b1a647a970 con 0x55b1a6476d70 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b1a647ab30 con 0x55b1a647f950 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe586da3740 10 client.0 _ll_register_callbacks cb 0x7fe5700099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe582000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe582000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:46970/0 (socket says 172.21.15.22:46970) 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.843+0000 7fe582000640 1 -- 172.21.15.22:0/1151580482 learned_addr learned my addr 172.21.15.22:0/1151580482 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe580c00640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a6478ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe582000640 1 -- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 msgr2=0x55b1a6478ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe582000640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a6478ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe582000640 1 -- 172.21.15.22:0/1151580482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 msgr2=0x55b1a6487860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe581600640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a6487860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe582000640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a6487860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe582000640 1 -- 172.21.15.22:0/1151580482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c070a80 con 0x55b1a647f950 2024-03-24T06:42:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.844+0000 7fe581600640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a6487860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe582000640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe574004af0 tx=0x7fe5740bb4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5b1bc38d8b408fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe55ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740c1130 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe5740bd6e0 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe56c019040 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe5740c5780 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe586da3740 10 client.26224 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.845+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55b1a639d6a0 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe56c019040 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe57403c070 con 0x55b1a647f950 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 10 client.26224 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 10 client.26224 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:42:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 20 client.26224 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a64f5f20 0x55b1a64f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 4 client.26224 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:42:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1a6168890 con 0x55b1a64f5f20 2024-03-24T06:42:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.846+0000 7fe581600640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a64f5f20 0x55b1a64f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:25.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.847+0000 7fe55ea00640 10 client.26224 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.847+0000 7fe581600640 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a64f5f20 0x55b1a64f8340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c12750f2f3d33d7 server_cookie=e5a79231224312de in_seq=0 out_seq=0 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe55ea00640 1 -- 172.21.15.22:0/1151580482 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1a6168890 con 0x55b1a64f5f20 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe55ea00640 10 client.26224 handle_command_reply: tid=0 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 client.26224 shutdown 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 msgr2=0x55b1a6479220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe574004af0 tx=0x7fe5740bb4f0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 shutdown_connections 2024-03-24T06:42:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b1a647f950 0x55b1a6479220 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55b1a64f5f20 0x55b1a64f8340 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b1a6476d70 0x55b1a6478ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 --2- 172.21.15.22:0/1151580482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b1a6476450 0x55b1a6487860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 >> 172.21.15.22:0/1151580482 conn(0x55b1a6470b80 msgr2=0x55b1a646f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe55ea00640 0 client.26224 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 shutdown_connections 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.848+0000 7fe586da3740 1 -- 172.21.15.22:0/1151580482 wait complete. 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.849+0000 7fe586da3740 20 client.26224 trim_cache size 0 max 16384 2024-03-24T06:42:25.850 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:42:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 msgr2=0x7fe5640a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 0x7fe5640a4840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55b1a6475f20 tx=0x7fe570009260 comp rx=0 tx=0).stop 2024-03-24T06:42:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 msgr2=0x7fe57c06be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c06be80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe57402c590 tx=0x7fe5740042e0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 shutdown_connections 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe5640a2380 0x7fe5640a4840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe57c10de20 0x7fe57c06be80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe57c074400 0x7fe57c073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 --2- 172.21.15.22:0/4104469417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe57c073ae0 0x7fe57c072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 >> 172.21.15.22:0/4104469417 conn(0x7fe57c0fd0b0 msgr2=0x7fe57c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.851+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 shutdown_connections 2024-03-24T06:42:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:25.852+0000 7fe582a00640 1 -- 172.21.15.22:0/4104469417 wait complete. 2024-03-24T06:42:25.874 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:25 smithi022 ceph-mon[32003]: pgmap v75: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 179 MiB/s wr, 77 op/s 2024-03-24T06:42:25.877 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:42:25.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:25 smithi143 ceph-mon[36520]: pgmap v75: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 179 MiB/s wr, 77 op/s 2024-03-24T06:42:26.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:25 smithi155 ceph-mon[36262]: pgmap v75: 129 pgs: 129 active+clean; 9.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 179 MiB/s wr, 77 op/s 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.154+0000 7f56ab200640 1 -- 172.21.15.22:0/346634916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 msgr2=0x7f56a406b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.154+0000 7f56ab200640 1 --2- 172.21.15.22:0/346634916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a406b3a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5694009900 tx=0x7f569402bcc0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 -- 172.21.15.22:0/346634916 shutdown_connections 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 --2- 172.21.15.22:0/346634916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a411d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 --2- 172.21.15.22:0/346634916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 0x7f56a411a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 --2- 172.21.15.22:0/346634916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a406b3a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 -- 172.21.15.22:0/346634916 >> 172.21.15.22:0/346634916 conn(0x7f56a4072ce0 msgr2=0x7f56a40730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 -- 172.21.15.22:0/346634916 shutdown_connections 2024-03-24T06:42:26.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.155+0000 7f56ab200640 1 -- 172.21.15.22:0/346634916 wait complete. 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.156+0000 7f56ab200640 1 Processor -- start 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.156+0000 7f56ab200640 1 -- start start 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 0x7f56a410e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a410e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f56a410edf0 con 0x7f56a411aeb0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f56a410ef60 con 0x7f56a406afc0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56ab200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41bb980 con 0x7f56a406b8e0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60100/0 (socket says 172.21.15.22:60100) 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.157+0000 7f56a3e00640 1 -- 172.21.15.22:0/1214099410 learned_addr learned my addr 172.21.15.22:0/1214099410 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3400640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 0x7f56a410e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a9200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a410e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 -- 172.21.15.22:0/1214099410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 msgr2=0x7f56a410e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 0x7f56a410e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 -- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 msgr2=0x7f56a410e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a410e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 -- 172.21.15.22:0/1214099410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5694009560 con 0x7f56a406afc0 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a9200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.158+0000 7f56a3e00640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5694005940 tx=0x7f569402cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.159+0000 7f56a0c00640 1 -- 172.21.15.22:0/1214099410 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694004250 con 0x7f56a406afc0 2024-03-24T06:42:26.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.159+0000 7f56a0c00640 1 -- 172.21.15.22:0/1214099410 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56940043b0 con 0x7f56a406afc0 2024-03-24T06:42:26.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.159+0000 7f56a0c00640 1 -- 172.21.15.22:0/1214099410 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56940039e0 con 0x7f56a406afc0 2024-03-24T06:42:26.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.159+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a41a6950 con 0x7f56a406afc0 2024-03-24T06:42:26.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.160+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a41a6e10 con 0x7f56a406afc0 2024-03-24T06:42:26.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.165+0000 7f56a0c00640 1 -- 172.21.15.22:0/1214099410 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f5694005360 con 0x7f56a406afc0 2024-03-24T06:42:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.166+0000 7f56a0c00640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 0x7f56880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.166+0000 7f56a0c00640 1 -- 172.21.15.22:0/1214099410 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f5694112720 con 0x7f56a406afc0 2024-03-24T06:42:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.167+0000 7f56a3400640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 0x7f56880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.167+0000 7f56adf83740 1 Processor -- start 2024-03-24T06:42:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.167+0000 7f56a3400640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 0x7f56880a4770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56a410f980 tx=0x7f5690005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 -- start start 2024-03-24T06:42:26.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 0x5616502c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 0x5616502cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5616502316e0 con 0x5616502c3280 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56165022ec10 con 0x5616502c3ba0 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.168+0000 7f56adf83740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56165022a3f0 con 0x5616502cc780 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47010/0 (socket says 172.21.15.22:47010) 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a9200640 1 -- 172.21.15.22:0/730327763 learned_addr learned my addr 172.21.15.22:0/730327763 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 0x5616502cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3e00640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 0x5616502c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 -- 172.21.15.22:0/730327763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 msgr2=0x5616502ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 -- 172.21.15.22:0/730327763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 msgr2=0x5616502c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 0x5616502c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3400640 1 -- 172.21.15.22:0/730327763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616501ebe40 con 0x5616502c3ba0 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.169+0000 7f56a3e00640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 0x5616502c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.170+0000 7f56a3400640 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 0x5616502cc240 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f56a410fa10 tx=0x7f5690014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc8a996bac960fee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.170+0000 7f5699600640 1 -- 172.21.15.22:0/730327763 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569001e420 con 0x5616502c3ba0 2024-03-24T06:42:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.170+0000 7f5699600640 1 -- 172.21.15.22:0/730327763 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569001ea60 con 0x5616502c3ba0 2024-03-24T06:42:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.170+0000 7f5699600640 1 -- 172.21.15.22:0/730327763 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569001dab0 con 0x5616502c3ba0 2024-03-24T06:42:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.171+0000 7f56adf83740 1 -- 172.21.15.22:0/730327763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 msgr2=0x5616502cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.171+0000 7f56adf83740 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 0x5616502cc240 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f56a410fa10 tx=0x7f5690014000 comp rx=0 tx=0).stop 2024-03-24T06:42:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 -- 172.21.15.22:0/730327763 shutdown_connections 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3ba0 0x5616502cc240 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 --2- 172.21.15.22:0/730327763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3280 0x5616502c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 -- 172.21.15.22:0/730327763 >> 172.21.15.22:0/730327763 conn(0x5616502bd9b0 msgr2=0x5616502bdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 -- 172.21.15.22:0/730327763 shutdown_connections 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.172+0000 7f56adf83740 1 -- 172.21.15.22:0/730327763 wait complete. 2024-03-24T06:42:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.173+0000 7f56adf83740 1 -- start start 2024-03-24T06:42:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.173+0000 7f56adf83740 1 Processor -- start 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.173+0000 7f56adf83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3ba0 0x5616502c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502cc780 0x5616502ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5616501cf060 con 0x5616502cc780 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5616501ced00 con 0x5616502c3280 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56164ff981c0 con 0x5616502c3ba0 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56adf83740 10 client.0 _ll_register_callbacks cb 0x56164ff3abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60132/0 (socket says 172.21.15.22:60132) 2024-03-24T06:42:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 -- 172.21.15.22:0/243824539 learned_addr learned my addr 172.21.15.22:0/243824539 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a9200640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502cc780 0x5616502ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3400640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3ba0 0x5616502c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 -- 172.21.15.22:0/243824539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3ba0 msgr2=0x5616502c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3ba0 0x5616502c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 -- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502cc780 msgr2=0x5616502ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502cc780 0x5616502ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.174+0000 7f56a3e00640 1 -- 172.21.15.22:0/243824539 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5690017070 con 0x5616502c3280 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f56a3e00640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56940db320 tx=0x7f56940df680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02f0616ddb8fe18 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f568e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694112540 con 0x5616502c3280 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f56940e7660 con 0x5616502c3280 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56164ff297e0 con 0x5616502c3280 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f56940e71d0 con 0x5616502c3280 2024-03-24T06:42:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f56adf83740 10 client.26195 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:26.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.175+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561650228a70 con 0x5616502c3280 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56940c1070 con 0x5616502c3280 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f56940de310 con 0x5616502c3280 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 10 client.26195 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 20 client.26195 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5616502dc990 0x5616502dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 4 client.26195 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56164ff66720 con 0x5616502dc990 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.176+0000 7f56a3400640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5616502dc990 0x5616502dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.177+0000 7f56a3400640 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5616502dc990 0x5616502dedd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea31fd1cc6fc8219 server_cookie=812c312c73f20ac2 in_seq=0 out_seq=0 2024-03-24T06:42:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.177+0000 7f568e000640 10 client.26195 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.179+0000 7f568e000640 1 -- 172.21.15.22:0/243824539 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56164ff66720 con 0x5616502dc990 2024-03-24T06:42:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.179+0000 7f568e000640 10 client.26195 handle_command_reply: tid=0 2024-03-24T06:42:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.179+0000 7f56adf83740 1 client.26195 shutdown 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 msgr2=0x5616502c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f56940db320 tx=0x7f56940df680 comp rx=0 tx=0).stop 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 shutdown_connections 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502cc780 0x5616502ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3ba0 0x5616502c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5616502dc990 0x5616502dedd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 --2- 172.21.15.22:0/243824539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502c9740 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 >> 172.21.15.22:0/243824539 conn(0x5616502bd9b0 msgr2=0x5616502bf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f568e000640 0 client.26195 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 shutdown_connections 2024-03-24T06:42:26.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 1 -- 172.21.15.22:0/243824539 wait complete. 2024-03-24T06:42:26.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.180+0000 7f56adf83740 20 client.26195 trim_cache size 0 max 16384 2024-03-24T06:42:26.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.200+0000 7f56adf83740 1 Processor -- start 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.201+0000 7f56adf83740 1 -- start start 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3280 0x5616502d8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5616502cf450 con 0x5616502c3ba0 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5616502be9b0 con 0x5616502cc780 2024-03-24T06:42:26.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56adf83740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5616502d9690 con 0x5616502c3280 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56a9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56a9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60144/0 (socket says 172.21.15.22:60144) 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.202+0000 7f56a9200640 1 -- 172.21.15.22:0/3087066724 learned_addr learned my addr 172.21.15.22:0/3087066724 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a3400640 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 -- 172.21.15.22:0/3087066724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3280 msgr2=0x5616502d8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3280 0x5616502d8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 -- 172.21.15.22:0/3087066724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 msgr2=0x5616502e46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502e46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 -- 172.21.15.22:0/3087066724 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616501c4b00 con 0x5616502cc780 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a3400640 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502e46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.203+0000 7f56a9200640 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f569c013460 tx=0x7f569c013920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae043667f511b1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.204+0000 7f5698c00640 1 -- 172.21.15.22:0/3087066724 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c011760 con 0x5616502cc780 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.204+0000 7f5698c00640 1 -- 172.21.15.22:0/3087066724 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569c011da0 con 0x5616502cc780 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.204+0000 7f5698c00640 1 -- 172.21.15.22:0/3087066724 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c00f3c0 con 0x5616502cc780 2024-03-24T06:42:26.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.205+0000 7f56adf83740 1 -- 172.21.15.22:0/3087066724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 msgr2=0x5616502c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.205+0000 7f56adf83740 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f569c013460 tx=0x7f569c013920 comp rx=0 tx=0).stop 2024-03-24T06:42:26.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.207+0000 7f56adf83740 1 -- 172.21.15.22:0/3087066724 shutdown_connections 2024-03-24T06:42:26.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.207+0000 7f56adf83740 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502cc780 0x5616502c9c90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.207+0000 7f56adf83740 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502e46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.207+0000 7f56adf83740 1 --2- 172.21.15.22:0/3087066724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502c3280 0x5616502d8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.207+0000 7f56adf83740 1 -- 172.21.15.22:0/3087066724 >> 172.21.15.22:0/3087066724 conn(0x5616502bd9b0 msgr2=0x5616502bca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.209+0000 7f56adf83740 1 -- 172.21.15.22:0/3087066724 shutdown_connections 2024-03-24T06:42:26.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.209+0000 7f56adf83740 1 -- 172.21.15.22:0/3087066724 wait complete. 2024-03-24T06:42:26.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.209+0000 7f56adf83740 1 -- start start 2024-03-24T06:42:26.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 Processor -- start 2024-03-24T06:42:26.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561650341e30 con 0x5616502c3ba0 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561650341fa0 con 0x5616502c3280 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561650342160 con 0x5616502cc780 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56adf83740 10 client.0 _ll_register_callbacks cb 0x7f5694005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.210+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60156/0 (socket says 172.21.15.22:60156) 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a3e00640 1 -- 172.21.15.22:0/2248654192 learned_addr learned my addr 172.21.15.22:0/2248654192 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:26.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a3400640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a9200640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a9200640 1 -- 172.21.15.22:0/2248654192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 msgr2=0x5616502d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a9200640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.211+0000 7f56a9200640 1 -- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 msgr2=0x5616502d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.212+0000 7f56a9200640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.212+0000 7f56a9200640 1 -- 172.21.15.22:0/2248654192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f569c009be0 con 0x5616502cc780 2024-03-24T06:42:26.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.212+0000 7f56a3400640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.213+0000 7f56a9200640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502d0c80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f569c009370 tx=0x7f569c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4644b9e6fb004073 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.213+0000 7f568e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.214+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c01dea0 con 0x5616502cc780 2024-03-24T06:42:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.214+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f569001ea60 con 0x5616502cc780 2024-03-24T06:42:26.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.214+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f569c004a40 con 0x5616502cc780 2024-03-24T06:42:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.215+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f569c004c00 con 0x5616502cc780 2024-03-24T06:42:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.215+0000 7f56adf83740 10 client.26239 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:26.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.215+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5616501ea6a0 con 0x5616502cc780 2024-03-24T06:42:26.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.216+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569001ea60 con 0x5616502cc780 2024-03-24T06:42:26.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.216+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f569c005470 con 0x5616502cc780 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 10 client.26239 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 20 client.26239 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x561650343860 0x561650345c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 4 client.26239 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:42:26.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.219+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56164ffb5890 con 0x561650343860 2024-03-24T06:42:26.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.220+0000 7f56a3e00640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x561650343860 0x561650345c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:26.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.229+0000 7f56a3e00640 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x561650343860 0x561650345c50 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15aee89845959d7a server_cookie=a109bc3e8829ea2 in_seq=0 out_seq=0 2024-03-24T06:42:26.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.229+0000 7f568e000640 10 client.26239 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:26.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.233+0000 7f568e000640 1 -- 172.21.15.22:0/2248654192 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56164ffb5890 con 0x561650343860 2024-03-24T06:42:26.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.233+0000 7f568e000640 10 client.26239 handle_command_reply: tid=0 2024-03-24T06:42:26.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.233+0000 7f56adf83740 1 client.26239 shutdown 2024-03-24T06:42:26.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 msgr2=0x5616502d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502d0c80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f569c009370 tx=0x7f569c000f40 comp rx=0 tx=0).stop 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 shutdown_connections 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616502cc780 0x5616502d0c80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5616502c3ba0 0x5616502d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x561650343860 0x561650345c50 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 --2- 172.21.15.22:0/2248654192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616502c3280 0x5616502d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f568e000640 0 client.26239 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:26.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.234+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 >> 172.21.15.22:0/2248654192 conn(0x5616502bd9b0 msgr2=0x5616502beaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.236+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 shutdown_connections 2024-03-24T06:42:26.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.236+0000 7f56adf83740 1 -- 172.21.15.22:0/2248654192 wait complete. 2024-03-24T06:42:26.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.237+0000 7f56adf83740 20 client.26239 trim_cache size 0 max 16384 2024-03-24T06:42:26.238 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.240+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 msgr2=0x7f56880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.240+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 0x7f56880a4770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56a410f980 tx=0x7f5690005f90 comp rx=0 tx=0).stop 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 msgr2=0x7f56a4116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5694005940 tx=0x7f569402cb90 comp rx=0 tx=0).stop 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 shutdown_connections 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f56880a22b0 0x7f56880a4770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f56a411aeb0 0x7f56a410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56a406b8e0 0x7f56a410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 --2- 172.21.15.22:0/1214099410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56a406afc0 0x7f56a4116200 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:26.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.241+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 >> 172.21.15.22:0/1214099410 conn(0x7f56a4072ce0 msgr2=0x7f56a411b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:26.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.242+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 shutdown_connections 2024-03-24T06:42:26.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:26.243+0000 7f56ab200640 1 -- 172.21.15.22:0/1214099410 wait complete. 2024-03-24T06:42:27.281 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:42:27.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:42:27.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.580+0000 7f76d2800640 1 -- 172.21.15.22:0/1298534779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc1029a0 msgr2=0x7f76cc102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.580+0000 7f76d2800640 1 --2- 172.21.15.22:0/1298534779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc1029a0 0x7f76cc102d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009900 tx=0x7f76bc02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 -- 172.21.15.22:0/1298534779 shutdown_connections 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 --2- 172.21.15.22:0/1298534779 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc10bd70 0x7f76cc10e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 --2- 172.21.15.22:0/1298534779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc10b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 --2- 172.21.15.22:0/1298534779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc1029a0 0x7f76cc102d80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 -- 172.21.15.22:0/1298534779 >> 172.21.15.22:0/1298534779 conn(0x7f76cc077770 msgr2=0x7f76cc077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 -- 172.21.15.22:0/1298534779 shutdown_connections 2024-03-24T06:42:27.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.581+0000 7f76d2800640 1 -- 172.21.15.22:0/1298534779 wait complete. 2024-03-24T06:42:27.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 Processor -- start 2024-03-24T06:42:27.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 -- start start 2024-03-24T06:42:27.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc1029a0 0x7f76cc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc10bd70 0x7f76cc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc19ee60 con 0x7f76cc1029a0 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc19efd0 con 0x7f76cc10bd70 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.582+0000 7f76d2800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc19f190 con 0x7f76cc1032c0 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76cb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc1029a0 0x7f76cc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc10bd70 0x7f76cc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47068/0 (socket says 172.21.15.22:47068) 2024-03-24T06:42:27.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 -- 172.21.15.22:0/3723031060 learned_addr learned my addr 172.21.15.22:0/3723031060 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 -- 172.21.15.22:0/3723031060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc10bd70 msgr2=0x7f76cc19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc10bd70 0x7f76cc19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 -- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc1029a0 msgr2=0x7f76cc199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc1029a0 0x7f76cc199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 -- 172.21.15.22:0/3723031060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc009560 con 0x7f76cc1032c0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.583+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76b400e9f0 tx=0x7f76b400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.584+0000 7f76c8c00640 1 -- 172.21.15.22:0/3723031060 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4009720 con 0x7f76cc1032c0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.584+0000 7f76c8c00640 1 -- 172.21.15.22:0/3723031060 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b400cd80 con 0x7f76cc1032c0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.584+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc19f470 con 0x7f76cc1032c0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.584+0000 7f76c8c00640 1 -- 172.21.15.22:0/3723031060 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4010690 con 0x7f76cc1032c0 2024-03-24T06:42:27.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.584+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1a6cf0 con 0x7f76cc1032c0 2024-03-24T06:42:27.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.588+0000 7f76d55af740 1 Processor -- start 2024-03-24T06:42:27.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 -- start start 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 0x55fb12dcf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dd8330 0x55fb12dda720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fb12d3d2d0 con 0x55fb12dcf750 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fb12d3a800 con 0x55fb12dd8330 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76d55af740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb12d35fe0 con 0x55fb12dcee30 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76c8c00640 1 -- 172.21.15.22:0/3723031060 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f76b40046c0 con 0x7f76cc1032c0 2024-03-24T06:42:27.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76cb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76cb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43608/0 (socket says 172.21.15.22:43608) 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76cb400640 1 -- 172.21.15.22:0/2997103916 learned_addr learned my addr 172.21.15.22:0/2997103916 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 0x55fb12dcf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.589+0000 7f76c3e00640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dd8330 0x55fb12dda720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 -- 172.21.15.22:0/2997103916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dd8330 msgr2=0x55fb12dda720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dd8330 0x55fb12dda720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 -- 172.21.15.22:0/2997103916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 msgr2=0x55fb12dd7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 -- 172.21.15.22:0/2997103916 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb12cf7e90 con 0x55fb12dcee30 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cb400640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 0x55fb12dcf210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f00 tx=0x7f76c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c6e5b36f094442f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76c0c00640 1 -- 172.21.15.22:0/2997103916 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4004d20 con 0x55fb12dcee30 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76c0c00640 1 -- 172.21.15.22:0/2997103916 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004e80 con 0x55fb12dcee30 2024-03-24T06:42:27.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.590+0000 7f76c0c00640 1 -- 172.21.15.22:0/2997103916 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4005280 con 0x55fb12dcee30 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76c8c00640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 0x7f76b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 -- 172.21.15.22:0/2997103916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 msgr2=0x55fb12dcf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 0x55fb12dcf210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f00 tx=0x7f76c4011810 comp rx=0 tx=0).stop 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76c8c00640 1 -- 172.21.15.22:0/3723031060 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76b4014070 con 0x7f76cc1032c0 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 -- 172.21.15.22:0/2997103916 shutdown_connections 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dd8330 0x55fb12dda720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcf750 0x55fb12dd7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 --2- 172.21.15.22:0/2997103916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcee30 0x55fb12dcf210 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76d55af740 1 -- 172.21.15.22:0/2997103916 >> 172.21.15.22:0/2997103916 conn(0x55fb12dc9560 msgr2=0x55fb12dc9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.591+0000 7f76cb400640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 0x7f76b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.592+0000 7f76d55af740 1 -- 172.21.15.22:0/2997103916 shutdown_connections 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.592+0000 7f76d55af740 1 -- 172.21.15.22:0/2997103916 wait complete. 2024-03-24T06:42:27.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.592+0000 7f76cb400640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 0x7f76b00a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f76bc005ec0 tx=0x7f76bc02e040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.592+0000 7f76d55af740 1 -- start start 2024-03-24T06:42:27.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 Processor -- start 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 0x55fb12dd5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 0x55fb12dd5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fb12cdafc0 con 0x55fb12dd8330 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fb12cdac60 con 0x55fb12dcee30 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.593+0000 7f76d55af740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb12a7e0c0 con 0x55fb12dcf750 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76d55af740 10 client.0 _ll_register_callbacks cb 0x55fb12cf2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60174/0 (socket says 172.21.15.22:60174) 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 -- 172.21.15.22:0/3366274825 learned_addr learned my addr 172.21.15.22:0/3366274825 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 0x55fb12dd5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 -- 172.21.15.22:0/3366274825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 msgr2=0x55fb12dd5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cbe00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 0x55fb12dd5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 0x55fb12dd5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 -- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 msgr2=0x55fb12dd5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 0x55fb12dd5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 -- 172.21.15.22:0/3366274825 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c4009be0 con 0x55fb12dcee30 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 0x55fb12dd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cbe00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 0x55fb12dd5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76cb400640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004540 tx=0x7f76bc02bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f10399c082d8752 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.594+0000 7f76b9600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc02ce00 con 0x55fb12dcee30 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76bc005000 con 0x55fb12dcee30 2024-03-24T06:42:27.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb12d2c210 con 0x55fb12dcee30 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76bc0051c0 con 0x55fb12dcee30 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76d55af740 10 client.26213 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb12d34660 con 0x55fb12dcee30 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb12d2c210 con 0x55fb12dcee30 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.595+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f76bc03c040 con 0x55fb12dcee30 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 10 client.26213 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 20 client.26213 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12de8930 0x55fb12dead90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 4 client.26213 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb12a35860 con 0x55fb12de8930 2024-03-24T06:42:27.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.596+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12de8930 0x55fb12dead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.597+0000 7f76c3e00640 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12de8930 0x55fb12dead90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29962b99ddb91679 server_cookie=43c1ccd0c1d531de in_seq=0 out_seq=0 2024-03-24T06:42:27.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.597+0000 7f76b9600640 10 client.26213 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.597+0000 7f76b9600640 1 -- 172.21.15.22:0/3366274825 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb12a35860 con 0x55fb12de8930 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.597+0000 7f76b9600640 10 client.26213 handle_command_reply: tid=0 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.597+0000 7f76d55af740 1 client.26213 shutdown 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 msgr2=0x55fb12dd5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004540 tx=0x7f76bc02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 shutdown_connections 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dd8330 0x55fb12dd5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dcf750 0x55fb12dd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12de8930 0x55fb12dead90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 --2- 172.21.15.22:0/3366274825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcee30 0x55fb12dd5340 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 >> 172.21.15.22:0/3366274825 conn(0x55fb12dc9560 msgr2=0x55fb12dcb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76b9600640 0 client.26213 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 shutdown_connections 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 1 -- 172.21.15.22:0/3366274825 wait complete. 2024-03-24T06:42:27.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.598+0000 7f76d55af740 20 client.26213 trim_cache size 0 max 16384 2024-03-24T06:42:27.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 Processor -- start 2024-03-24T06:42:27.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 -- start start 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12dd7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12de3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fb12de5250 con 0x55fb12dcee30 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fb12dc4c80 con 0x55fb12dcf750 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.638+0000 7f76d55af740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb12dce0f0 con 0x55fb12dd8330 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47094/0 (socket says 172.21.15.22:47094) 2024-03-24T06:42:27.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 -- 172.21.15.22:0/2763986878 learned_addr learned my addr 172.21.15.22:0/2763986878 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cb400640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12dd7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76c3e00640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12de3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 -- 172.21.15.22:0/2763986878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 msgr2=0x55fb12de3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12de3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 -- 172.21.15.22:0/2763986878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 msgr2=0x55fb12dd7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12dd7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cbe00640 1 -- 172.21.15.22:0/2763986878 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb12cd0ae0 con 0x55fb12dd8330 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76cb400640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12dd7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:27.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.639+0000 7f76c3e00640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12de3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.640+0000 7f76cbe00640 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009730 tx=0x7f76c4003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70542a9b8a55e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.640+0000 7f76bbe00640 1 -- 172.21.15.22:0/2763986878 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4003e20 con 0x55fb12dd8330 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.640+0000 7f76bbe00640 1 -- 172.21.15.22:0/2763986878 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c40054e0 con 0x55fb12dd8330 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.640+0000 7f76bbe00640 1 -- 172.21.15.22:0/2763986878 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4023bc0 con 0x55fb12dd8330 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 -- 172.21.15.22:0/2763986878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 msgr2=0x55fb12df0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009730 tx=0x7f76c4003970 comp rx=0 tx=0).stop 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 -- 172.21.15.22:0/2763986878 shutdown_connections 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12df0660 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12de3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 --2- 172.21.15.22:0/2763986878 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12dd7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.641+0000 7f76d55af740 1 -- 172.21.15.22:0/2763986878 >> 172.21.15.22:0/2763986878 conn(0x55fb12dc9560 msgr2=0x55fb12dd5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.642+0000 7f76d55af740 1 -- 172.21.15.22:0/2763986878 shutdown_connections 2024-03-24T06:42:27.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.642+0000 7f76d55af740 1 -- 172.21.15.22:0/2763986878 wait complete. 2024-03-24T06:42:27.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 -- start start 2024-03-24T06:42:27.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 Processor -- start 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12de0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12ddd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12ddd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fb12de07b0 con 0x55fb12dcee30 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fb12de0920 con 0x55fb12dcf750 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.643+0000 7f76d55af740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb12dddc70 con 0x55fb12dd8330 2024-03-24T06:42:27.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76d55af740 10 client.0 _ll_register_callbacks cb 0x7f76bc044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12ddd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12ddd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47096/0 (socket says 172.21.15.22:47096) 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76cbe00640 1 -- 172.21.15.22:0/4161782025 learned_addr learned my addr 172.21.15.22:0/4161782025 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12ddd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76cb400640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12de0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 -- 172.21.15.22:0/4161782025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 msgr2=0x55fb12ddd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12ddd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 -- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 msgr2=0x55fb12de0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12de0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.644+0000 7f76c3e00640 1 -- 172.21.15.22:0/4161782025 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb12d34660 con 0x55fb12dcf750 2024-03-24T06:42:27.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76c3e00640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12ddd1f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f76b40cbb60 tx=0x7f76b409aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613122bdbcfcaf37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76b9600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:27.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b409d070 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76b409ae00 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb12dd0d10 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.645+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76b409ef50 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 10 client.26216 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb12cf7e90 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb12dd0d10 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f76b40cd070 con 0x55fb12dcf750 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 10 client.26216 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 20 client.26216 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12e60eb0 0x55fb12e632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 4 client.26216 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.646+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb12d5ae90 con 0x55fb12e60eb0 2024-03-24T06:42:27.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.647+0000 7f76cb400640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12e60eb0 0x55fb12e632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.648+0000 7f76cb400640 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12e60eb0 0x55fb12e632a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f9d06689dad0e70 server_cookie=b29020995e5abd39 in_seq=0 out_seq=0 2024-03-24T06:42:27.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.648+0000 7f76b9600640 10 client.26216 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:27.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.649+0000 7f76b9600640 1 -- 172.21.15.22:0/4161782025 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb12d5ae90 con 0x55fb12e60eb0 2024-03-24T06:42:27.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.649+0000 7f76b9600640 10 client.26216 handle_command_reply: tid=0 2024-03-24T06:42:27.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.649+0000 7f76d55af740 1 client.26216 shutdown 2024-03-24T06:42:27.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.649+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 msgr2=0x55fb12ddd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.649+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12ddd1f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f76b40cbb60 tx=0x7f76b409aa60 comp rx=0 tx=0).stop 2024-03-24T06:42:27.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 shutdown_connections 2024-03-24T06:42:27.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb12dd8330 0x55fb12ddd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fb12e60eb0 0x55fb12e632a0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fb12dcf750 0x55fb12ddd1f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 --2- 172.21.15.22:0/4161782025 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fb12dcee30 0x55fb12de0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76b9600640 0 client.26216 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.650+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 >> 172.21.15.22:0/4161782025 conn(0x55fb12dc9560 msgr2=0x55fb12dd5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.651+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 shutdown_connections 2024-03-24T06:42:27.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.651+0000 7f76d55af740 1 -- 172.21.15.22:0/4161782025 wait complete. 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.651+0000 7f76d55af740 20 client.26216 trim_cache size 0 max 16384 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "0ba70511-c0fb-47b8-8ee0-aca8e8f92b51", 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:42:27.652 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:42:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 msgr2=0x7f76b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 0x7f76b00a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f76bc005ec0 tx=0x7f76bc02e040 comp rx=0 tx=0).stop 2024-03-24T06:42:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 msgr2=0x7f76cc19a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76b400e9f0 tx=0x7f76b400edf0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 shutdown_connections 2024-03-24T06:42:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76b00a2230 0x7f76b00a46f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc10bd70 0x7f76cc19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76cc1032c0 0x7f76cc19a4e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 --2- 172.21.15.22:0/3723031060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76cc1029a0 0x7f76cc199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.654 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.653+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 >> 172.21.15.22:0/3723031060 conn(0x7f76cc077770 msgr2=0x7f76cc107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.656+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 shutdown_connections 2024-03-24T06:42:27.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.656+0000 7f76d2800640 1 -- 172.21.15.22:0/3723031060 wait complete. 2024-03-24T06:42:27.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:42:27.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:42:27.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:27 smithi143 ceph-mon[36520]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 265 MiB/s wr, 109 op/s 2024-03-24T06:42:27.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.981+0000 7f1c01600640 1 -- 172.21.15.22:0/944558861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc103790 msgr2=0x7f1bfc103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.981+0000 7f1c01600640 1 --2- 172.21.15.22:0/944558861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc103790 0x7f1bfc103c10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009cd0 tx=0x7f1bec02c0e0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 -- 172.21.15.22:0/944558861 shutdown_connections 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 --2- 172.21.15.22:0/944558861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 --2- 172.21.15.22:0/944558861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc103790 0x7f1bfc103c10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 --2- 172.21.15.22:0/944558861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc1097a0 0x7f1bfc109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 -- 172.21.15.22:0/944558861 >> 172.21.15.22:0/944558861 conn(0x7f1bfc0ff220 msgr2=0x7f1bfc101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 -- 172.21.15.22:0/944558861 shutdown_connections 2024-03-24T06:42:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.982+0000 7f1c01600640 1 -- 172.21.15.22:0/944558861 wait complete. 2024-03-24T06:42:27.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.985+0000 7f1c01600640 1 Processor -- start 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.985+0000 7f1c01600640 1 -- start start 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc103790 0x7f1bfc075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc1097a0 0x7f1bfc0766a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1aa490 con 0x7f1bfc1097a0 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1aa5d0 con 0x7f1bfc104150 2024-03-24T06:42:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1c01600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc1aa710 con 0x7f1bfc103790 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60208/0 (socket says 172.21.15.22:60208) 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.986+0000 7f1bfb400640 1 -- 172.21.15.22:0/448483398 learned_addr learned my addr 172.21.15.22:0/448483398 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1c00c00640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc1097a0 0x7f1bfc0766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc103790 0x7f1bfc075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfb400640 1 -- 172.21.15.22:0/448483398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc103790 msgr2=0x7f1bfc075c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfb400640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc103790 0x7f1bfc075c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfb400640 1 -- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc1097a0 msgr2=0x7f1bfc0766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfb400640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc1097a0 0x7f1bfc0766a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.987+0000 7f1bfb400640 1 -- 172.21.15.22:0/448483398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec009900 con 0x7f1bfc104150 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.988+0000 7f1bfb400640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000f80 tx=0x7f1bec034cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.988+0000 7f1bf8c00640 1 -- 172.21.15.22:0/448483398 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec032ac0 con 0x7f1bfc104150 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.988+0000 7f1bf8c00640 1 -- 172.21.15.22:0/448483398 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bec032eb0 con 0x7f1bfc104150 2024-03-24T06:42:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.988+0000 7f1bf8c00640 1 -- 172.21.15.22:0/448483398 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec040450 con 0x7f1bfc104150 2024-03-24T06:42:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.989+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc1aa960 con 0x7f1bfc104150 2024-03-24T06:42:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.989+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc1aadc0 con 0x7f1bfc104150 2024-03-24T06:42:27.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:27.994+0000 7f1bf8c00640 1 -- 172.21.15.22:0/448483398 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1bec00a040 con 0x7f1bfc104150 2024-03-24T06:42:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.000+0000 7f1bf8c00640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 0x7f1be00a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.000+0000 7f1bf8c00640 1 -- 172.21.15.22:0/448483398 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1bec0f0e60 con 0x7f1bfc104150 2024-03-24T06:42:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.001+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 0x7f1be00a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.002+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 0x7f1be00a47f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4005fd0 tx=0x7f1bf4005850 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.006+0000 7f1c05974740 1 Processor -- start 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.006+0000 7f1c05974740 1 -- start start 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e8b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032e8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e976c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d032dfa790 con 0x55d032e952d0 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d032df7cc0 con 0x55d032e8cc40 2024-03-24T06:42:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.007+0000 7f1c05974740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d032df34a0 con 0x55d032e8bf60 2024-03-24T06:42:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.008+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032e8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.008+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032e8c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47124/0 (socket says 172.21.15.22:47124) 2024-03-24T06:42:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.008+0000 7f1bfb400640 1 -- 172.21.15.22:0/63859802 learned_addr learned my addr 172.21.15.22:0/63859802 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.008+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e8b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.008+0000 7f1c00c00640 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.010+0000 7f1bfbe00640 1 -- 172.21.15.22:0/63859802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 msgr2=0x55d032e8c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.010+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032e8c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.010+0000 7f1bfbe00640 1 -- 172.21.15.22:0/63859802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 msgr2=0x55d032e976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.010+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.010+0000 7f1bfbe00640 1 -- 172.21.15.22:0/63859802 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d032db4c20 con 0x55d032e8cc40 2024-03-24T06:42:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.011+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e8b990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40058c0 tx=0x7f1bf4012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298d415b4b472339 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.011+0000 7f1bf1200640 1 -- 172.21.15.22:0/63859802 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf401e450 con 0x55d032e8cc40 2024-03-24T06:42:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.011+0000 7f1bf1200640 1 -- 172.21.15.22:0/63859802 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bf401ea90 con 0x55d032e8cc40 2024-03-24T06:42:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.011+0000 7f1bf1200640 1 -- 172.21.15.22:0/63859802 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4010a20 con 0x55d032e8cc40 2024-03-24T06:42:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.012+0000 7f1c05974740 1 -- 172.21.15.22:0/63859802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 msgr2=0x55d032e8b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.012+0000 7f1c05974740 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e8b990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40058c0 tx=0x7f1bf4012000 comp rx=0 tx=0).stop 2024-03-24T06:42:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 -- 172.21.15.22:0/63859802 shutdown_connections 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032e8c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 --2- 172.21.15.22:0/63859802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e8b990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 -- 172.21.15.22:0/63859802 >> 172.21.15.22:0/63859802 conn(0x55d032e52b30 msgr2=0x55d032e869b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 -- 172.21.15.22:0/63859802 shutdown_connections 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 -- 172.21.15.22:0/63859802 wait complete. 2024-03-24T06:42:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.013+0000 7f1c05974740 1 -- start start 2024-03-24T06:42:28.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.014+0000 7f1c05974740 1 Processor -- start 2024-03-24T06:42:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d032d97f20 con 0x55d032e952d0 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d032d97bc0 con 0x55d032e8bf60 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d032b611c0 con 0x55d032e8cc40 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.015+0000 7f1c05974740 10 client.0 _ll_register_callbacks cb 0x55d032b03be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.016+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.016+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.016+0000 7f1c00c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.017+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47140/0 (socket says 172.21.15.22:47140) 2024-03-24T06:42:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.017+0000 7f1bfb400640 1 -- 172.21.15.22:0/1095035640 learned_addr learned my addr 172.21.15.22:0/1095035640 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.017+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60234/0 (socket says 172.21.15.22:60234) 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.017+0000 7f1c00c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43674/0 (socket says 172.21.15.22:43674) 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1c00c00640 1 -- 172.21.15.22:0/1095035640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 msgr2=0x55d032e927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1c00c00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1c00c00640 1 -- 172.21.15.22:0/1095035640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 msgr2=0x55d032e922a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1c00c00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1c00c00640 1 -- 172.21.15.22:0/1095035640 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf400fb30 con 0x55d032e952d0 2024-03-24T06:42:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:42:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.018+0000 7f1bfb400640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.019+0000 7f1c00c00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1be400bdc0 tx=0x7f1be4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea2caed68ea087b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:28.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.022+0000 7f1be9600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:42:28.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.022+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40107e0 con 0x55d032e952d0 2024-03-24T06:42:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.022+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1be4010940 con 0x55d032e952d0 2024-03-24T06:42:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.022+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d032deb160 con 0x55d032e952d0 2024-03-24T06:42:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.022+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1be4010b00 con 0x55d032e952d0 2024-03-24T06:42:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.023+0000 7f1c05974740 10 client.15981 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.023+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d032df1b20 con 0x55d032e952d0 2024-03-24T06:42:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d032deb160 con 0x55d032e952d0 2024-03-24T06:42:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f1be401a3e0 con 0x55d032e952d0 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 10 client.15981 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 20 client.15981 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032ec7a30 0x55d032ec9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 4 client.15981 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.024+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d032af27e0 con 0x55d032ec7a30 2024-03-24T06:42:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.025+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032ec7a30 0x55d032ec9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.026+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032ec7a30 0x55d032ec9e50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38cfc80f3f70808e server_cookie=a7c892f623311667 in_seq=0 out_seq=0 2024-03-24T06:42:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.026+0000 7f1be9600640 10 client.15981 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:28.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:27 smithi155 ceph-mon[36262]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 265 MiB/s wr, 109 op/s 2024-03-24T06:42:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.027+0000 7f1be9600640 1 -- 172.21.15.22:0/1095035640 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d032af27e0 con 0x55d032ec7a30 2024-03-24T06:42:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.027+0000 7f1be9600640 10 client.15981 handle_command_reply: tid=0 2024-03-24T06:42:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.027+0000 7f1c05974740 1 client.15981 shutdown 2024-03-24T06:42:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 msgr2=0x55d032e92d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1be400bdc0 tx=0x7f1be4012820 comp rx=0 tx=0).stop 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 shutdown_connections 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032e92d20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.028+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032ec7a30 0x55d032ec9e50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1c05974740 1 --2- 172.21.15.22:0/1095035640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8bf60 0x55d032e922a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 >> 172.21.15.22:0/1095035640 conn(0x55d032e52b30 msgr2=0x55d032e88700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1be9600640 0 client.15981 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 shutdown_connections 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1c05974740 1 -- 172.21.15.22:0/1095035640 wait complete. 2024-03-24T06:42:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.029+0000 7f1c05974740 20 client.15981 trim_cache size 0 max 16384 2024-03-24T06:42:28.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.072+0000 7f1c05974740 1 Processor -- start 2024-03-24T06:42:28.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.072+0000 7f1c05974740 1 -- start start 2024-03-24T06:42:28.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.072+0000 7f1c05974740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.072+0000 7f1c05974740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.073+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.073+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.073+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60248/0 (socket says 172.21.15.22:60248) 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.074+0000 7f1c05974740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032ecf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.074+0000 7f1c05974740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d032ea9030 con 0x55d032e952d0 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.074+0000 7f1c05974740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d032e8af90 con 0x55d032e8cc40 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.074+0000 7f1c05974740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d032e825f0 con 0x55d032e8bf60 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.074+0000 7f1bfb400640 1 -- 172.21.15.22:0/141806452 learned_addr learned my addr 172.21.15.22:0/141806452 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:28.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.073+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47152/0 (socket says 172.21.15.22:47152) 2024-03-24T06:42:28.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1c00c00640 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032ecf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1bfb400640 1 -- 172.21.15.22:0/141806452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 msgr2=0x55d032eb1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1bfb400640 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1bfb400640 1 -- 172.21.15.22:0/141806452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 msgr2=0x55d032ecf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1bfb400640 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032ecf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.075+0000 7f1bfb400640 1 -- 172.21.15.22:0/141806452 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d032d8d9a0 con 0x55d032e8cc40 2024-03-24T06:42:28.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.076+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:28.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.077+0000 7f1bfb400640 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0b9ac0 tx=0x7f1bec0dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af70ba4ce5ee0268 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:28.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.077+0000 7f1bebe00640 1 -- 172.21.15.22:0/141806452 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0c17b0 con 0x55d032e8cc40 2024-03-24T06:42:28.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.077+0000 7f1bebe00640 1 -- 172.21.15.22:0/141806452 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bec0c5e00 con 0x55d032e8cc40 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.077+0000 7f1bebe00640 1 -- 172.21.15.22:0/141806452 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0c17b0 con 0x55d032e8cc40 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.078+0000 7f1c05974740 1 -- 172.21.15.22:0/141806452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 msgr2=0x55d032e85a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.078+0000 7f1c05974740 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0b9ac0 tx=0x7f1bec0dd040 comp rx=0 tx=0).stop 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 -- 172.21.15.22:0/141806452 shutdown_connections 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e952d0 0x55d032ecf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e8cc40 0x55d032e85a50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 --2- 172.21.15.22:0/141806452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8bf60 0x55d032eb1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 -- 172.21.15.22:0/141806452 >> 172.21.15.22:0/141806452 conn(0x55d032e52b30 msgr2=0x55d032e85340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:28.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 -- 172.21.15.22:0/141806452 shutdown_connections 2024-03-24T06:42:28.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.079+0000 7f1c05974740 1 -- 172.21.15.22:0/141806452 wait complete. 2024-03-24T06:42:28.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.080+0000 7f1c05974740 1 -- start start 2024-03-24T06:42:28.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.080+0000 7f1c05974740 1 Processor -- start 2024-03-24T06:42:28.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.081+0000 7f1c05974740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 0x55d032eae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.081+0000 7f1c05974740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e8f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.082+0000 7f1c05974740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.082+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e8f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.082+0000 7f1bfb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e8f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:47168/0 (socket says 172.21.15.22:47168) 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.082+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 0x55d032eae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 0x55d032eae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43694/0 (socket says 172.21.15.22:43694) 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c05974740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d032e90050 con 0x55d032e8bf60 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c05974740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d032e90190 con 0x55d032e952d0 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c05974740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d032e90350 con 0x55d032e8cc40 2024-03-24T06:42:28.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c05974740 10 client.0 _ll_register_callbacks cb 0x7f1be400b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c00c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1c00c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60262/0 (socket says 172.21.15.22:60262) 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.083+0000 7f1bfb400640 1 -- 172.21.15.22:0/2784730722 learned_addr learned my addr 172.21.15.22:0/2784730722 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.084+0000 7f1c00c00640 1 -- 172.21.15.22:0/2784730722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 msgr2=0x55d032e8f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.084+0000 7f1c00c00640 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e8f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.084+0000 7f1c00c00640 1 -- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 msgr2=0x55d032eae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.084+0000 7f1c00c00640 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 0x55d032eae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.084+0000 7f1c00c00640 1 -- 172.21.15.22:0/2784730722 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d032db4c20 con 0x55d032e952d0 2024-03-24T06:42:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.085+0000 7f1c00c00640 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1be401a9b0 tx=0x7f1be401ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eb20d831c62a410 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.085+0000 7f1be9600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.085+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4004a50 con 0x55d032e952d0 2024-03-24T06:42:28.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.085+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1be4004bb0 con 0x55d032e952d0 2024-03-24T06:42:28.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.085+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bf401ea90 con 0x55d032e952d0 2024-03-24T06:42:28.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.086+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1be4004db0 con 0x55d032e952d0 2024-03-24T06:42:28.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.086+0000 7f1c05974740 10 client.26234 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:28.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.086+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x7f1bfc1aadc0 con 0x55d032e952d0 2024-03-24T06:42:28.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.086+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bf401ea90 con 0x55d032e952d0 2024-03-24T06:42:28.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.087+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f1be4005040 con 0x55d032e952d0 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 10 client.26234 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 20 client.26234 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032f2e450 0x55d032f30870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 4 client.26234 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.088+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d032b2f720 con 0x55d032f2e450 2024-03-24T06:42:28.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.089+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032f2e450 0x55d032f30870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:28.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.090+0000 7f1bfbe00640 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032f2e450 0x55d032f30870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5d6da9cc211782 server_cookie=3829125b4044dddb in_seq=0 out_seq=0 2024-03-24T06:42:28.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.090+0000 7f1be9600640 10 client.26234 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:28.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.091+0000 7f1be9600640 1 -- 172.21.15.22:0/2784730722 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d032b2f720 con 0x55d032f2e450 2024-03-24T06:42:28.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.091+0000 7f1be9600640 10 client.26234 handle_command_reply: tid=0 2024-03-24T06:42:28.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.091+0000 7f1c05974740 1 client.26234 shutdown 2024-03-24T06:42:28.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.094+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 msgr2=0x55d032e8fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.094+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1be401a9b0 tx=0x7f1be401ac60 comp rx=0 tx=0).stop 2024-03-24T06:42:28.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 shutdown_connections 2024-03-24T06:42:28.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d032f2e450 0x55d032f30870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d032e952d0 0x55d032e8fac0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1be9600640 0 client.26234 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d032e8cc40 0x55d032e8f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 --2- 172.21.15.22:0/2784730722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d032e8bf60 0x55d032eae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.095+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 >> 172.21.15.22:0/2784730722 conn(0x55d032e52b30 msgr2=0x55d032e85bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:28.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.096+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 shutdown_connections 2024-03-24T06:42:28.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.096+0000 7f1c05974740 1 -- 172.21.15.22:0/2784730722 wait complete. 2024-03-24T06:42:28.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.097+0000 7f1c05974740 20 client.26234 trim_cache size 0 max 16384 2024-03-24T06:42:28.099 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "0ba70511-c0fb-47b8-8ee0-aca8e8f92b51": { 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "0ba70511-c0fb-47b8-8ee0-aca8e8f92b51", 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:42:28.100 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.100+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 msgr2=0x7f1be00a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.100+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 0x7f1be00a47f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4005fd0 tx=0x7f1bf4005850 comp rx=0 tx=0).stop 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.100+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 msgr2=0x7f1bfc076160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.100+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1bec000f80 tx=0x7f1bec034cc0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 shutdown_connections 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1be00a2330 0x7f1be00a47f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1bfc1097a0 0x7f1bfc0766a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bfc104150 0x7f1bfc076160 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 --2- 172.21.15.22:0/448483398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bfc103790 0x7f1bfc075c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 >> 172.21.15.22:0/448483398 conn(0x7f1bfc0ff220 msgr2=0x7f1bfc0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 shutdown_connections 2024-03-24T06:42:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:28.101+0000 7f1c01600640 1 -- 172.21.15.22:0/448483398 wait complete. 2024-03-24T06:42:28.133 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:27 smithi022 ceph-mon[32003]: pgmap v76: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 265 MiB/s wr, 109 op/s 2024-03-24T06:42:28.138 INFO:tasks.cephfs.filesystem:scrub status for tag:0ba70511-c0fb-47b8-8ee0-aca8e8f92b51 - {'path': '/', 'tag': '0ba70511-c0fb-47b8-8ee0-aca8e8f92b51', 'options': 'recursive,force'} 2024-03-24T06:42:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:42:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:42:28.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:28.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:29.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:42:30.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:29 smithi155 ceph-mon[36262]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 195 MiB/s wr, 75 op/s 2024-03-24T06:42:30.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:29 smithi022 ceph-mon[32003]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 195 MiB/s wr, 75 op/s 2024-03-24T06:42:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:29 smithi143 ceph-mon[36520]: pgmap v77: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 195 MiB/s wr, 75 op/s 2024-03-24T06:42:32.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:31 smithi155 ceph-mon[36262]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 73 op/s 2024-03-24T06:42:32.153 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:31 smithi022 ceph-mon[32003]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 73 op/s 2024-03-24T06:42:32.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:31 smithi143 ceph-mon[36520]: pgmap v78: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 190 MiB/s wr, 73 op/s 2024-03-24T06:42:34.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:33 smithi155 ceph-mon[36262]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 290 MiB/s wr, 106 op/s 2024-03-24T06:42:34.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:33 smithi022 ceph-mon[32003]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 290 MiB/s wr, 106 op/s 2024-03-24T06:42:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:33 smithi143 ceph-mon[36520]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 290 MiB/s wr, 106 op/s 2024-03-24T06:42:35.859 INFO:tasks.workunit.client.0.smithi022.stdout:Syncing()...12 sec 2024-03-24T06:42:35.862 INFO:tasks.workunit.client.0.smithi022.stdout:Starting Actual Benchmark At: Sun Mar 24 06:42:35 2024 2024-03-24T06:42:35.862 INFO:tasks.workunit.client.0.smithi022.stdout: 2024-03-24T06:42:35.955 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:35 smithi022 ceph-mon[32003]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 196 MiB/s wr, 68 op/s 2024-03-24T06:42:36.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:35 smithi155 ceph-mon[36262]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 196 MiB/s wr, 68 op/s 2024-03-24T06:42:36.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:35 smithi143 ceph-mon[36520]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 196 MiB/s wr, 68 op/s 2024-03-24T06:42:38.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:37 smithi155 ceph-mon[36262]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 MiB/s wr, 104 op/s 2024-03-24T06:42:38.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:37 smithi022 ceph-mon[32003]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 MiB/s wr, 104 op/s 2024-03-24T06:42:38.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:37 smithi143 ceph-mon[36520]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 286 MiB/s wr, 104 op/s 2024-03-24T06:42:40.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:39 smithi155 ceph-mon[36262]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 200 MiB/s wr, 72 op/s 2024-03-24T06:42:40.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:39 smithi022 ceph-mon[32003]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 200 MiB/s wr, 72 op/s 2024-03-24T06:42:40.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:39 smithi143 ceph-mon[36520]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 200 MiB/s wr, 72 op/s 2024-03-24T06:42:42.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:41 smithi155 ceph-mon[36262]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 69 op/s 2024-03-24T06:42:42.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:41 smithi022 ceph-mon[32003]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 69 op/s 2024-03-24T06:42:42.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:41 smithi143 ceph-mon[36520]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 190 MiB/s wr, 69 op/s 2024-03-24T06:42:43.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:42 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:43.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:42 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:43.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:42 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:44.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:43 smithi155 ceph-mon[36262]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 199 MiB/s wr, 1.69k op/s 2024-03-24T06:42:44.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:44.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:44.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:42:44.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:43 smithi022 ceph-mon[32003]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 199 MiB/s wr, 1.69k op/s 2024-03-24T06:42:44.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:44.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:44.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:42:44.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:43 smithi143 ceph-mon[36520]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 199 MiB/s wr, 1.69k op/s 2024-03-24T06:42:44.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:44.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:44.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:42:46.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:45 smithi155 ceph-mon[36262]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 99 MiB/s wr, 1.66k op/s 2024-03-24T06:42:46.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:45 smithi022 ceph-mon[32003]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 99 MiB/s wr, 1.66k op/s 2024-03-24T06:42:46.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:45 smithi143 ceph-mon[36520]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 99 MiB/s wr, 1.66k op/s 2024-03-24T06:42:48.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:47 smithi155 ceph-mon[36262]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 115 MiB/s wr, 4.25k op/s 2024-03-24T06:42:48.151 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:47 smithi022 ceph-mon[32003]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 115 MiB/s wr, 4.25k op/s 2024-03-24T06:42:48.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:47 smithi143 ceph-mon[36520]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 115 MiB/s wr, 4.25k op/s 2024-03-24T06:42:50.027 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:49 smithi155 ceph-mon[36262]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:50.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:49 smithi022 ceph-mon[32003]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:50.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:49 smithi143 ceph-mon[36520]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:52.155 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:51 smithi022 ceph-mon[32003]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:52.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:51 smithi143 ceph-mon[36520]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:52.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:51 smithi155 ceph-mon[36262]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 31 MiB/s wr, 5.18k op/s 2024-03-24T06:42:54.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:53 smithi022 ceph-mon[32003]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 52 MiB/s wr, 8.05k op/s 2024-03-24T06:42:54.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:53 smithi143 ceph-mon[36520]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 52 MiB/s wr, 8.05k op/s 2024-03-24T06:42:54.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:53 smithi155 ceph-mon[36262]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 52 MiB/s wr, 8.05k op/s 2024-03-24T06:42:56.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:55 smithi022 ceph-mon[32003]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 48 MiB/s wr, 7.17k op/s 2024-03-24T06:42:56.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:55 smithi143 ceph-mon[36520]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 48 MiB/s wr, 7.17k op/s 2024-03-24T06:42:56.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:55 smithi155 ceph-mon[36262]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 48 MiB/s wr, 7.17k op/s 2024-03-24T06:42:58.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:42:58.139 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:42:58.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:57 smithi022 ceph-mon[32003]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 65 MiB/s wr, 9.13k op/s 2024-03-24T06:42:58.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:57 smithi143 ceph-mon[36520]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 65 MiB/s wr, 9.13k op/s 2024-03-24T06:42:58.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:57 smithi155 ceph-mon[36262]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 65 MiB/s wr, 9.13k op/s 2024-03-24T06:42:58.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.454+0000 7f8bb4000640 1 -- 172.21.15.22:0/2585303141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac114860 msgr2=0x7f8bac116c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.454+0000 7f8bb4000640 1 --2- 172.21.15.22:0/2585303141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac114860 0x7f8bac116c50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009900 tx=0x7f8ba002bc60 comp rx=0 tx=0).stop 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 -- 172.21.15.22:0/2585303141 shutdown_connections 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 --2- 172.21.15.22:0/2585303141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac114860 0x7f8bac116c50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 --2- 172.21.15.22:0/2585303141 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8bac109210 0x7f8bac114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 --2- 172.21.15.22:0/2585303141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac1088f0 0x7f8bac108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 -- 172.21.15.22:0/2585303141 >> 172.21.15.22:0/2585303141 conn(0x7f8bac06c7f0 msgr2=0x7f8bac06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 -- 172.21.15.22:0/2585303141 shutdown_connections 2024-03-24T06:42:58.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.455+0000 7f8bb4000640 1 -- 172.21.15.22:0/2585303141 wait complete. 2024-03-24T06:42:58.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.457+0000 7f8bb4000640 1 Processor -- start 2024-03-24T06:42:58.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.457+0000 7f8bb4000640 1 -- start start 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8bac1088f0 0x7f8bac1a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac109210 0x7f8bac1a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a75e0 con 0x7f8bac1088f0 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a7750 con 0x7f8bac114860 2024-03-24T06:42:58.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb4000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1a7910 con 0x7f8bac109210 2024-03-24T06:42:58.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34306/0 (socket says 172.21.15.22:34306) 2024-03-24T06:42:58.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 -- 172.21.15.22:0/3961747573 learned_addr learned my addr 172.21.15.22:0/3961747573 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.458+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac109210 0x7f8bac1a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 -- 172.21.15.22:0/3961747573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac109210 msgr2=0x7f8bac1a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac109210 0x7f8bac1a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 -- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8bac1088f0 msgr2=0x7f8bac1a2770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8bac1088f0 0x7f8bac1a2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.459+0000 7f8bb2000640 1 -- 172.21.15.22:0/3961747573 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba0009560 con 0x7f8bac114860 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.460+0000 7f8bb2000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002cd90 tx=0x7f8ba002cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.460+0000 7f8baa000640 1 -- 172.21.15.22:0/3961747573 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba00044c0 con 0x7f8bac114860 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.460+0000 7f8baa000640 1 -- 172.21.15.22:0/3961747573 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba00052d0 con 0x7f8bac114860 2024-03-24T06:42:58.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.460+0000 7f8baa000640 1 -- 172.21.15.22:0/3961747573 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0003dc0 con 0x7f8bac114860 2024-03-24T06:42:58.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.461+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac1a7b90 con 0x7f8bac114860 2024-03-24T06:42:58.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.461+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac1af470 con 0x7f8bac114860 2024-03-24T06:42:58.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.465+0000 7f8bb6e13740 1 Processor -- start 2024-03-24T06:42:58.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.466+0000 7f8baa000640 1 -- 172.21.15.22:0/3961747573 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8ba00035f0 con 0x7f8bac114860 2024-03-24T06:42:58.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.466+0000 7f8bb6e13740 1 -- start start 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6c7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d55c635790 con 0x55d55c6c6d80 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d55c632cc0 con 0x55d55c6c7a60 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d55c62e4a0 con 0x55d55c6d00f0 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34318/0 (socket says 172.21.15.22:34318) 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb0c00640 1 -- 172.21.15.22:0/3720108193 learned_addr learned my addr 172.21.15.22:0/3720108193 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.467+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8baa000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 0x7f8b940a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb1600640 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8baa000640 1 -- 172.21.15.22:0/3961747573 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ba00f0a20 con 0x7f8bac114860 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb0c00640 1 -- 172.21.15.22:0/3720108193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 msgr2=0x55d55c6d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb0c00640 1 -- 172.21.15.22:0/3720108193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 msgr2=0x55d55c6c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.468+0000 7f8bb0c00640 1 -- 172.21.15.22:0/3720108193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d55c5efc20 con 0x55d55c6c7a60 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.469+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8b98016530 tx=0x7f8b9800a2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e96dde0ab51078 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.469+0000 7f8bb2000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 0x7f8b940a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.469+0000 7f8b9e600640 1 -- 172.21.15.22:0/3720108193 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98018430 con 0x55d55c6c7a60 2024-03-24T06:42:58.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.469+0000 7f8b9e600640 1 -- 172.21.15.22:0/3720108193 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b9800c030 con 0x55d55c6c7a60 2024-03-24T06:42:58.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.469+0000 7f8b9e600640 1 -- 172.21.15.22:0/3720108193 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98017800 con 0x55d55c6c7a60 2024-03-24T06:42:58.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb2000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 0x7f8b940a47f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002c510 tx=0x7f8ba002c480 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3720108193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 msgr2=0x55d55c6c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8b98016530 tx=0x7f8b9800a2b0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3720108193 shutdown_connections 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3720108193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6c67b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3720108193 >> 172.21.15.22:0/3720108193 conn(0x55d55c68db30 msgr2=0x55d55c6c17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.470+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3720108193 shutdown_connections 2024-03-24T06:42:58.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.471+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3720108193 wait complete. 2024-03-24T06:42:58.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.471+0000 7f8bb6e13740 1 -- start start 2024-03-24T06:42:58.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.471+0000 7f8bb6e13740 1 Processor -- start 2024-03-24T06:42:58.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.475+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d55c5d2f20 con 0x55d55c6c6d80 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d55c5d2bc0 con 0x55d55c6c7a60 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d55c39c1c0 con 0x55d55c6d00f0 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb6e13740 10 client.0 _ll_register_callbacks cb 0x55d55c33ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:58.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34334/0 (socket says 172.21.15.22:34334) 2024-03-24T06:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.476+0000 7f8bb1600640 1 -- 172.21.15.22:0/3613061137 learned_addr learned my addr 172.21.15.22:0/3613061137 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.478+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.478+0000 7f8bb1600640 1 -- 172.21.15.22:0/3613061137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 msgr2=0x55d55c6cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.478+0000 7f8bb1600640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.478+0000 7f8bb1600640 1 -- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 msgr2=0x55d55c6cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.478+0000 7f8bb1600640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.479+0000 7f8bb1600640 1 -- 172.21.15.22:0/3613061137 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9800e040 con 0x55d55c6c7a60 2024-03-24T06:42:58.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.479+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:58.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.480+0000 7f8bb1600640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400e9f0 tx=0x7f8ba400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe0b3528ab752f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.480+0000 7f8b92a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:58.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.480+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400ccc0 con 0x55d55c6c7a60 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.480+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d55c626160 con 0x55d55c6c7a60 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.480+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ba400ce20 con 0x55d55c6c7a60 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.481+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8ba4010430 con 0x55d55c6c7a60 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.481+0000 7f8bb6e13740 10 client.26243 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.481+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d55c62cb20 con 0x55d55c6c7a60 2024-03-24T06:42:58.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.481+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d55c626160 con 0x55d55c6c7a60 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f8ba401a6f0 con 0x55d55c6c7a60 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 10 client.26243 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 10 client.26243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 20 client.26243 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c6e04e0 0x55d55c6e2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 4 client.26243 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d55c32d7e0 con 0x55d55c6e04e0 2024-03-24T06:42:58.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.482+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c6e04e0 0x55d55c6e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.483+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c6e04e0 0x55d55c6e2900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5bd8c9027bbacf2 server_cookie=6db59ab9a4ad5ee6 in_seq=0 out_seq=0 2024-03-24T06:42:58.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.483+0000 7f8b92a00640 10 client.26243 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:58.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.485+0000 7f8b92a00640 1 -- 172.21.15.22:0/3613061137 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d55c32d7e0 con 0x55d55c6e04e0 2024-03-24T06:42:58.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.485+0000 7f8b92a00640 10 client.26243 handle_command_reply: tid=0 2024-03-24T06:42:58.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.485+0000 7f8bb6e13740 1 client.26243 shutdown 2024-03-24T06:42:58.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.485+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 msgr2=0x55d55c6cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.485+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400e9f0 tx=0x7f8ba400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 shutdown_connections 2024-03-24T06:42:58.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c6e04e0 0x55d55c6e2900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c7a60 0x55d55c6cd6c0 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400e9f0 tx=0x7f8ba400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/3613061137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6c6d80 0x55d55c6cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 >> 172.21.15.22:0/3613061137 conn(0x55d55c68db30 msgr2=0x55d55c6c3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.486+0000 7f8b92a00640 0 client.26243 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.489+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 shutdown_connections 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.489+0000 7f8bb6e13740 1 -- 172.21.15.22:0/3613061137 wait complete. 2024-03-24T06:42:58.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.490+0000 7f8bb6e13740 20 client.26243 trim_cache size 0 max 16384 2024-03-24T06:42:58.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.520+0000 7f8bb6e13740 1 Processor -- start 2024-03-24T06:42:58.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.520+0000 7f8bb6e13740 1 -- start start 2024-03-24T06:42:58.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6c6d80 0x55d55c6cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d55c626160 con 0x55d55c6db980 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d55c6c7fb0 con 0x55d55c6d00f0 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.521+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d55c6c3b30 con 0x55d55c6c6d80 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.522+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.522+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.522+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34346/0 (socket says 172.21.15.22:34346) 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.522+0000 7f8bb1600640 1 -- 172.21.15.22:0/2788772008 learned_addr learned my addr 172.21.15.22:0/2788772008 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.522+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6c6d80 0x55d55c6cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 -- 172.21.15.22:0/2788772008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6c6d80 msgr2=0x55d55c6cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6c6d80 0x55d55c6cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 -- 172.21.15.22:0/2788772008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 msgr2=0x55d55c6cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 -- 172.21.15.22:0/2788772008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d55c5c89a0 con 0x55d55c6d00f0 2024-03-24T06:42:58.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.523+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4002230 tx=0x7f8ba400e5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50c233b2b898a288 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.524+0000 7f8b9dc00640 1 -- 172.21.15.22:0/2788772008 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba400c6a0 con 0x55d55c6d00f0 2024-03-24T06:42:58.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.524+0000 7f8b9dc00640 1 -- 172.21.15.22:0/2788772008 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba4007770 con 0x55d55c6d00f0 2024-03-24T06:42:58.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.524+0000 7f8b9dc00640 1 -- 172.21.15.22:0/2788772008 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba401a770 con 0x55d55c6d00f0 2024-03-24T06:42:58.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2788772008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 msgr2=0x55d55c6c1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4002230 tx=0x7f8ba400e5e0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2788772008 shutdown_connections 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6d00f0 0x55d55c6c1300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2788772008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6c6d80 0x55d55c6cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.526+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2788772008 >> 172.21.15.22:0/2788772008 conn(0x55d55c68db30 msgr2=0x55d55c6c3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.528+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2788772008 shutdown_connections 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.528+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2788772008 wait complete. 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.529+0000 7f8bb6e13740 1 -- start start 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.529+0000 7f8bb6e13740 1 Processor -- start 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c6d80 0x55d55c6d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6d6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d55c6d1ee0 con 0x55d55c6db980 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d55c6d69d0 con 0x55d55c6c6d80 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d55c6d6b90 con 0x55d55c6d00f0 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.530+0000 7f8bb6e13740 10 client.0 _ll_register_callbacks cb 0x7f8ba4007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c6d80 0x55d55c6d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56802/0 (socket says 172.21.15.22:56802) 2024-03-24T06:42:58.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 -- 172.21.15.22:0/2934566077 learned_addr learned my addr 172.21.15.22:0/2934566077 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6d6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 -- 172.21.15.22:0/2934566077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c6d80 msgr2=0x55d55c6d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c6d80 0x55d55c6d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 -- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 msgr2=0x55d55c6d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6d6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.531+0000 7f8bb1600640 1 -- 172.21.15.22:0/2934566077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba401f040 con 0x55d55c6d00f0 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.532+0000 7f8bb1600640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000f80 tx=0x7f8ba401a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7da9e87345121508 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.532+0000 7f8b92a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.532+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6d6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.533+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4036070 con 0x55d55c6d00f0 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.533+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8ba400cac0 con 0x55d55c6d00f0 2024-03-24T06:42:58.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.533+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d55c6c1960 con 0x55d55c6d00f0 2024-03-24T06:42:58.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.533+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8ba401a890 con 0x55d55c6d00f0 2024-03-24T06:42:58.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.534+0000 7f8bb6e13740 10 client.26293 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:58.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.534+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d55c5ee480 con 0x55d55c6d00f0 2024-03-24T06:42:58.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d55c6c1960 con 0x55d55c6d00f0 2024-03-24T06:42:58.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f8ba4039070 con 0x55d55c6d00f0 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 10 client.26293 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 20 client.26293 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c74fc30 0x55d55c752050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 4 client.26293 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:42:58.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f8b9800c030 con 0x55d55c74fc30 2024-03-24T06:42:58.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.535+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c74fc30 0x55d55c752050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.538+0000 7f8bb0c00640 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c74fc30 0x55d55c752050 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d32d2f49aa45b0d9 server_cookie=72301b2f9fd2e79a in_seq=0 out_seq=0 2024-03-24T06:42:58.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.538+0000 7f8b92a00640 10 client.26293 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:58.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.539+0000 7f8b92a00640 1 -- 172.21.15.22:0/2934566077 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f8b9800c030 con 0x55d55c74fc30 2024-03-24T06:42:58.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.539+0000 7f8b92a00640 10 client.26293 handle_command_reply: tid=0 2024-03-24T06:42:58.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.539+0000 7f8bb6e13740 1 client.26293 shutdown 2024-03-24T06:42:58.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.542+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 msgr2=0x55d55c6d17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.542+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000f80 tx=0x7f8ba401a610 comp rx=0 tx=0).stop 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 shutdown_connections 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d55c6db980 0x55d55c6d6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d55c6d00f0 0x55d55c6d17e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d55c74fc30 0x55d55c752050 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 --2- 172.21.15.22:0/2934566077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d55c6c6d80 0x55d55c6d12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8b92a00640 0 client.26293 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 >> 172.21.15.22:0/2934566077 conn(0x55d55c68db30 msgr2=0x55d55c6c2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 shutdown_connections 2024-03-24T06:42:58.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 1 -- 172.21.15.22:0/2934566077 wait complete. 2024-03-24T06:42:58.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.543+0000 7f8bb6e13740 20 client.26293 trim_cache size 0 max 16384 2024-03-24T06:42:58.543 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:42:58.543 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:42:58.543 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:42:58.543 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.544+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 msgr2=0x7f8b940a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.544+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 0x7f8b940a47f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002c510 tx=0x7f8ba002c480 comp rx=0 tx=0).stop 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 msgr2=0x7f8bac1a6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002cd90 tx=0x7f8ba002cdc0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 shutdown_connections 2024-03-24T06:42:58.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8b940a2330 0x7f8b940a47f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bac114860 0x7f8bac1a6fe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bac109210 0x7f8bac1a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 --2- 172.21.15.22:0/3961747573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8bac1088f0 0x7f8bac1a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 >> 172.21.15.22:0/3961747573 conn(0x7f8bac06c7f0 msgr2=0x7f8bac071650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 shutdown_connections 2024-03-24T06:42:58.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.545+0000 7f8bb4000640 1 -- 172.21.15.22:0/3961747573 wait complete. 2024-03-24T06:42:58.575 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:42:58.575 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:42:58.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.975+0000 7f5742000640 1 -- 172.21.15.22:0/1305643574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 msgr2=0x7f573c115010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.975+0000 7f5742000640 1 --2- 172.21.15.22:0/1305643574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c115010 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5734005fd0 tx=0x7f573402cc30 comp rx=0 tx=0).stop 2024-03-24T06:42:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.976+0000 7f5742000640 1 -- 172.21.15.22:0/1305643574 shutdown_connections 2024-03-24T06:42:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.976+0000 7f5742000640 1 --2- 172.21.15.22:0/1305643574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c115010 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.976+0000 7f5742000640 1 --2- 172.21.15.22:0/1305643574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c109330 0x7f573c10cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.976+0000 7f5742000640 1 --2- 172.21.15.22:0/1305643574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c108980 0x7f573c108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.976+0000 7f5742000640 1 -- 172.21.15.22:0/1305643574 >> 172.21.15.22:0/1305643574 conn(0x7f573c06c800 msgr2=0x7f573c06cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:58.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.977+0000 7f5742000640 1 -- 172.21.15.22:0/1305643574 shutdown_connections 2024-03-24T06:42:58.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.977+0000 7f5742000640 1 -- 172.21.15.22:0/1305643574 wait complete. 2024-03-24T06:42:58.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.978+0000 7f5742000640 1 Processor -- start 2024-03-24T06:42:58.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.979+0000 7f5742000640 1 -- start start 2024-03-24T06:42:58.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.980+0000 7f5742000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f5742000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c11e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f5742000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 0x7f573c119920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f5742000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f573c11a0b0 con 0x7f573c119470 2024-03-24T06:42:58.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f5742000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f573c11a220 con 0x7f573c108980 2024-03-24T06:42:58.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f5742000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f573c124020 con 0x7f573c10d440 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.981+0000 7f573b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34378/0 (socket says 172.21.15.22:34378) 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f5733e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c11e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573b400640 1 -- 172.21.15.22:0/150463174 learned_addr learned my addr 172.21.15.22:0/150463174 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573b400640 1 -- 172.21.15.22:0/150463174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 msgr2=0x7f573c11e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573b400640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c11e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573b400640 1 -- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 msgr2=0x7f573c119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:58.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.982+0000 7f573be00640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 0x7f573c119920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.983+0000 7f573b400640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 0x7f573c119920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:58.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.984+0000 7f573b400640 1 -- 172.21.15.22:0/150463174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5734007490 con 0x7f573c108980 2024-03-24T06:42:58.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.984+0000 7f573be00640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 0x7f573c119920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:58.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.984+0000 7f573b400640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f572c00d950 tx=0x7f572c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:58.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.984+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c009a30 con 0x7f573c108980 2024-03-24T06:42:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.985+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573c124300 con 0x7f573c108980 2024-03-24T06:42:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.985+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f572c009b90 con 0x7f573c108980 2024-03-24T06:42:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.985+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c004a00 con 0x7f573c108980 2024-03-24T06:42:58.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.985+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f573c124860 con 0x7f573c108980 2024-03-24T06:42:58.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.986+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f573c109930 con 0x7f573c108980 2024-03-24T06:42:58.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.991+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f572c009d00 con 0x7f573c108980 2024-03-24T06:42:58.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.992+0000 7f5738c00640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 0x7f57200a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:58.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.993+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f572c0ce510 con 0x7f573c108980 2024-03-24T06:42:58.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.996+0000 7f5733e00640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 0x7f57200a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:58.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.996+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f572c096e60 con 0x7f573c108980 2024-03-24T06:42:58.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:58.998+0000 7f5733e00640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 0x7f57200a47f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f573c11abe0 tx=0x7f5724005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:59.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.446+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f573c068500 con 0x7f573c108980 2024-03-24T06:42:59.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.452+0000 7f5738c00640 1 -- 172.21.15.22:0/150463174 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11953 (secure 0 0 0) 0x7f573c068500 con 0x7f573c108980 2024-03-24T06:42:59.456 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:42:59.456 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:42:05.168986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 33 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 msgr2=0x7f57200a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 0x7f57200a47f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f573c11abe0 tx=0x7f5724005f90 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 msgr2=0x7f573c11e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f572c00d950 tx=0x7f572c00de10 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 shutdown_connections 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f57200a2330 0x7f57200a47f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f573c119470 0x7f573c119920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f573c10d440 0x7f573c11e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 --2- 172.21.15.22:0/150463174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f573c108980 0x7f573c11e310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.458+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 >> 172.21.15.22:0/150463174 conn(0x7f573c06c800 msgr2=0x7f573c0712b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:59.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.470+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 shutdown_connections 2024-03-24T06:42:59.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.471+0000 7f5742000640 1 -- 172.21.15.22:0/150463174 wait complete. 2024-03-24T06:42:59.512 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:42:59.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.940+0000 7f76f8e00640 1 -- 172.21.15.22:0/3814279614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41029a0 msgr2=0x7f76f4102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.940+0000 7f76f8e00640 1 --2- 172.21.15.22:0/3814279614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41029a0 0x7f76f4102d80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009900 tx=0x7f76e002bcc0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.941+0000 7f76f8e00640 1 -- 172.21.15.22:0/3814279614 shutdown_connections 2024-03-24T06:42:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.941+0000 7f76f8e00640 1 --2- 172.21.15.22:0/3814279614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f410bce0 0x7f76f410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.941+0000 7f76f8e00640 1 --2- 172.21.15.22:0/3814279614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76f41032c0 0x7f76f410b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.941+0000 7f76f8e00640 1 --2- 172.21.15.22:0/3814279614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41029a0 0x7f76f4102d80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.941+0000 7f76f8e00640 1 -- 172.21.15.22:0/3814279614 >> 172.21.15.22:0/3814279614 conn(0x7f76f4077770 msgr2=0x7f76f4077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.943+0000 7f76f8e00640 1 -- 172.21.15.22:0/3814279614 shutdown_connections 2024-03-24T06:42:59.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.943+0000 7f76f8e00640 1 -- 172.21.15.22:0/3814279614 wait complete. 2024-03-24T06:42:59.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.945+0000 7f76f8e00640 1 Processor -- start 2024-03-24T06:42:59.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.945+0000 7f76f8e00640 1 -- start start 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41032c0 0x7f76f419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76f410bce0 0x7f76f419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f76f419ee60 con 0x7f76f410bce0 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f76f419efd0 con 0x7f76f41032c0 2024-03-24T06:42:59.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.946+0000 7f76f8e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76f419f190 con 0x7f76f41029a0 2024-03-24T06:42:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56844/0 (socket says 172.21.15.22:56844) 2024-03-24T06:42:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 -- 172.21.15.22:0/883906427 learned_addr learned my addr 172.21.15.22:0/883906427 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1400640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41032c0 0x7f76f419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 -- 172.21.15.22:0/883906427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41032c0 msgr2=0x7f76f419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41032c0 0x7f76f419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 -- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76f410bce0 msgr2=0x7f76f419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76f410bce0 0x7f76f419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.947+0000 7f76f1e00640 1 -- 172.21.15.22:0/883906427 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e0009560 con 0x7f76f41029a0 2024-03-24T06:42:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.948+0000 7f76f1e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76e0000c00 tx=0x7f76e002d880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.948+0000 7f76eaa00640 1 -- 172.21.15.22:0/883906427 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e002dc00 con 0x7f76f41029a0 2024-03-24T06:42:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.949+0000 7f76eaa00640 1 -- 172.21.15.22:0/883906427 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e002dd60 con 0x7f76f41029a0 2024-03-24T06:42:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.949+0000 7f76eaa00640 1 -- 172.21.15.22:0/883906427 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0003b20 con 0x7f76f41029a0 2024-03-24T06:42:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.949+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f419f410 con 0x7f76f41029a0 2024-03-24T06:42:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.949+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f41a6cf0 con 0x7f76f41029a0 2024-03-24T06:42:59.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.955+0000 7f76eaa00640 1 -- 172.21.15.22:0/883906427 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f76e0005460 con 0x7f76f41029a0 2024-03-24T06:42:59.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.958+0000 7f76eaa00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 0x7f76d80a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.958+0000 7f76eaa00640 1 -- 172.21.15.22:0/883906427 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76e00f0a30 con 0x7f76f41029a0 2024-03-24T06:42:59.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.958+0000 7f76fbc02740 1 Processor -- start 2024-03-24T06:42:59.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.959+0000 7f76f1400640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 0x7f76d80a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 -- start start 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d92660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d9b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d9db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561ed5d006e0 con 0x561ed5d92ba0 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561ed5cfdc10 con 0x561ed5d9b780 2024-03-24T06:42:59.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.960+0000 7f76fbc02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561ed5cf93f0 con 0x561ed5d92280 2024-03-24T06:42:59.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.961+0000 7f76f1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d9b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.961+0000 7f76f1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d9b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35098/0 (socket says 172.21.15.22:35098) 2024-03-24T06:42:59.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.961+0000 7f76f1400640 1 -- 172.21.15.22:0/4287460473 learned_addr learned my addr 172.21.15.22:0/4287460473 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:59.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d9db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f1400640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 0x7f76d80a47b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f76f40fdc70 tx=0x7f76e4009db0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:59.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f1e00640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d92660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 -- 172.21.15.22:0/4287460473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 msgr2=0x561ed5d92660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d92660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 -- 172.21.15.22:0/4287460473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 msgr2=0x561ed5d9b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d9b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f2800640 1 -- 172.21.15.22:0/4287460473 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed5cbae40 con 0x561ed5d9b780 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.963+0000 7f76f1e00640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d92660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.964+0000 7f76f2800640 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d9db70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009dd0 tx=0x7f76ec00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b703f697f09e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:59.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.964+0000 7f76df400640 1 -- 172.21.15.22:0/4287460473 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec004b20 con 0x561ed5d9b780 2024-03-24T06:42:59.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.964+0000 7f76df400640 1 -- 172.21.15.22:0/4287460473 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec003e50 con 0x561ed5d9b780 2024-03-24T06:42:59.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.964+0000 7f76df400640 1 -- 172.21.15.22:0/4287460473 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec005020 con 0x561ed5d9b780 2024-03-24T06:42:59.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.966+0000 7f76fbc02740 1 -- 172.21.15.22:0/4287460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 msgr2=0x561ed5d9db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.966+0000 7f76fbc02740 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d9db70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009dd0 tx=0x7f76ec00e720 comp rx=0 tx=0).stop 2024-03-24T06:42:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.967+0000 7f76fbc02740 1 -- 172.21.15.22:0/4287460473 shutdown_connections 2024-03-24T06:42:59.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.967+0000 7f76fbc02740 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d9db70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.967+0000 7f76fbc02740 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d9b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.967+0000 7f76fbc02740 1 --2- 172.21.15.22:0/4287460473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d92660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.967+0000 7f76fbc02740 1 -- 172.21.15.22:0/4287460473 >> 172.21.15.22:0/4287460473 conn(0x561ed5d8c9b0 msgr2=0x561ed5d8cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.968+0000 7f76fbc02740 1 -- 172.21.15.22:0/4287460473 shutdown_connections 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.968+0000 7f76fbc02740 1 -- 172.21.15.22:0/4287460473 wait complete. 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.969+0000 7f76fbc02740 1 -- start start 2024-03-24T06:42:59.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.969+0000 7f76fbc02740 1 Processor -- start 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d98800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d98d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561ed5c9e060 con 0x561ed5d92ba0 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561ed5c9dd00 con 0x561ed5d9b780 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561ed5a671c0 con 0x561ed5d92280 2024-03-24T06:42:59.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.971+0000 7f76fbc02740 10 client.0 _ll_register_callbacks cb 0x561ed5a09be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:42:59.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.973+0000 7f76f2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.973+0000 7f76f2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34402/0 (socket says 172.21.15.22:34402) 2024-03-24T06:42:59.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.973+0000 7f76f1e00640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d98800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.973+0000 7f76f1400640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d98d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.973+0000 7f76f2800640 1 -- 172.21.15.22:0/2168849830 learned_addr learned my addr 172.21.15.22:0/2168849830 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:42:59.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.975+0000 7f76f2800640 1 -- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 msgr2=0x561ed5d98800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:59.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.975+0000 7f76f2800640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d98800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.975+0000 7f76f2800640 1 -- 172.21.15.22:0/2168849830 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 msgr2=0x561ed5d98d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:42:59.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.975+0000 7f76f2800640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d98d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.975+0000 7f76f2800640 1 -- 172.21.15.22:0/2168849830 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec009ab0 con 0x561ed5d9b780 2024-03-24T06:42:59.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.976+0000 7f76f1400640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d98d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:42:59.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.977+0000 7f76f2800640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f76ec004330 tx=0x7f76ec0045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5b93efe32cbed32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:42:59.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.977+0000 7f76d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:42:59.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.978+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec026070 con 0x561ed5d9b780 2024-03-24T06:42:59.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.978+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76ec014070 con 0x561ed5d9b780 2024-03-24T06:42:59.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.978+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561ed59f87e0 con 0x561ed5d9b780 2024-03-24T06:42:59.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.981+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76ec0047f0 con 0x561ed5d9b780 2024-03-24T06:42:59.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.981+0000 7f76fbc02740 10 client.26267 fetch_fsmap learned FSMap version 33 2024-03-24T06:42:59.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.981+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ed5cf7a70 con 0x561ed5d9b780 2024-03-24T06:42:59.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561ed59f87e0 con 0x561ed5d9b780 2024-03-24T06:42:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f76ec029020 con 0x561ed5d9b780 2024-03-24T06:42:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 10 client.26267 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:42:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 10 client.26267 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:42:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 20 client.26267 populate_metadata read hostname 'smithi022' 2024-03-24T06:42:59.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5dbca90 0x561ed5dbeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:42:59.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 4 client.26267 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:42:59.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.982+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ed5a35720 con 0x561ed5dbca90 2024-03-24T06:42:59.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.983+0000 7f76f1e00640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5dbca90 0x561ed5dbeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:42:59.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.985+0000 7f76f1e00640 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5dbca90 0x561ed5dbeeb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=988e5fe4f43c4f39 server_cookie=f61af741f0560da8 in_seq=0 out_seq=0 2024-03-24T06:42:59.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.985+0000 7f76d3e00640 10 client.26267 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:59.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.987+0000 7f76d3e00640 1 -- 172.21.15.22:0/2168849830 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ed5a35720 con 0x561ed5dbca90 2024-03-24T06:42:59.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.988+0000 7f76d3e00640 10 client.26267 handle_command_reply: tid=0 2024-03-24T06:42:59.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.988+0000 7f76fbc02740 1 client.26267 shutdown 2024-03-24T06:42:59.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.991+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 msgr2=0x561ed5d99280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.991+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f76ec004330 tx=0x7f76ec0045d0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 shutdown_connections 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5dbca90 0x561ed5dbeeb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d9b780 0x561ed5d99280 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5d98d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2168849830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d92280 0x561ed5d98800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 >> 172.21.15.22:0/2168849830 conn(0x561ed5d8c9b0 msgr2=0x561ed5d8e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:42:59.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.994+0000 7f76d3e00640 0 client.26267 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:42:59.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.997+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 shutdown_connections 2024-03-24T06:42:59.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.998+0000 7f76fbc02740 1 -- 172.21.15.22:0/2168849830 wait complete. 2024-03-24T06:42:59.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:42:59.998+0000 7f76fbc02740 20 client.26267 trim_cache size 0 max 16384 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.042+0000 7f76fbc02740 1 Processor -- start 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 -- start start 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92280 0x561ed5d9af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5db8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561ed59f87e0 con 0x561ed5d92ba0 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561ed5d997d0 con 0x561ed5d92280 2024-03-24T06:43:00.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.043+0000 7f76fbc02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561ed5d91680 con 0x561ed5d9b780 2024-03-24T06:43:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.044+0000 7f76f1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5db8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.044+0000 7f76f2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.044+0000 7f76f2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56868/0 (socket says 172.21.15.22:56868) 2024-03-24T06:43:00.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.044+0000 7f76f2800640 1 -- 172.21.15.22:0/2316238070 learned_addr learned my addr 172.21.15.22:0/2316238070 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.045+0000 7f76f2800640 1 -- 172.21.15.22:0/2316238070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92280 msgr2=0x561ed5d9af30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:43:00.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.045+0000 7f76f2800640 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92280 0x561ed5d9af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.045+0000 7f76f2800640 1 -- 172.21.15.22:0/2316238070 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 msgr2=0x561ed5db8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.046+0000 7f76f2800640 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5db8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.046+0000 7f76f2800640 1 -- 172.21.15.22:0/2316238070 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ed5c93b00 con 0x561ed5d9b780 2024-03-24T06:43:00.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.046+0000 7f76f2800640 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d818aaa396766be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.047+0000 7f76dea00640 1 -- 172.21.15.22:0/2316238070 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec026070 con 0x561ed5d9b780 2024-03-24T06:43:00.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.048+0000 7f76dea00640 1 -- 172.21.15.22:0/2316238070 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec0059e0 con 0x561ed5d9b780 2024-03-24T06:43:00.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.048+0000 7f76dea00640 1 -- 172.21.15.22:0/2316238070 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec0106e0 con 0x561ed5d9b780 2024-03-24T06:43:00.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.049+0000 7f76fbc02740 1 -- 172.21.15.22:0/2316238070 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 msgr2=0x561ed5d8ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.049+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec009520 comp rx=0 tx=0).stop 2024-03-24T06:43:00.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.051+0000 7f76fbc02740 1 -- 172.21.15.22:0/2316238070 shutdown_connections 2024-03-24T06:43:00.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.051+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5d8ba70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.051+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92ba0 0x561ed5db8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.051+0000 7f76fbc02740 1 --2- 172.21.15.22:0/2316238070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92280 0x561ed5d9af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.051+0000 7f76fbc02740 1 -- 172.21.15.22:0/2316238070 >> 172.21.15.22:0/2316238070 conn(0x561ed5d8c9b0 msgr2=0x561ed5d8cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.052+0000 7f76fbc02740 1 -- 172.21.15.22:0/2316238070 shutdown_connections 2024-03-24T06:43:00.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.054+0000 7f76fbc02740 1 -- 172.21.15.22:0/2316238070 wait complete. 2024-03-24T06:43:00.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.055+0000 7f76fbc02740 1 -- start start 2024-03-24T06:43:00.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.055+0000 7f76fbc02740 1 Processor -- start 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 0x561ed5db1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5db1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561ed5db4bc0 con 0x561ed5d92280 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561ed5db4d30 con 0x561ed5d92ba0 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561ed5db4e70 con 0x561ed5d9b780 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76fbc02740 10 client.0 _ll_register_callbacks cb 0x7f76e400e850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:00.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.056+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.057+0000 7f76f1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35118/0 (socket says 172.21.15.22:35118) 2024-03-24T06:43:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.057+0000 7f76f1e00640 1 -- 172.21.15.22:0/3450359199 learned_addr learned my addr 172.21.15.22:0/3450359199 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.057+0000 7f76f1400640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 0x561ed5db1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.057+0000 7f76f1e00640 1 -- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 msgr2=0x561ed5db1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.057+0000 7f76f2800640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5db1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f1e00640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5db1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f1e00640 1 -- 172.21.15.22:0/3450359199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 msgr2=0x561ed5db1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f1e00640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 0x561ed5db1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f1e00640 1 -- 172.21.15.22:0/3450359199 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ed5cf7a70 con 0x561ed5d92280 2024-03-24T06:43:00.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f1400640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 0x561ed5db1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:43:00.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.058+0000 7f76f2800640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5db1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:43:00.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.059+0000 7f76f1e00640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f76e00b9b40 tx=0x7f76e00053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc13b04a71272d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.059+0000 7f76d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:43:00.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.061+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e00d5ea0 con 0x561ed5d92280 2024-03-24T06:43:00.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.061+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f76e00ee040 con 0x561ed5d92280 2024-03-24T06:43:00.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.061+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ec003e50 con 0x561ed5d92280 2024-03-24T06:43:00.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.063+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76e00ee200 con 0x561ed5d92280 2024-03-24T06:43:00.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.063+0000 7f76fbc02740 10 client.16011 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:00.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.063+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561ed5cb96a0 con 0x561ed5d92280 2024-03-24T06:43:00.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.064+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec003e50 con 0x561ed5d92280 2024-03-24T06:43:00.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.064+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f76e003c070 con 0x561ed5d92280 2024-03-24T06:43:00.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 10 client.16011 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:00.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 10 client.16011 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:43:00.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 20 client.16011 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:00.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5e1bbb0 0x561ed5e1dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 4 client.16011 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:43:00.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ed5a84890 con 0x561ed5e1bbb0 2024-03-24T06:43:00.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.065+0000 7f76f1400640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5e1bbb0 0x561ed5e1dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.066+0000 7f76f1400640 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5e1bbb0 0x561ed5e1dfd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=910f05f79e67c061 server_cookie=491185f3169d62ca in_seq=0 out_seq=0 2024-03-24T06:43:00.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.067+0000 7f76d3e00640 10 client.16011 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:43:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.067+0000 7f76d3e00640 1 -- 172.21.15.22:0/3450359199 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ed5a84890 con 0x561ed5e1bbb0 2024-03-24T06:43:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.067+0000 7f76d3e00640 10 client.16011 handle_command_reply: tid=0 2024-03-24T06:43:00.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.067+0000 7f76fbc02740 1 client.16011 shutdown 2024-03-24T06:43:00.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.068+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 msgr2=0x561ed5db4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.068+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f76e00b9b40 tx=0x7f76e00053d0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 shutdown_connections 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561ed5e1bbb0 0x561ed5e1dfd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561ed5d9b780 0x561ed5db1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561ed5d92ba0 0x561ed5db1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76d3e00640 0 client.16011 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 --2- 172.21.15.22:0/3450359199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561ed5d92280 0x561ed5db4680 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f76e00b9b40 tx=0x7f76e00053d0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.069+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 >> 172.21.15.22:0/3450359199 conn(0x561ed5d8c9b0 msgr2=0x561ed5d99030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.070+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 shutdown_connections 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.070+0000 7f76fbc02740 1 -- 172.21.15.22:0/3450359199 wait complete. 2024-03-24T06:43:00.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.070+0000 7f76fbc02740 20 client.16011 trim_cache size 0 max 16384 2024-03-24T06:43:00.070 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:43:00.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 msgr2=0x7f76d80a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 0x7f76d80a47b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f76f40fdc70 tx=0x7f76e4009db0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 msgr2=0x7f76f4199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76e0000c00 tx=0x7f76e002d880 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 shutdown_connections 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f76d80a22f0 0x7f76d80a47b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76f410bce0 0x7f76f419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f41032c0 0x7f76f419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 --2- 172.21.15.22:0/883906427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76f41029a0 0x7f76f4199fa0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.072+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 >> 172.21.15.22:0/883906427 conn(0x7f76f4077770 msgr2=0x7f76f4101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.073+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 shutdown_connections 2024-03-24T06:43:00.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.073+0000 7f76f8e00640 1 -- 172.21.15.22:0/883906427 wait complete. 2024-03-24T06:43:00.104 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:59 smithi022 ceph-mon[32003]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 64 MiB/s wr, 8.42k op/s 2024-03-24T06:43:00.104 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:42:59 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/150463174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:43:00.106 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:43:00.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:59 smithi143 ceph-mon[36520]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 64 MiB/s wr, 8.42k op/s 2024-03-24T06:43:00.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:42:59 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/150463174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:43:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:59 smithi155 ceph-mon[36262]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 64 MiB/s wr, 8.42k op/s 2024-03-24T06:43:00.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:42:59 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/150463174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:43:00.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.460+0000 7fe9edc00640 1 -- 172.21.15.22:0/2839420368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8075c40 msgr2=0x7fe9e8076020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.460+0000 7fe9edc00640 1 --2- 172.21.15.22:0/2839420368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8075c40 0x7fe9e8076020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009930 tx=0x7fe9d802bc30 comp rx=0 tx=0).stop 2024-03-24T06:43:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 -- 172.21.15.22:0/2839420368 shutdown_connections 2024-03-24T06:43:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 --2- 172.21.15.22:0/2839420368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8112550 0x7fe9e8114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 --2- 172.21.15.22:0/2839420368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e8112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 --2- 172.21.15.22:0/2839420368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8075c40 0x7fe9e8076020 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 -- 172.21.15.22:0/2839420368 >> 172.21.15.22:0/2839420368 conn(0x7fe9e80ff380 msgr2=0x7fe9e81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 -- 172.21.15.22:0/2839420368 shutdown_connections 2024-03-24T06:43:00.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.461+0000 7fe9edc00640 1 -- 172.21.15.22:0/2839420368 wait complete. 2024-03-24T06:43:00.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.463+0000 7fe9edc00640 1 Processor -- start 2024-03-24T06:43:00.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.463+0000 7fe9edc00640 1 -- start start 2024-03-24T06:43:00.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8075c40 0x7fe9e819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8112550 0x7fe9e81a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e81a32d0 con 0x7fe9e8075c40 2024-03-24T06:43:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e81a3440 con 0x7fe9e8112550 2024-03-24T06:43:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9edc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e81a3600 con 0x7fe9e8076560 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9e6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9e6000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56890/0 (socket says 172.21.15.22:56890) 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9e6000640 1 -- 172.21.15.22:0/1689917382 learned_addr learned my addr 172.21.15.22:0/1689917382 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.464+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8075c40 0x7fe9e819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e7400640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8112550 0x7fe9e81a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 -- 172.21.15.22:0/1689917382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8112550 msgr2=0x7fe9e81a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8112550 0x7fe9e81a2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 -- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8075c40 msgr2=0x7fe9e819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8075c40 0x7fe9e819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 -- 172.21.15.22:0/1689917382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d8009560 con 0x7fe9e8076560 2024-03-24T06:43:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9e6000640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9f0 tx=0x7fe9d400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.465+0000 7fe9df400640 1 -- 172.21.15.22:0/1689917382 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d400cd80 con 0x7fe9e8076560 2024-03-24T06:43:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.466+0000 7fe9df400640 1 -- 172.21.15.22:0/1689917382 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d4005500 con 0x7fe9e8076560 2024-03-24T06:43:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.466+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e81a38e0 con 0x7fe9e8076560 2024-03-24T06:43:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.466+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e81ab160 con 0x7fe9e8076560 2024-03-24T06:43:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.467+0000 7fe9df400640 1 -- 172.21.15.22:0/1689917382 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4010690 con 0x7fe9e8076560 2024-03-24T06:43:00.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.472+0000 7fe9f0880740 1 Processor -- start 2024-03-24T06:43:00.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 -- start start 2024-03-24T06:43:00.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f79660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f82780 0x555aa8f84b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555aa8ee76e0 con 0x555aa8f82780 2024-03-24T06:43:00.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555aa8ee4c10 con 0x555aa8f79280 2024-03-24T06:43:00.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.473+0000 7fe9f0880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555aa8ee03f0 con 0x555aa8f79ba0 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f79660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56896/0 (socket says 172.21.15.22:56896) 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 -- 172.21.15.22:0/901104240 learned_addr learned my addr 172.21.15.22:0/901104240 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 -- 172.21.15.22:0/901104240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 msgr2=0x555aa8f79660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f79660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 -- 172.21.15.22:0/901104240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f82780 msgr2=0x555aa8f84b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f82780 0x555aa8f84b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.477+0000 7fe9e6a00640 1 -- 172.21.15.22:0/901104240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aa8ea1e40 con 0x555aa8f79ba0 2024-03-24T06:43:00.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.478+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe9d802c240 tx=0x7fe9d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1956d45141bf6fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.478+0000 7fe9d3e00640 1 -- 172.21.15.22:0/901104240 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8004af0 con 0x555aa8f79ba0 2024-03-24T06:43:00.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.478+0000 7fe9d3e00640 1 -- 172.21.15.22:0/901104240 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8004040 con 0x555aa8f79ba0 2024-03-24T06:43:00.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.478+0000 7fe9d3e00640 1 -- 172.21.15.22:0/901104240 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8005350 con 0x555aa8f79ba0 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.479+0000 7fe9f0880740 1 -- 172.21.15.22:0/901104240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 msgr2=0x555aa8f82240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.479+0000 7fe9f0880740 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe9d802c240 tx=0x7fe9d80046c0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.480+0000 7fe9f0880740 1 -- 172.21.15.22:0/901104240 shutdown_connections 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.480+0000 7fe9f0880740 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f82780 0x555aa8f84b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.480+0000 7fe9f0880740 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79ba0 0x555aa8f82240 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.480+0000 7fe9f0880740 1 --2- 172.21.15.22:0/901104240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f79660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.480+0000 7fe9f0880740 1 -- 172.21.15.22:0/901104240 >> 172.21.15.22:0/901104240 conn(0x555aa8f739b0 msgr2=0x555aa8f73d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.482+0000 7fe9df400640 1 -- 172.21.15.22:0/1689917382 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe9d40046c0 con 0x7fe9e8076560 2024-03-24T06:43:00.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.482+0000 7fe9f0880740 1 -- 172.21.15.22:0/901104240 shutdown_connections 2024-03-24T06:43:00.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.484+0000 7fe9f0880740 1 -- 172.21.15.22:0/901104240 wait complete. 2024-03-24T06:43:00.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.484+0000 7fe9f0880740 1 -- start start 2024-03-24T06:43:00.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.485+0000 7fe9f0880740 1 Processor -- start 2024-03-24T06:43:00.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.487+0000 7fe9df400640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 0x7fe9cc0a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9e7400640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 0x7fe9cc0a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f7fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9df400640 1 -- 172.21.15.22:0/1689917382 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe9d4014070 con 0x7fe9e8076560 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f9f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555aa8e85060 con 0x555aa8f79ba0 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555aa8e84d00 con 0x555aa8f79280 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.488+0000 7fe9f0880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555aa8c4e1c0 con 0x555aa8f803a0 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9f0880740 10 client.0 _ll_register_callbacks cb 0x555aa8bf0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34450/0 (socket says 172.21.15.22:34450) 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 -- 172.21.15.22:0/2282514447 learned_addr learned my addr 172.21.15.22:0/2282514447 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e7400640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f9f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 -- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 msgr2=0x555aa8f9f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f9f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.489+0000 7fe9e6a00640 1 -- 172.21.15.22:0/2282514447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 msgr2=0x555aa8f7fe60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.490+0000 7fe9e6000640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f7fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.490+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f7fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.490+0000 7fe9e6a00640 1 -- 172.21.15.22:0/2282514447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e81153d0 con 0x555aa8f79280 2024-03-24T06:43:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.490+0000 7fe9e7400640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 0x7fe9cc0a46c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806abc0 tx=0x7fe9e000e040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.492+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036040 tx=0x7fe9d8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41c0ddbeb376220 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.492+0000 7fe9d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.492+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d802cbf0 con 0x555aa8f79280 2024-03-24T06:43:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.492+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe9d804f070 con 0x555aa8f79280 2024-03-24T06:43:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.492+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555aa8bdf7e0 con 0x555aa8f79280 2024-03-24T06:43:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.493+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9d8004490 con 0x555aa8f79280 2024-03-24T06:43:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.494+0000 7fe9f0880740 10 client.26279 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.494+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555aa8edea70 con 0x555aa8f79280 2024-03-24T06:43:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.495+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8004490 con 0x555aa8f79280 2024-03-24T06:43:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe9d803c040 con 0x555aa8f79280 2024-03-24T06:43:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 10 client.26279 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 10 client.26279 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:43:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 20 client.26279 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa8fa3dc0 0x555aa8fa61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 4 client.26279 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.496+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555aa8c1c720 con 0x555aa8fa3dc0 2024-03-24T06:43:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.498+0000 7fe9e6000640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa8fa3dc0 0x555aa8fa61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.500+0000 7fe9d0c00640 10 client.26279 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:43:00.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.500+0000 7fe9e6000640 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa8fa3dc0 0x555aa8fa61d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf5378b2a4e7a040 server_cookie=78934f48bc290a07 in_seq=0 out_seq=0 2024-03-24T06:43:00.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.503+0000 7fe9d0c00640 1 -- 172.21.15.22:0/2282514447 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555aa8c1c720 con 0x555aa8fa3dc0 2024-03-24T06:43:00.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.503+0000 7fe9d0c00640 10 client.26279 handle_command_reply: tid=0 2024-03-24T06:43:00.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.503+0000 7fe9f0880740 1 client.26279 shutdown 2024-03-24T06:43:00.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.504+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 msgr2=0x555aa8f7f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.504+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036040 tx=0x7fe9d8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 shutdown_connections 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa8fa3dc0 0x555aa8fa61d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f9f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f7fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 --2- 172.21.15.22:0/2282514447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8f7f920 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 >> 172.21.15.22:0/2282514447 conn(0x555aa8f739b0 msgr2=0x555aa8f75700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9d0c00640 0 client.26279 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:43:00.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.506+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 shutdown_connections 2024-03-24T06:43:00.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.507+0000 7fe9f0880740 1 -- 172.21.15.22:0/2282514447 wait complete. 2024-03-24T06:43:00.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.508+0000 7fe9f0880740 20 client.26279 trim_cache size 0 max 16384 2024-03-24T06:43:00.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.546+0000 7fe9f0880740 1 Processor -- start 2024-03-24T06:43:00.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.547+0000 7fe9f0880740 1 -- start start 2024-03-24T06:43:00.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8faba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f72e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f7de70 con 0x555aa8f79ba0 2024-03-24T06:43:00.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f74330 con 0x555aa8f79280 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9f0880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f6ec70 con 0x555aa8f803a0 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9e7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f72e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9e7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f72e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56918/0 (socket says 172.21.15.22:56918) 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8faba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.548+0000 7fe9e7400640 1 -- 172.21.15.22:0/4023175598 learned_addr learned my addr 172.21.15.22:0/4023175598 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.549+0000 7fe9e6a00640 1 -- 172.21.15.22:0/4023175598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 msgr2=0x555aa8f72e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.549+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f72e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.549+0000 7fe9e6a00640 1 -- 172.21.15.22:0/4023175598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 msgr2=0x555aa8f728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.549+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.549+0000 7fe9e6a00640 1 -- 172.21.15.22:0/4023175598 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aa8e7ab00 con 0x555aa8f79280 2024-03-24T06:43:00.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.550+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8faba40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8004010 tx=0x7fe9d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c893dcc3a79e97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.550+0000 7fe9d3400640 1 -- 172.21.15.22:0/4023175598 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8004df0 con 0x555aa8f79280 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.550+0000 7fe9d3400640 1 -- 172.21.15.22:0/4023175598 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555aa8bdf7e0 con 0x555aa8f79280 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.550+0000 7fe9d3400640 1 -- 172.21.15.22:0/4023175598 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d80052a0 con 0x555aa8f79280 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.551+0000 7fe9f0880740 1 -- 172.21.15.22:0/4023175598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 msgr2=0x555aa8faba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.551+0000 7fe9f0880740 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8faba40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8004010 tx=0x7fe9d80046c0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 -- 172.21.15.22:0/4023175598 shutdown_connections 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f803a0 0x555aa8f72e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f79ba0 0x555aa8f728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 --2- 172.21.15.22:0/4023175598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79280 0x555aa8faba40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 -- 172.21.15.22:0/4023175598 >> 172.21.15.22:0/4023175598 conn(0x555aa8f739b0 msgr2=0x555aa8f721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 -- 172.21.15.22:0/4023175598 shutdown_connections 2024-03-24T06:43:00.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.552+0000 7fe9f0880740 1 -- 172.21.15.22:0/4023175598 wait complete. 2024-03-24T06:43:00.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.553+0000 7fe9f0880740 1 -- start start 2024-03-24T06:43:00.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.553+0000 7fe9f0880740 1 Processor -- start 2024-03-24T06:43:00.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.556+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.556+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.556+0000 7fe9f0880740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f803a0 0x555aa8f970f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9e6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9e6000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34466/0 (socket says 172.21.15.22:34466) 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9e6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56926/0 (socket says 172.21.15.22:56926) 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9f0880740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f976d0 con 0x555aa8f803a0 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9f0880740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f97840 con 0x555aa8f79ba0 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9f0880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555aa8f97a00 con 0x555aa8f79280 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9e6000640 1 -- 172.21.15.22:0/945652048 learned_addr learned my addr 172.21.15.22:0/945652048 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.557+0000 7fe9f0880740 10 client.0 _ll_register_callbacks cb 0x7fe9d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:00.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.558+0000 7fe9e6a00640 1 -- 172.21.15.22:0/945652048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 msgr2=0x555aa8f96bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.558+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.558+0000 7fe9e6a00640 1 -- 172.21.15.22:0/945652048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f803a0 msgr2=0x555aa8f970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.558+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f803a0 0x555aa8f970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.558+0000 7fe9e6000640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.559+0000 7fe9e6a00640 1 -- 172.21.15.22:0/945652048 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aa8edea70 con 0x555aa8f79280 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.560+0000 7fe9e6a00640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8005ab0 tx=0x7fe9d8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13e40cb3726de9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.560+0000 7fe9d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:43:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.562+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8003a00 con 0x555aa8f79280 2024-03-24T06:43:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.562+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe9d804f040 con 0x555aa8f79280 2024-03-24T06:43:00.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.565+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9d8004040 con 0x555aa8f79280 2024-03-24T06:43:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.565+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe9d8003b60 con 0x555aa8f79280 2024-03-24T06:43:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.565+0000 7fe9f0880740 10 client.26329 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.565+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555aa8ea06a0 con 0x555aa8f79280 2024-03-24T06:43:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8004040 con 0x555aa8f79280 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fe9d8054c60 con 0x555aa8f79280 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9f0880740 10 client.26329 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9f0880740 10 client.26329 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9f0880740 20 client.26329 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.566+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa900a290 0x555aa900c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.568+0000 7fe9f0880740 4 client.26329 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:43:00.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.568+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555aa8c6b890 con 0x555aa900a290 2024-03-24T06:43:00.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.569+0000 7fe9e6000640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa900a290 0x555aa900c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.569+0000 7fe9e6000640 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa900a290 0x555aa900c6b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ab1396247990f8a server_cookie=442dd436a8d887c9 in_seq=0 out_seq=0 2024-03-24T06:43:00.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.569+0000 7fe9d0c00640 10 client.26329 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:43:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.570+0000 7fe9d0c00640 1 -- 172.21.15.22:0/945652048 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555aa8c6b890 con 0x555aa900a290 2024-03-24T06:43:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.570+0000 7fe9d0c00640 10 client.26329 handle_command_reply: tid=0 2024-03-24T06:43:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.570+0000 7fe9f0880740 1 client.26329 shutdown 2024-03-24T06:43:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.570+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 msgr2=0x555aa8f9b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.570+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8005ab0 tx=0x7fe9d8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 shutdown_connections 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555aa900a290 0x555aa900c6b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555aa8f803a0 0x555aa8f970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555aa8f79ba0 0x555aa8f96bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 --2- 172.21.15.22:0/945652048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555aa8f79280 0x555aa8f9b810 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8005ab0 tx=0x7fe9d8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 >> 172.21.15.22:0/945652048 conn(0x555aa8f739b0 msgr2=0x555aa8f72a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.571+0000 7fe9d0c00640 0 client.26329 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:43:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.573+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 shutdown_connections 2024-03-24T06:43:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.573+0000 7fe9f0880740 1 -- 172.21.15.22:0/945652048 wait complete. 2024-03-24T06:43:00.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.573+0000 7fe9f0880740 20 client.26329 trim_cache size 0 max 16384 2024-03-24T06:43:00.575 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 msgr2=0x7fe9cc0a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 0x7fe9cc0a46c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806abc0 tx=0x7fe9e000e040 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 msgr2=0x7fe9e819e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9f0 tx=0x7fe9d400edf0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 shutdown_connections 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe9cc0a2200 0x7fe9cc0a46c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e8112550 0x7fe9e81a2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9e8076560 0x7fe9e819e9a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 --2- 172.21.15.22:0/1689917382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe9e8075c40 0x7fe9e819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 >> 172.21.15.22:0/1689917382 conn(0x7fe9e80ff380 msgr2=0x7fe9e8113c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 shutdown_connections 2024-03-24T06:43:00.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.578+0000 7fe9edc00640 1 -- 172.21.15.22:0/1689917382 wait complete. 2024-03-24T06:43:00.603 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:43:00.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.891+0000 7f0243400640 1 -- 172.21.15.22:0/2214167326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 msgr2=0x7f023c076020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.891+0000 7f0243400640 1 --2- 172.21.15.22:0/2214167326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c076020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0230009860 tx=0x7f023002bbb0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 -- 172.21.15.22:0/2214167326 shutdown_connections 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 --2- 172.21.15.22:0/2214167326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c112550 0x7f023c114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 --2- 172.21.15.22:0/2214167326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c076560 0x7f023c112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 --2- 172.21.15.22:0/2214167326 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c076020 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 -- 172.21.15.22:0/2214167326 >> 172.21.15.22:0/2214167326 conn(0x7f023c0ff3c0 msgr2=0x7f023c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 -- 172.21.15.22:0/2214167326 shutdown_connections 2024-03-24T06:43:00.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.901+0000 7f0243400640 1 -- 172.21.15.22:0/2214167326 wait complete. 2024-03-24T06:43:00.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.903+0000 7f0243400640 1 Processor -- start 2024-03-24T06:43:00.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.903+0000 7f0243400640 1 -- start start 2024-03-24T06:43:00.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.903+0000 7f0243400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0243400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c112550 0x7f023c073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0243400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 0x7f023c06c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0243400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f023c06cb50 con 0x7f023c075c40 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0243400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f023c06ccc0 con 0x7f023c06bf40 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0243400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f023c070940 con 0x7f023c112550 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0240a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0240a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35190/0 (socket says 172.21.15.22:35190) 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0240a00640 1 -- 172.21.15.22:0/3197880618 learned_addr learned my addr 172.21.15.22:0/3197880618 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f023be00640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c112550 0x7f023c073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.904+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 0x7f023c06c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.905+0000 7f0241400640 1 -- 172.21.15.22:0/3197880618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c112550 msgr2=0x7f023c073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.905+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c112550 0x7f023c073360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.905+0000 7f0241400640 1 -- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 msgr2=0x7f023c072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.906+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.906+0000 7f0241400640 1 -- 172.21.15.22:0/3197880618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0230009490 con 0x7f023c06bf40 2024-03-24T06:43:00.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.906+0000 7f0240a00640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:00.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.906+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 0x7f023c06c3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0234005fd0 tx=0x7f023400c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.907+0000 7f0239600640 1 -- 172.21.15.22:0/3197880618 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234003b70 con 0x7f023c06bf40 2024-03-24T06:43:00.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.907+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c070bf0 con 0x7f023c06bf40 2024-03-24T06:43:00.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.907+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c071150 con 0x7f023c06bf40 2024-03-24T06:43:00.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.911+0000 7f0239600640 1 -- 172.21.15.22:0/3197880618 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f023400ee30 con 0x7f023c06bf40 2024-03-24T06:43:00.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.911+0000 7f0239600640 1 -- 172.21.15.22:0/3197880618 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02340178e0 con 0x7f023c06bf40 2024-03-24T06:43:00.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.919+0000 7f024602c740 1 Processor -- start 2024-03-24T06:43:00.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.920+0000 7f0239600640 1 -- 172.21.15.22:0/3197880618 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f023400a040 con 0x7f023c06bf40 2024-03-24T06:43:00.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.925+0000 7f024602c740 1 -- start start 2024-03-24T06:43:00.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a657b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a6586b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614a64e96e0 con 0x5614a6584780 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614a64e6c10 con 0x5614a657bba0 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.926+0000 7f024602c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614a64e23f0 con 0x5614a657b280 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.927+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.927+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34500/0 (socket says 172.21.15.22:34500) 2024-03-24T06:43:00.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.928+0000 7f0240a00640 1 -- 172.21.15.22:0/1597993852 learned_addr learned my addr 172.21.15.22:0/1597993852 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.929+0000 7f0239600640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 0x7f02240a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.929+0000 7f0240a00640 1 -- 172.21.15.22:0/1597993852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 msgr2=0x5614a657b660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:43:00.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.929+0000 7f0240a00640 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a657b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.929+0000 7f0240a00640 1 -- 172.21.15.22:0/1597993852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 msgr2=0x5614a6586b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f023be00640 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a6586b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f0239600640 1 -- 172.21.15.22:0/3197880618 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f02340ce060 con 0x7f023c06bf40 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 0x7f02240a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f0240a00640 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a6586b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f0240a00640 1 -- 172.21.15.22:0/1597993852 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a64a3e40 con 0x5614a657bba0 2024-03-24T06:43:00.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.930+0000 7f023be00640 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a6586b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:43:00.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.931+0000 7f0240a00640 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0230009780 tx=0x7f02300056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3df174da76c2a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.932+0000 7f0229c00640 1 -- 172.21.15.22:0/1597993852 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230004340 con 0x5614a657bba0 2024-03-24T06:43:00.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.932+0000 7f0229c00640 1 -- 172.21.15.22:0/1597993852 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02300044a0 con 0x5614a657bba0 2024-03-24T06:43:00.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.932+0000 7f0229c00640 1 -- 172.21.15.22:0/1597993852 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230003850 con 0x5614a657bba0 2024-03-24T06:43:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.933+0000 7f024602c740 1 -- 172.21.15.22:0/1597993852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 msgr2=0x5614a6584240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.933+0000 7f024602c740 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0230009780 tx=0x7f02300056d0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.933+0000 7f0241400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 0x7f02240a4770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0234004990 tx=0x7f02340ce320 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 -- 172.21.15.22:0/1597993852 shutdown_connections 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a6586b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6584240 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 --2- 172.21.15.22:0/1597993852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a657b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 -- 172.21.15.22:0/1597993852 >> 172.21.15.22:0/1597993852 conn(0x5614a65759b0 msgr2=0x5614a6575d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 -- 172.21.15.22:0/1597993852 shutdown_connections 2024-03-24T06:43:00.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.934+0000 7f024602c740 1 -- 172.21.15.22:0/1597993852 wait complete. 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.935+0000 7f024602c740 1 -- start start 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.935+0000 7f024602c740 1 Processor -- start 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a6581770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a65821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614a6487060 con 0x5614a6584780 2024-03-24T06:43:00.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614a6486d00 con 0x5614a657bba0 2024-03-24T06:43:00.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614a62501c0 con 0x5614a657b280 2024-03-24T06:43:00.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.936+0000 7f024602c740 10 client.0 _ll_register_callbacks cb 0x5614a61f2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:00.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.938+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.938+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34506/0 (socket says 172.21.15.22:34506) 2024-03-24T06:43:00.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.938+0000 7f0240a00640 1 -- 172.21.15.22:0/1921850986 learned_addr learned my addr 172.21.15.22:0/1921850986 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:00.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.939+0000 7f023be00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a65821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.939+0000 7f023be00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a6581770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.941+0000 7f0240a00640 1 -- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 msgr2=0x5614a6581770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.941+0000 7f0240a00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a6581770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.941+0000 7f0240a00640 1 -- 172.21.15.22:0/1921850986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 msgr2=0x5614a65821f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:00.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.941+0000 7f0240a00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a65821f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.941+0000 7f0240a00640 1 -- 172.21.15.22:0/1921850986 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c1153d0 con 0x5614a657bba0 2024-03-24T06:43:00.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.942+0000 7f0240a00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f02300057c0 tx=0x7f0230034650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d710da95cad15f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:00.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.942+0000 7f021e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:00.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.943+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230034a50 con 0x5614a657bba0 2024-03-24T06:43:00.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.943+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f023002cba0 con 0x5614a657bba0 2024-03-24T06:43:00.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.944+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614a61e17e0 con 0x5614a657bba0 2024-03-24T06:43:00.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.944+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0230032800 con 0x5614a657bba0 2024-03-24T06:43:00.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.944+0000 7f024602c740 10 client.26294 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:00.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.944+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5614a64e0a70 con 0x5614a657bba0 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5614a61e17e0 con 0x5614a657bba0 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f023005a050 con 0x5614a657bba0 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 10 client.26294 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 10 client.26294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 20 client.26294 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a65a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 4 client.26294 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:00.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.945+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614a621e720 con 0x5614a65a57b0 2024-03-24T06:43:00.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.946+0000 7f023be00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a65a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:00.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.947+0000 7f021e000640 10 client.26294 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:43:00.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.947+0000 7f023be00640 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a65a7bd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea03925078eb3ef2 server_cookie=2287a74fe5d4b232 in_seq=0 out_seq=0 2024-03-24T06:43:00.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.952+0000 7f021e000640 1 -- 172.21.15.22:0/1921850986 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614a621e720 con 0x5614a65a57b0 2024-03-24T06:43:00.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.952+0000 7f021e000640 10 client.26294 handle_command_reply: tid=0 2024-03-24T06:43:00.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.952+0000 7f024602c740 1 client.26294 shutdown 2024-03-24T06:43:00.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 msgr2=0x5614a6581cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:00.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f02300057c0 tx=0x7f0230034650 comp rx=0 tx=0).stop 2024-03-24T06:43:00.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 shutdown_connections 2024-03-24T06:43:00.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a65a7bd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f021e000640 0 client.26294 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a6584780 0x5614a65821f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6581cb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 --2- 172.21.15.22:0/1921850986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657b280 0x5614a6581770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 >> 172.21.15.22:0/1921850986 conn(0x5614a65759b0 msgr2=0x5614a6577700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.953+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 shutdown_connections 2024-03-24T06:43:00.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.954+0000 7f024602c740 1 -- 172.21.15.22:0/1921850986 wait complete. 2024-03-24T06:43:00.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:00.954+0000 7f024602c740 20 client.26294 trim_cache size 0 max 16384 2024-03-24T06:43:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.001+0000 7f024602c740 1 Processor -- start 2024-03-24T06:43:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.001+0000 7f024602c740 1 -- start start 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a6583ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a6584780 0x5614a659d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614a6586b70 con 0x5614a657b280 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614a65a2680 con 0x5614a6584780 2024-03-24T06:43:01.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.002+0000 7f024602c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614a657aee0 con 0x5614a657bba0 2024-03-24T06:43:01.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.004+0000 7f0240a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.004+0000 7f0240a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56984/0 (socket says 172.21.15.22:56984) 2024-03-24T06:43:01.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.004+0000 7f023be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a6584780 0x5614a659d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.004+0000 7f023be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a6583ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.005+0000 7f023be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a6583ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35202/0 (socket says 172.21.15.22:35202) 2024-03-24T06:43:01.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.005+0000 7f023be00640 1 -- 172.21.15.22:0/329193050 learned_addr learned my addr 172.21.15.22:0/329193050 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.006+0000 7f0240a00640 1 -- 172.21.15.22:0/329193050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a6584780 msgr2=0x5614a659d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.006+0000 7f0240a00640 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a6584780 0x5614a659d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.006+0000 7f0240a00640 1 -- 172.21.15.22:0/329193050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 msgr2=0x5614a6583ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.007+0000 7f0240a00640 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a6583ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.007+0000 7f0240a00640 1 -- 172.21.15.22:0/329193050 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a647cb00 con 0x5614a657bba0 2024-03-24T06:43:01.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.008+0000 7f0240a00640 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0230047a10 tx=0x7f0230004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7174f77ba9362b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.010+0000 7f0229200640 1 -- 172.21.15.22:0/329193050 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230032800 con 0x5614a657bba0 2024-03-24T06:43:01.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.010+0000 7f0229200640 1 -- 172.21.15.22:0/329193050 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f023004d070 con 0x5614a657bba0 2024-03-24T06:43:01.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.010+0000 7f0229200640 1 -- 172.21.15.22:0/329193050 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023005b410 con 0x5614a657bba0 2024-03-24T06:43:01.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.011+0000 7f024602c740 1 -- 172.21.15.22:0/329193050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 msgr2=0x5614a65ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.013+0000 7f024602c740 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0230047a10 tx=0x7f0230004610 comp rx=0 tx=0).stop 2024-03-24T06:43:01.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 -- 172.21.15.22:0/329193050 shutdown_connections 2024-03-24T06:43:01.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a6584780 0x5614a659d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a657bba0 0x5614a65ad4a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 --2- 172.21.15.22:0/329193050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a6583ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 -- 172.21.15.22:0/329193050 >> 172.21.15.22:0/329193050 conn(0x5614a65759b0 msgr2=0x5614a6576d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.014+0000 7f024602c740 1 -- 172.21.15.22:0/329193050 shutdown_connections 2024-03-24T06:43:01.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.015+0000 7f024602c740 1 -- 172.21.15.22:0/329193050 wait complete. 2024-03-24T06:43:01.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.015+0000 7f024602c740 1 -- start start 2024-03-24T06:43:01.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.015+0000 7f024602c740 1 Processor -- start 2024-03-24T06:43:01.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.017+0000 7f024602c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a659d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.017+0000 7f024602c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.018+0000 7f024602c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a6584780 0x5614a6599200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.018+0000 7f024602c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614a6599770 con 0x5614a657b280 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.018+0000 7f024602c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614a65998e0 con 0x5614a657bba0 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.018+0000 7f024602c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614a6599aa0 con 0x5614a6584780 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.018+0000 7f024602c740 10 client.0 _ll_register_callbacks cb 0x7f0230032340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.019+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.019+0000 7f0240a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34534/0 (socket says 172.21.15.22:34534) 2024-03-24T06:43:01.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.019+0000 7f0240a00640 1 -- 172.21.15.22:0/3981456938 learned_addr learned my addr 172.21.15.22:0/3981456938 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.022+0000 7f023be00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a659d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 -- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a6584780 msgr2=0x5614a6599200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a6584780 0x5614a6599200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 -- 172.21.15.22:0/3981456938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 msgr2=0x5614a659d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a659d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 -- 172.21.15.22:0/3981456938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a64e0a70 con 0x5614a657bba0 2024-03-24T06:43:01.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f0240a00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0230009830 tx=0x7f023002bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a14c8394c78c088 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.023+0000 7f021e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:01.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.024+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230063070 con 0x5614a657bba0 2024-03-24T06:43:01.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.024+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f023005b3f0 con 0x5614a657bba0 2024-03-24T06:43:01.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.025+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02300044a0 con 0x5614a657bba0 2024-03-24T06:43:01.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.025+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0230048ed0 con 0x5614a657bba0 2024-03-24T06:43:01.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.025+0000 7f024602c740 10 client.26300 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:01.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.025+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5614a64a26a0 con 0x5614a657bba0 2024-03-24T06:43:01.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.026+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02300044a0 con 0x5614a657bba0 2024-03-24T06:43:01.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.026+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f023005a050 con 0x5614a657bba0 2024-03-24T06:43:01.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 10 client.26300 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:01.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 10 client.26300 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:43:01.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 20 client.26300 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:01.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a660dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 4 client.26300 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:43:01.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.027+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614a626d890 con 0x5614a65a57b0 2024-03-24T06:43:01.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.029+0000 7f023be00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a660dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.032+0000 7f023be00640 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a660dd20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3424a0ab50ceecdc server_cookie=ce09b344858d4322 in_seq=0 out_seq=0 2024-03-24T06:43:01.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.032+0000 7f021e000640 10 client.26300 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:43:01.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.032+0000 7f021e000640 1 -- 172.21.15.22:0/3981456938 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614a626d890 con 0x5614a65a57b0 2024-03-24T06:43:01.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.032+0000 7f021e000640 10 client.26300 handle_command_reply: tid=0 2024-03-24T06:43:01.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.033+0000 7f024602c740 1 client.26300 shutdown 2024-03-24T06:43:01.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.035+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 msgr2=0x5614a6598cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.035+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0230009830 tx=0x7f023002bbb0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.035+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 shutdown_connections 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5614a65a57b0 0x5614a660dd20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5614a6584780 0x5614a6599200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5614a657bba0 0x5614a6598cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 --2- 172.21.15.22:0/3981456938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5614a657b280 0x5614a659d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 >> 172.21.15.22:0/3981456938 conn(0x5614a65759b0 msgr2=0x5614a65772a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f021e000640 0 client.26300 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 shutdown_connections 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 1 -- 172.21.15.22:0/3981456938 wait complete. 2024-03-24T06:43:01.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.036+0000 7f024602c740 20 client.26300 trim_cache size 0 max 16384 2024-03-24T06:43:01.036 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:43:01.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 msgr2=0x7f02240a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 0x7f02240a4770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0234004990 tx=0x7f02340ce320 comp rx=0 tx=0).stop 2024-03-24T06:43:01.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 msgr2=0x7f023c06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 0x7f023c06c3c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0234005fd0 tx=0x7f023400c6e0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 shutdown_connections 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f02240a22b0 0x7f02240a4770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c06bf40 0x7f023c06c3c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f023c112550 0x7f023c073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 --2- 172.21.15.22:0/3197880618 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f023c075c40 0x7f023c072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.037+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 >> 172.21.15.22:0/3197880618 conn(0x7f023c0ff3c0 msgr2=0x7f023c1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.038+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 shutdown_connections 2024-03-24T06:43:01.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.038+0000 7f0243400640 1 -- 172.21.15.22:0/3197880618 wait complete. 2024-03-24T06:43:01.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 -- 172.21.15.22:0/576423444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff011aef0 msgr2=0x7f7ff011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 --2- 172.21.15.22:0/576423444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff011aef0 0x7f7ff011d2e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8009ca0 tx=0x7f7fe802c100 comp rx=0 tx=0).stop 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 -- 172.21.15.22:0/576423444 shutdown_connections 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 --2- 172.21.15.22:0/576423444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff011aef0 0x7f7ff011d2e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 --2- 172.21.15.22:0/576423444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 --2- 172.21.15.22:0/576423444 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ff010ff30 0x7f7ff010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.388+0000 7f7ff8400640 1 -- 172.21.15.22:0/576423444 >> 172.21.15.22:0/576423444 conn(0x7f7ff006c7f0 msgr2=0x7f7ff006cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.389+0000 7f7ff8400640 1 -- 172.21.15.22:0/576423444 shutdown_connections 2024-03-24T06:43:01.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.390+0000 7f7ff8400640 1 -- 172.21.15.22:0/576423444 wait complete. 2024-03-24T06:43:01.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.394+0000 7f7ff8400640 1 Processor -- start 2024-03-24T06:43:01.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 -- start start 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff010ff30 0x7f7ff01acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ff011aef0 0x7f7ff01a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01a7000 con 0x7f7ff011aef0 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01a7140 con 0x7f7ff010df60 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.395+0000 7f7ff8400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff01a7300 con 0x7f7ff010ff30 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34546/0 (socket says 172.21.15.22:34546) 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3528068268 learned_addr learned my addr 172.21.15.22:0/3528068268 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3528068268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff010ff30 msgr2=0x7f7ff01acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff010ff30 0x7f7ff01acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ff011aef0 msgr2=0x7f7ff01a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ff011aef0 0x7f7ff01a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.396+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3528068268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe8009900 con 0x7f7ff010df60 2024-03-24T06:43:01.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.397+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400b4d0 tx=0x7f7fe400b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.398+0000 7f7fee000640 1 -- 172.21.15.22:0/3528068268 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4010870 con 0x7f7ff010df60 2024-03-24T06:43:01.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.398+0000 7f7fee000640 1 -- 172.21.15.22:0/3528068268 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe400f3c0 con 0x7f7ff010df60 2024-03-24T06:43:01.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.399+0000 7f7fee000640 1 -- 172.21.15.22:0/3528068268 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40175c0 con 0x7f7ff010df60 2024-03-24T06:43:01.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.399+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff01a75e0 con 0x7f7ff010df60 2024-03-24T06:43:01.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.399+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff01108b0 con 0x7f7ff010df60 2024-03-24T06:43:01.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.407+0000 7f7fee000640 1 -- 172.21.15.22:0/3528068268 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7fe4017720 con 0x7f7ff010df60 2024-03-24T06:43:01.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.409+0000 7f7ffb20c740 1 Processor -- start 2024-03-24T06:43:01.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.409+0000 7f7ffb20c740 1 -- start start 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.410+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 0x562fc93c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.410+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.410+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93cf780 0x562fc93d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562fc93346e0 con 0x562fc93cf780 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562fc9331c10 con 0x562fc93c6280 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fc932d3f0 con 0x562fc93c6ba0 2024-03-24T06:43:01.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff6400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 0x562fc93c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff6400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 0x562fc93c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34562/0 (socket says 172.21.15.22:34562) 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff6400640 1 -- 172.21.15.22:0/3802774958 learned_addr learned my addr 172.21.15.22:0/3802774958 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3802774958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 msgr2=0x562fc93c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 0x562fc93c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3802774958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93cf780 msgr2=0x562fc93d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93cf780 0x562fc93d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7fee000640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 0x7f7fd80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.411+0000 7f7ff5a00640 1 -- 172.21.15.22:0/3802774958 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc92eee40 con 0x562fc93c6ba0 2024-03-24T06:43:01.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.412+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93cf240 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40cb250 tx=0x7f7fe40d67e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9709f5c8c1a94d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.412+0000 7f7fe2a00640 1 -- 172.21.15.22:0/3802774958 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe409b2b0 con 0x562fc93c6ba0 2024-03-24T06:43:01.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.413+0000 7f7fe2a00640 1 -- 172.21.15.22:0/3802774958 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe409b020 con 0x562fc93c6ba0 2024-03-24T06:43:01.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.413+0000 7f7fee000640 1 -- 172.21.15.22:0/3528068268 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7fe40cdd50 con 0x7f7ff010df60 2024-03-24T06:43:01.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.414+0000 7f7fe2a00640 1 -- 172.21.15.22:0/3802774958 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe409b2b0 con 0x562fc93c6ba0 2024-03-24T06:43:01.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.414+0000 7f7ffb20c740 1 -- 172.21.15.22:0/3802774958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 msgr2=0x562fc93cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.414+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93cf240 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40cb250 tx=0x7f7fe40d67e0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.414+0000 7f7ff6400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 0x7f7fd80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ff6400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 0x7f7fd80a4770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8004af0 tx=0x7f7fe8005d80 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 -- 172.21.15.22:0/3802774958 shutdown_connections 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93cf780 0x562fc93d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93cf240 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/3802774958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93c6280 0x562fc93c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 -- 172.21.15.22:0/3802774958 >> 172.21.15.22:0/3802774958 conn(0x562fc93c09b0 msgr2=0x562fc93c0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 -- 172.21.15.22:0/3802774958 shutdown_connections 2024-03-24T06:43:01.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.418+0000 7f7ffb20c740 1 -- 172.21.15.22:0/3802774958 wait complete. 2024-03-24T06:43:01.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.419+0000 7f7ffb20c740 1 -- start start 2024-03-24T06:43:01.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.419+0000 7f7ffb20c740 1 Processor -- start 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.420+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562fc92d2060 con 0x562fc93c6280 2024-03-24T06:43:01.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562fc92d1d00 con 0x562fc93cf780 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fc909b1c0 con 0x562fc93c6ba0 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ffb20c740 10 client.0 _ll_register_callbacks cb 0x562fc903dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35238/0 (socket says 172.21.15.22:35238) 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.421+0000 7f7ff5a00640 1 -- 172.21.15.22:0/2379364613 learned_addr learned my addr 172.21.15.22:0/2379364613 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff6400640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 -- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 msgr2=0x562fc93cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 -- 172.21.15.22:0/2379364613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 msgr2=0x562fc93cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.422+0000 7f7ff5000640 1 -- 172.21.15.22:0/2379364613 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff01108b0 con 0x562fc93c6ba0 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.423+0000 7f7fd7400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:43:01.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.423+0000 7f7ff5000640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93ccc20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7ff011d3c0 tx=0x7f7fdc00e7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=349b9d87d698da93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.423+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc005220 con 0x562fc93c6ba0 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.424+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562fc902c7e0 con 0x562fc93c6ba0 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.424+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7fdc014070 con 0x562fc93c6ba0 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.424+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7fdc005220 con 0x562fc93c6ba0 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.424+0000 7f7ffb20c740 10 client.26350 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:01.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.424+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562fc932ba70 con 0x562fc93c6ba0 2024-03-24T06:43:01.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.426+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fdc004c80 con 0x562fc93c6ba0 2024-03-24T06:43:01.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.427+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f7fdc010870 con 0x562fc93c6ba0 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 10 client.26350 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 10 client.26350 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 20 client.26350 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc93dfaa0 0x562fc93e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 4 client.26350 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:01.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.428+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fc902c7e0 con 0x562fc93dfaa0 2024-03-24T06:43:01.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.429+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc93dfaa0 0x562fc93e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.430+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc93dfaa0 0x562fc93e1f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3cf57127b7daef64 server_cookie=e3c44787475a7dca in_seq=0 out_seq=0 2024-03-24T06:43:01.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.430+0000 7f7fd7400640 10 client.26350 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:43:01.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.431+0000 7f7fd7400640 1 -- 172.21.15.22:0/2379364613 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fc902c7e0 con 0x562fc93dfaa0 2024-03-24T06:43:01.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.431+0000 7f7fd7400640 10 client.26350 handle_command_reply: tid=0 2024-03-24T06:43:01.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.431+0000 7f7ffb20c740 1 client.26350 shutdown 2024-03-24T06:43:01.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 msgr2=0x562fc93ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93ccc20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7ff011d3c0 tx=0x7f7fdc00e7e0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 shutdown_connections 2024-03-24T06:43:01.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc93dfaa0 0x562fc93e1f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93ccc20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/2379364613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7fd7400640 0 client.26350 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:43:01.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.432+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 >> 172.21.15.22:0/2379364613 conn(0x562fc93c09b0 msgr2=0x562fc93c2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.434+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 shutdown_connections 2024-03-24T06:43:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.434+0000 7f7ffb20c740 1 -- 172.21.15.22:0/2379364613 wait complete. 2024-03-24T06:43:01.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.434+0000 7f7ffb20c740 20 client.26350 trim_cache size 0 max 16384 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 Processor -- start 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 -- start start 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6ba0 0x562fc93ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93e7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562fc93db5f0 con 0x562fc93c6ba0 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562fc93c07e0 con 0x562fc93cf780 2024-03-24T06:43:01.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.472+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fc93c5480 con 0x562fc93c6280 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57036/0 (socket says 172.21.15.22:57036) 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 -- 172.21.15.22:0/644675552 learned_addr learned my addr 172.21.15.22:0/644675552 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff6400640 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 -- 172.21.15.22:0/644675552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 msgr2=0x562fc93e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 -- 172.21.15.22:0/644675552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6ba0 msgr2=0x562fc93ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6ba0 0x562fc93ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.474+0000 7f7ff5a00640 1 -- 172.21.15.22:0/644675552 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc92c7b00 con 0x562fc93c6280 2024-03-24T06:43:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.475+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40cb300 tx=0x7f7fe401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=526c4678bb5da51f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.475+0000 7f7fe2000640 1 -- 172.21.15.22:0/644675552 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400f710 con 0x562fc93c6280 2024-03-24T06:43:01.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.476+0000 7f7fe2000640 1 -- 172.21.15.22:0/644675552 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe409ae40 con 0x562fc93c6280 2024-03-24T06:43:01.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.476+0000 7f7fe2000640 1 -- 172.21.15.22:0/644675552 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe400f710 con 0x562fc93c6280 2024-03-24T06:43:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 -- 172.21.15.22:0/644675552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 msgr2=0x562fc93db0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40cb300 tx=0x7f7fe401c000 comp rx=0 tx=0).stop 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 -- 172.21.15.22:0/644675552 shutdown_connections 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6ba0 0x562fc93ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/644675552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6280 0x562fc93db0b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 -- 172.21.15.22:0/644675552 >> 172.21.15.22:0/644675552 conn(0x562fc93c09b0 msgr2=0x562fc93bf1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.480+0000 7f7ffb20c740 1 -- 172.21.15.22:0/644675552 shutdown_connections 2024-03-24T06:43:01.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.481+0000 7f7ffb20c740 1 -- 172.21.15.22:0/644675552 wait complete. 2024-03-24T06:43:01.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.481+0000 7f7ffb20c740 1 -- start start 2024-03-24T06:43:01.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.481+0000 7f7ffb20c740 1 Processor -- start 2024-03-24T06:43:01.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.484+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.484+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.485+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93d78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.485+0000 7f7ffb20c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93d27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.485+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35254/0 (socket says 172.21.15.22:35254) 2024-03-24T06:43:01.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562fc93d2e90 con 0x562fc93c6280 2024-03-24T06:43:01.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562fc93d3000 con 0x562fc93cf780 2024-03-24T06:43:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ffb20c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fc93d31c0 con 0x562fc93c6ba0 2024-03-24T06:43:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ff5a00640 1 -- 172.21.15.22:0/1719427385 learned_addr learned my addr 172.21.15.22:0/1719427385 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ffb20c740 10 client.0 _ll_register_callbacks cb 0x7f7fdc009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ff5000640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93d78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.486+0000 7f7ff6400640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93d27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff5a00640 1 -- 172.21.15.22:0/1719427385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 msgr2=0x562fc93d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93d78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff5a00640 1 -- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 msgr2=0x562fc93d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93d27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff5a00640 1 -- 172.21.15.22:0/1719427385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc92eee40 con 0x562fc93c6280 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.487+0000 7f7ff6400640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93d27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:01.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.489+0000 7f7ff5a00640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40971c0 tx=0x7f7fe400f7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4589e7752b4f14bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.490+0000 7f7fd7400640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:43:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.490+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40092b0 con 0x562fc93c6280 2024-03-24T06:43:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.490+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7fe40d6e40 con 0x562fc93c6280 2024-03-24T06:43:01.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.490+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fe409b020 con 0x562fc93c6280 2024-03-24T06:43:01.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.491+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7fe40d15e0 con 0x562fc93c6280 2024-03-24T06:43:01.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.491+0000 7f7ffb20c740 10 client.16029 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:01.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.491+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562fc92ed6a0 con 0x562fc93c6280 2024-03-24T06:43:01.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.491+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe409b020 con 0x562fc93c6280 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f7fe4097b50 con 0x562fc93c6280 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 10 client.16029 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 10 client.16029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 20 client.16029 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc9445bc0 0x562fc9447fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 4 client.16029 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.492+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fc9069720 con 0x562fc9445bc0 2024-03-24T06:43:01.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.493+0000 7f7ff5000640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc9445bc0 0x562fc9447fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ff5000640 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc9445bc0 0x562fc9447fe0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=967498afa7d898c0 server_cookie=72fc97effec57991 in_seq=0 out_seq=0 2024-03-24T06:43:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7fd7400640 10 client.16029 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:43:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7fd7400640 1 -- 172.21.15.22:0/1719427385 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fc9069720 con 0x562fc9445bc0 2024-03-24T06:43:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7fd7400640 10 client.16029 handle_command_reply: tid=0 2024-03-24T06:43:01.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ffb20c740 1 client.16029 shutdown 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 msgr2=0x562fc93d7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40971c0 tx=0x7f7fe400f7e0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 shutdown_connections 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.496+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x562fc9445bc0 0x562fc9447fe0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562fc93cf780 0x562fc93d27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fc93c6ba0 0x562fc93d78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7fd7400640 0 client.16029 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 --2- 172.21.15.22:0/1719427385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562fc93c6280 0x562fc93d7370 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 >> 172.21.15.22:0/1719427385 conn(0x562fc93c09b0 msgr2=0x562fc93bfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 shutdown_connections 2024-03-24T06:43:01.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.497+0000 7f7ffb20c740 1 -- 172.21.15.22:0/1719427385 wait complete. 2024-03-24T06:43:01.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.498+0000 7f7ffb20c740 20 client.16029 trim_cache size 0 max 16384 2024-03-24T06:43:01.497 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 msgr2=0x7f7fd80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 0x7f7fd80a4770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8004af0 tx=0x7f7fe8005d80 comp rx=0 tx=0).stop 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 msgr2=0x7f7ff01ac800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400b4d0 tx=0x7f7fe400b990 comp rx=0 tx=0).stop 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 shutdown_connections 2024-03-24T06:43:01.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7fd80a22b0 0x7f7fd80a4770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ff011aef0 0x7f7ff01a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ff010ff30 0x7f7ff01acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 --2- 172.21.15.22:0/3528068268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ff010df60 0x7f7ff01ac800 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 >> 172.21.15.22:0/3528068268 conn(0x7f7ff006c7f0 msgr2=0x7f7ff00721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 shutdown_connections 2024-03-24T06:43:01.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.499+0000 7f7ff8400640 1 -- 172.21.15.22:0/3528068268 wait complete. 2024-03-24T06:43:01.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:43:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.849+0000 7f10dba00640 1 -- 172.21.15.22:0/2536173453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4112360 msgr2=0x7f10d4114750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.849+0000 7f10dba00640 1 --2- 172.21.15.22:0/2536173453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4112360 0x7f10d4114750 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009ca0 tx=0x7f10cc02c100 comp rx=0 tx=0).stop 2024-03-24T06:43:01.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.850+0000 7f10dba00640 1 -- 172.21.15.22:0/2536173453 shutdown_connections 2024-03-24T06:43:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.850+0000 7f10dba00640 1 --2- 172.21.15.22:0/2536173453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4112360 0x7f10d4114750 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.850+0000 7f10dba00640 1 --2- 172.21.15.22:0/2536173453 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4068ec0 0x7f10d4069320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.850+0000 7f10dba00640 1 --2- 172.21.15.22:0/2536173453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4102590 0x7f10d40688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.850+0000 7f10dba00640 1 -- 172.21.15.22:0/2536173453 >> 172.21.15.22:0/2536173453 conn(0x7f10d4077710 msgr2=0x7f10d4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.857+0000 7f10dba00640 1 -- 172.21.15.22:0/2536173453 shutdown_connections 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.857+0000 7f10dba00640 1 -- 172.21.15.22:0/2536173453 wait complete. 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.858+0000 7f10dba00640 1 Processor -- start 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.859+0000 7f10dba00640 1 -- start start 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 0x7f10d41117b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 0x7f10d4111cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4112360 0x7f10d410a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f10d410b070 con 0x7f10d4112360 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f10d410b1e0 con 0x7f10d4068ec0 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10dba00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10d410b3a0 con 0x7f10d4102590 2024-03-24T06:43:01.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 0x7f10d4111cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 0x7f10d4111cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57048/0 (socket says 172.21.15.22:57048) 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10d9000640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 0x7f10d41117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.860+0000 7f10cbe00640 1 -- 172.21.15.22:0/3784886520 learned_addr learned my addr 172.21.15.22:0/3784886520 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.861+0000 7f10d9000640 1 -- 172.21.15.22:0/3784886520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 msgr2=0x7f10d4111cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.861+0000 7f10d9000640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 0x7f10d4111cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.861+0000 7f10d9000640 1 -- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4112360 msgr2=0x7f10d410a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.862+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4112360 0x7f10d410a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.862+0000 7f10d9000640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4112360 0x7f10d410a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.862+0000 7f10d9000640 1 -- 172.21.15.22:0/3784886520 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc009900 con 0x7f10d4068ec0 2024-03-24T06:43:01.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.863+0000 7f10d9000640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 0x7f10d41117b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d720 tx=0x7f10c400dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.863+0000 7f10d2000640 1 -- 172.21.15.22:0/3784886520 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4004220 con 0x7f10d4068ec0 2024-03-24T06:43:01.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.863+0000 7f10d2000640 1 -- 172.21.15.22:0/3784886520 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c400de30 con 0x7f10d4068ec0 2024-03-24T06:43:01.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.863+0000 7f10d2000640 1 -- 172.21.15.22:0/3784886520 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4004220 con 0x7f10d4068ec0 2024-03-24T06:43:01.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.863+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d410b680 con 0x7f10d4068ec0 2024-03-24T06:43:01.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.864+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d410f510 con 0x7f10d4068ec0 2024-03-24T06:43:01.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.875+0000 7f10de781740 1 Processor -- start 2024-03-24T06:43:01.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.876+0000 7f10d2000640 1 -- 172.21.15.22:0/3784886520 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f10c40099d0 con 0x7f10d4068ec0 2024-03-24T06:43:01.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.879+0000 7f10de781740 1 -- start start 2024-03-24T06:43:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.880+0000 7f10d2000640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 0x7f10b80a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.881+0000 7f10d2000640 1 -- 172.21.15.22:0/3784886520 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f10c40cdca0 con 0x7f10d4068ec0 2024-03-24T06:43:01.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.882+0000 7f10de781740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca5c280 0x5605aca5c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.883+0000 7f10de781740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.884+0000 7f10de781740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 0x5605aca67b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.884+0000 7f10de781740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ac9ca6e0 con 0x5605aca5c280 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.884+0000 7f10de781740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ac9c7c10 con 0x5605aca65780 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.884+0000 7f10de781740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605ac9c33f0 con 0x5605aca5cba0 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.884+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 0x7f10b80a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.885+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.885+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57052/0 (socket says 172.21.15.22:57052) 2024-03-24T06:43:01.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.885+0000 7f10cbe00640 1 -- 172.21.15.22:0/3496834702 learned_addr learned my addr 172.21.15.22:0/3496834702 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.886+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 0x5605aca67b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.885+0000 7f10d9000640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca5c280 0x5605aca5c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10cbe00640 1 -- 172.21.15.22:0/3496834702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 msgr2=0x5605aca67b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 0x5605aca67b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10cbe00640 1 -- 172.21.15.22:0/3496834702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca5c280 msgr2=0x5605aca5c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca5c280 0x5605aca5c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10cbe00640 1 -- 172.21.15.22:0/3496834702 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ac984e40 con 0x5605aca5cba0 2024-03-24T06:43:01.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.887+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 0x5605aca67b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:43:01.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.888+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ab0 tx=0x7f10bc009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5161c7900f63b9b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.889+0000 7f10ca000640 1 -- 172.21.15.22:0/3496834702 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00eba0 con 0x5605aca5cba0 2024-03-24T06:43:01.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.889+0000 7f10ca000640 1 -- 172.21.15.22:0/3496834702 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc00ed00 con 0x5605aca5cba0 2024-03-24T06:43:01.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.889+0000 7f10ca000640 1 -- 172.21.15.22:0/3496834702 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0126c0 con 0x5605aca5cba0 2024-03-24T06:43:01.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.890+0000 7f10de781740 1 -- 172.21.15.22:0/3496834702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 msgr2=0x5605aca65240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.890+0000 7f10de781740 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ab0 tx=0x7f10bc009f70 comp rx=0 tx=0).stop 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.891+0000 7f10de781740 1 -- 172.21.15.22:0/3496834702 shutdown_connections 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.892+0000 7f10de781740 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca65780 0x5605aca67b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.892+0000 7f10de781740 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca65240 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.892+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 0x7f10b80a4850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5605aca57b40 tx=0x7f10cc0093d0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.892+0000 7f10de781740 1 --2- 172.21.15.22:0/3496834702 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca5c280 0x5605aca5c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.892+0000 7f10de781740 1 -- 172.21.15.22:0/3496834702 >> 172.21.15.22:0/3496834702 conn(0x5605aca569b0 msgr2=0x5605aca56d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.893+0000 7f10de781740 1 -- 172.21.15.22:0/3496834702 shutdown_connections 2024-03-24T06:43:01.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.905+0000 7f10de781740 1 -- 172.21.15.22:0/3496834702 wait complete. 2024-03-24T06:43:01.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.905+0000 7f10de781740 1 -- start start 2024-03-24T06:43:01.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.915+0000 7f10de781740 1 Processor -- start 2024-03-24T06:43:01.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.916+0000 7f10de781740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10de781740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10d9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10d9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34636/0 (socket says 172.21.15.22:34636) 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10de781740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca632d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca62d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57060/0 (socket says 172.21.15.22:57060) 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.917+0000 7f10d9000640 1 -- 172.21.15.22:0/2911610861 learned_addr learned my addr 172.21.15.22:0/2911610861 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605ac968060 con 0x5605aca65780 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ac967d00 con 0x5605aca5c280 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605ac7311c0 con 0x5605aca5cba0 2024-03-24T06:43:01.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10de781740 10 client.0 _ll_register_callbacks cb 0x5605ac6d3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:01.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10d9000640 1 -- 172.21.15.22:0/2911610861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 msgr2=0x5605aca62d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10d9000640 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca62d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10d9000640 1 -- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 msgr2=0x5605aca632d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.918+0000 7f10d9000640 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.919+0000 7f10d9000640 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc0096e0 con 0x5605aca5c280 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.919+0000 7f10d9000640 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10c409a260 tx=0x7f10c409ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04f093508921e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.919+0000 7f10c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:01.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.920+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c40a2a70 con 0x5605aca5c280 2024-03-24T06:43:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.920+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f10c4094e40 con 0x5605aca5c280 2024-03-24T06:43:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.920+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5605ac6c27e0 con 0x5605aca5c280 2024-03-24T06:43:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.921+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f10c4009c80 con 0x5605aca5c280 2024-03-24T06:43:01.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.922+0000 7f10de781740 10 client.26327 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:01.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.922+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5605ac9c1a70 con 0x5605aca5c280 2024-03-24T06:43:01.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.923+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5605ac6c27e0 con 0x5605aca5c280 2024-03-24T06:43:01.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.923+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f10c4099720 con 0x5605aca5c280 2024-03-24T06:43:01.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.925+0000 7f10de781740 10 client.26327 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:01.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.925+0000 7f10de781740 10 client.26327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:01.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.925+0000 7f10de781740 20 client.26327 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:01.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.925+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605aca75ca0 0x5605aca780e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.926+0000 7f10de781740 4 client.26327 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:01.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.926+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605ac6ff720 con 0x5605aca75ca0 2024-03-24T06:43:01.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.928+0000 7f10cbe00640 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605aca75ca0 0x5605aca780e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.929+0000 7f10cbe00640 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605aca75ca0 0x5605aca780e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f93fd62bb7304ca server_cookie=641294be6f7de7d0 in_seq=0 out_seq=0 2024-03-24T06:43:01.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.930+0000 7f10c2a00640 10 client.26327 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:01.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.931+0000 7f10c2a00640 1 -- 172.21.15.22:0/2911610861 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605ac6ff720 con 0x5605aca75ca0 2024-03-24T06:43:01.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.932+0000 7f10c2a00640 10 client.26327 handle_command_reply: tid=0 2024-03-24T06:43:01.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.932+0000 7f10de781740 1 client.26327 shutdown 2024-03-24T06:43:01.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.933+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 msgr2=0x5605aca62850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.933+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10c409a260 tx=0x7f10c409ab10 comp rx=0 tx=0).stop 2024-03-24T06:43:01.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.933+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 shutdown_connections 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.933+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.933+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca62d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.934+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605aca75ca0 0x5605aca780e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.934+0000 7f10de781740 1 --2- 172.21.15.22:0/2911610861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca62850 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.934+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 >> 172.21.15.22:0/2911610861 conn(0x5605aca569b0 msgr2=0x5605aca58700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.934+0000 7f10c2a00640 0 client.26327 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:01.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.934+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 shutdown_connections 2024-03-24T06:43:01.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.935+0000 7f10de781740 1 -- 172.21.15.22:0/2911610861 wait complete. 2024-03-24T06:43:01.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.936+0000 7f10de781740 20 client.26327 trim_cache size 0 max 16384 2024-03-24T06:43:01.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.974+0000 7f10de781740 1 Processor -- start 2024-03-24T06:43:01.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.974+0000 7f10de781740 1 -- start start 2024-03-24T06:43:01.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.975+0000 7f10de781740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 0x5605aca71010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.975+0000 7f10de781740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.975+0000 7f10de781740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca7d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.976+0000 7f10de781740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605aca5bee0 con 0x5605aca65780 2024-03-24T06:43:01.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.976+0000 7f10de781740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605aca5e450 con 0x5605aca5cba0 2024-03-24T06:43:01.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.976+0000 7f10de781740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605aca5ba20 con 0x5605aca5c280 2024-03-24T06:43:01.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.976+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.976+0000 7f10d9000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 0x5605aca71010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.977+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34650/0 (socket says 172.21.15.22:34650) 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.977+0000 7f10cbe00640 1 -- 172.21.15.22:0/3575988141 learned_addr learned my addr 172.21.15.22:0/3575988141 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10cbe00640 1 -- 172.21.15.22:0/3575988141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 msgr2=0x5605aca71010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 0x5605aca71010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10cbe00640 1 -- 172.21.15.22:0/3575988141 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 msgr2=0x5605aca7d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca7d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10cbe00640 1 -- 172.21.15.22:0/3575988141 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ac95db00 con 0x5605aca5cba0 2024-03-24T06:43:01.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.978+0000 7f10d9000640 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 0x5605aca71010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:01.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.979+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ab0 tx=0x7f10bc013f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a74596a6f651e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.981+0000 7f10c9600640 1 -- 172.21.15.22:0/3575988141 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc02e070 con 0x5605aca5cba0 2024-03-24T06:43:01.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.982+0000 7f10c9600640 1 -- 172.21.15.22:0/3575988141 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc023040 con 0x5605aca5cba0 2024-03-24T06:43:01.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.982+0000 7f10c9600640 1 -- 172.21.15.22:0/3575988141 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc021040 con 0x5605aca5cba0 2024-03-24T06:43:01.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 -- 172.21.15.22:0/3575988141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 msgr2=0x5605aca558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:01.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ab0 tx=0x7f10bc013f90 comp rx=0 tx=0).stop 2024-03-24T06:43:01.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 -- 172.21.15.22:0/3575988141 shutdown_connections 2024-03-24T06:43:01.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605aca7d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5cba0 0x5605aca558f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 --2- 172.21.15.22:0/3575988141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5c280 0x5605aca71010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.983+0000 7f10de781740 1 -- 172.21.15.22:0/3575988141 >> 172.21.15.22:0/3575988141 conn(0x5605aca569b0 msgr2=0x5605aca551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.984+0000 7f10de781740 1 -- 172.21.15.22:0/3575988141 shutdown_connections 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.984+0000 7f10de781740 1 -- 172.21.15.22:0/3575988141 wait complete. 2024-03-24T06:43:01.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.984+0000 7f10de781740 1 -- start start 2024-03-24T06:43:01.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.985+0000 7f10de781740 1 Processor -- start 2024-03-24T06:43:01.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.985+0000 7f10de781740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.985+0000 7f10de781740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca5ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.986+0000 7f10d9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.986+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca5ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.986+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca5ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57086/0 (socket says 172.21.15.22:57086) 2024-03-24T06:43:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.988+0000 7f10d9000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34658/0 (socket says 172.21.15.22:34658) 2024-03-24T06:43:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.988+0000 7f10cbe00640 1 -- 172.21.15.22:0/3008797112 learned_addr learned my addr 172.21.15.22:0/3008797112 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:01.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.988+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605acadb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.988+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5605aca60630 con 0x5605aca65780 2024-03-24T06:43:01.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.989+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605acadb560 con 0x5605aca5c280 2024-03-24T06:43:01.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.989+0000 7f10d9a00640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605acadb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:01.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.989+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605acadb6a0 con 0x5605aca5cba0 2024-03-24T06:43:01.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.989+0000 7f10de781740 10 client.0 _ll_register_callbacks cb 0x7f10c4096570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:01.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.992+0000 7f10d9000640 1 -- 172.21.15.22:0/3008797112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 msgr2=0x5605aca5ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.992+0000 7f10d9000640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca5ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.992+0000 7f10d9000640 1 -- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 msgr2=0x5605acadb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:01.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.992+0000 7f10d9000640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605acadb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:01.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.992+0000 7f10d9000640 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ac9c1a70 con 0x5605aca5c280 2024-03-24T06:43:01.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.993+0000 7f10d9000640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f10c4005fd0 tx=0x7f10c409a9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb241eca3435a540 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:01.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.993+0000 7f10c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:01.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.995+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4099930 con 0x5605aca5c280 2024-03-24T06:43:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.995+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f10c4094e40 con 0x5605aca5c280 2024-03-24T06:43:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.995+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10bc00ed00 con 0x5605aca5c280 2024-03-24T06:43:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.996+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f10c4099c30 con 0x5605aca5c280 2024-03-24T06:43:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.996+0000 7f10de781740 10 client.26333 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:01.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.996+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5605ac984e40 con 0x5605aca5c280 2024-03-24T06:43:01.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c40a6080 con 0x5605aca5c280 2024-03-24T06:43:01.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f10c40cb2b0 con 0x5605aca5c280 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 10 client.26333 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 20 client.26333 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605acadcc40 0x5605acadf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 4 client.26333 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:43:01.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:01.998+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f10bc00ed00 con 0x5605acadcc40 2024-03-24T06:43:02.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.006+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605acadcc40 0x5605acadf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:02.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.009+0000 7f10cbe00640 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605acadcc40 0x5605acadf060 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de6ab172e3a898e0 server_cookie=60ea766d7e61efcf in_seq=0 out_seq=0 2024-03-24T06:43:02.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.010+0000 7f10c2a00640 10 client.26333 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:02.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.011+0000 7f10c2a00640 1 -- 172.21.15.22:0/3008797112 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f10bc00ed00 con 0x5605acadcc40 2024-03-24T06:43:02.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.011+0000 7f10c2a00640 10 client.26333 handle_command_reply: tid=0 2024-03-24T06:43:02.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.011+0000 7f10de781740 1 client.26333 shutdown 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 msgr2=0x5605aca5fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f10c4005fd0 tx=0x7f10c409a9f0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 shutdown_connections 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5605aca65780 0x5605acadb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605aca5cba0 0x5605aca5ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5605acadcc40 0x5605acadf060 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 --2- 172.21.15.22:0/3008797112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605aca5c280 0x5605aca5fa40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 >> 172.21.15.22:0/3008797112 conn(0x5605aca569b0 msgr2=0x5605aca55a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:02.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.012+0000 7f10c2a00640 0 client.26333 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:02.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.013+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 shutdown_connections 2024-03-24T06:43:02.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.014+0000 7f10de781740 1 -- 172.21.15.22:0/3008797112 wait complete. 2024-03-24T06:43:02.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.014+0000 7f10de781740 20 client.26333 trim_cache size 0 max 16384 2024-03-24T06:43:02.015 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:43:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.016+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 msgr2=0x7f10b80a4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.016+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 0x7f10b80a4850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5605aca57b40 tx=0x7f10cc0093d0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.016+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 msgr2=0x7f10d41117b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:02.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.016+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 0x7f10d41117b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10c400d720 tx=0x7f10c400dbe0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 shutdown_connections 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f10b80a2390 0x7f10b80a4850 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10d4112360 0x7f10d410a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10d4102590 0x7f10d4111cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 --2- 172.21.15.22:0/3784886520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d4068ec0 0x7f10d41117b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 >> 172.21.15.22:0/3784886520 conn(0x7f10d4077710 msgr2=0x7f10d40ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 shutdown_connections 2024-03-24T06:43:02.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:02.017+0000 7f10dba00640 1 -- 172.21.15.22:0/3784886520 wait complete. 2024-03-24T06:43:02.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:01 smithi022 ceph-mon[32003]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 58 MiB/s wr, 7.46k op/s 2024-03-24T06:43:02.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:01 smithi143 ceph-mon[36520]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 58 MiB/s wr, 7.46k op/s 2024-03-24T06:43:02.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:01 smithi155 ceph-mon[36262]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 58 MiB/s wr, 7.46k op/s 2024-03-24T06:43:03.050 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:43:03.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:43:03.050 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:43:03.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.408+0000 7f8861000640 1 -- 172.21.15.22:0/899068013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1098c0 msgr2=0x7f885c109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.408+0000 7f8861000640 1 --2- 172.21.15.22:0/899068013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1098c0 0x7f885c109ca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f884c009900 tx=0x7f884c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:43:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 -- 172.21.15.22:0/899068013 shutdown_connections 2024-03-24T06:43:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 --2- 172.21.15.22:0/899068013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c104270 0x7f885c110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 --2- 172.21.15.22:0/899068013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1038b0 0x7f885c103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 --2- 172.21.15.22:0/899068013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1098c0 0x7f885c109ca0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 -- 172.21.15.22:0/899068013 >> 172.21.15.22:0/899068013 conn(0x7f885c0ff380 msgr2=0x7f885c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 -- 172.21.15.22:0/899068013 shutdown_connections 2024-03-24T06:43:03.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.409+0000 7f8861000640 1 -- 172.21.15.22:0/899068013 wait complete. 2024-03-24T06:43:03.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.411+0000 7f8861000640 1 Processor -- start 2024-03-24T06:43:03.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.411+0000 7f8861000640 1 -- start start 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 0x7f885c076160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c1098c0 0x7f885c0766a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1aa570 con 0x7f885c104270 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1aa6b0 con 0x7f885c1038b0 2024-03-24T06:43:03.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.412+0000 7f8861000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1aa820 con 0x7f885c1098c0 2024-03-24T06:43:03.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34666/0 (socket says 172.21.15.22:34666) 2024-03-24T06:43:03.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 -- 172.21.15.22:0/1895171501 learned_addr learned my addr 172.21.15.22:0/1895171501 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f885a800640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c1098c0 0x7f885c0766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859400640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 0x7f885c076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 -- 172.21.15.22:0/1895171501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c1098c0 msgr2=0x7f885c0766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c1098c0 0x7f885c0766a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 -- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 msgr2=0x7f885c076160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 0x7f885c076160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859e00640 1 -- 172.21.15.22:0/1895171501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c009560 con 0x7f885c1038b0 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.413+0000 7f8859400640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 0x7f885c076160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:43:03.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.414+0000 7f8859e00640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f884c02c240 tx=0x7f884c004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.414+0000 7f8852a00640 1 -- 172.21.15.22:0/1895171501 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c02dea0 con 0x7f885c1038b0 2024-03-24T06:43:03.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.415+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c1aaaa0 con 0x7f885c1038b0 2024-03-24T06:43:03.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.415+0000 7f8852a00640 1 -- 172.21.15.22:0/1895171501 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884c0043e0 con 0x7f885c1038b0 2024-03-24T06:43:03.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.415+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1ab000 con 0x7f885c1038b0 2024-03-24T06:43:03.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.415+0000 7f8852a00640 1 -- 172.21.15.22:0/1895171501 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c003c20 con 0x7f885c1038b0 2024-03-24T06:43:03.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.423+0000 7f8852a00640 1 -- 172.21.15.22:0/1895171501 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f884c004df0 con 0x7f885c1038b0 2024-03-24T06:43:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.426+0000 7f8863dcb740 1 Processor -- start 2024-03-24T06:43:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.426+0000 7f8852a00640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 0x7f88400a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.427+0000 7f8852a00640 1 -- 172.21.15.22:0/1895171501 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884c0f04b0 con 0x7f885c1038b0 2024-03-24T06:43:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 -- start start 2024-03-24T06:43:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac08210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac11330 0x55fcfac13720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fcfab762d0 con 0x55fcfac07e30 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fcfab73800 con 0x55fcfac08750 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.429+0000 7f8863dcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fcfab6efe0 con 0x55fcfac11330 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac08210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34672/0 (socket says 172.21.15.22:34672) 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 -- 172.21.15.22:0/2523253685 learned_addr learned my addr 172.21.15.22:0/2523253685 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f885a800640 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac11330 0x55fcfac13720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 -- 172.21.15.22:0/2523253685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac11330 msgr2=0x55fcfac13720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac11330 0x55fcfac13720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 -- 172.21.15.22:0/2523253685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 msgr2=0x55fcfac08210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac08210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.430+0000 7f8859400640 1 -- 172.21.15.22:0/2523253685 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcfab30e90 con 0x55fcfac08750 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.431+0000 7f885a800640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 0x7f88400a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.431+0000 7f8859400640 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f884800bce0 tx=0x7f884800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b011cb0c037cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.431+0000 7f8847400640 1 -- 172.21.15.22:0/2523253685 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884801a460 con 0x55fcfac08750 2024-03-24T06:43:03.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.432+0000 7f8847400640 1 -- 172.21.15.22:0/2523253685 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884801aaa0 con 0x55fcfac08750 2024-03-24T06:43:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.433+0000 7f8863dcb740 1 -- 172.21.15.22:0/2523253685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 msgr2=0x55fcfac10df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.433+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f884800bce0 tx=0x7f884800f730 comp rx=0 tx=0).stop 2024-03-24T06:43:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.433+0000 7f885a800640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 0x7f88400a4730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55fcfac036f0 tx=0x7f8854004a40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 -- 172.21.15.22:0/2523253685 shutdown_connections 2024-03-24T06:43:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac11330 0x55fcfac13720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac08750 0x55fcfac10df0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2523253685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac08210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 -- 172.21.15.22:0/2523253685 >> 172.21.15.22:0/2523253685 conn(0x55fcfac02560 msgr2=0x55fcfac02940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 -- 172.21.15.22:0/2523253685 shutdown_connections 2024-03-24T06:43:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.434+0000 7f8863dcb740 1 -- 172.21.15.22:0/2523253685 wait complete. 2024-03-24T06:43:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.435+0000 7f8863dcb740 1 -- start start 2024-03-24T06:43:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.435+0000 7f8863dcb740 1 Processor -- start 2024-03-24T06:43:03.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.436+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.437+0000 7f8859e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.437+0000 7f8859e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34686/0 (socket says 172.21.15.22:34686) 2024-03-24T06:43:03.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.437+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.437+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 0x55fcfac0ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f885a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 0x55fcfac0ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f885a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 0x55fcfac0ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35352/0 (socket says 172.21.15.22:35352) 2024-03-24T06:43:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f885a800640 1 -- 172.21.15.22:0/2411665448 learned_addr learned my addr 172.21.15.22:0/2411665448 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fcfab13fc0 con 0x55fcfac11330 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fcfab13c60 con 0x55fcfac07e30 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.438+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fcfa8b70c0 con 0x55fcfac08750 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.439+0000 7f8863dcb740 10 client.0 _ll_register_callbacks cb 0x55fcfab2be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.439+0000 7f8859400640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.440+0000 7f8859e00640 1 -- 172.21.15.22:0/2411665448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 msgr2=0x55fcfac0e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.440+0000 7f8859e00640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.440+0000 7f8859e00640 1 -- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 msgr2=0x55fcfac0ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.440+0000 7f8859e00640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 0x55fcfac0ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.440+0000 7f8859e00640 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884800b940 con 0x55fcfac07e30 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.441+0000 7f8859e00640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f884c0ed9b0 tx=0x7f884c0bc5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e5b09a27c57be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.441+0000 7f883be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.441+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c0c0da0 con 0x55fcfac07e30 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.441+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884c0a7040 con 0x55fcfac07e30 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.441+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fcfab93e90 con 0x55fcfac07e30 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.442+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f884c0c53f0 con 0x55fcfac07e30 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.442+0000 7f8863dcb740 10 client.26345 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.442+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fcfab6d660 con 0x55fcfac07e30 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.442+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fcfab93e90 con 0x55fcfac07e30 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f884c03c070 con 0x55fcfac07e30 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 10 client.26345 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 20 client.26345 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfac216e0 0x55fcfac23b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 4 client.26345 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcfab65210 con 0x55fcfac216e0 2024-03-24T06:43:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.443+0000 7f8859400640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfac216e0 0x55fcfac23b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.445+0000 7f8859400640 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfac216e0 0x55fcfac23b00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6821269c1058b026 server_cookie=e19c8fd254ddc242 in_seq=0 out_seq=0 2024-03-24T06:43:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.445+0000 7f883be00640 10 client.26345 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.447+0000 7f883be00640 1 -- 172.21.15.22:0/2411665448 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcfab65210 con 0x55fcfac216e0 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.447+0000 7f883be00640 10 client.26345 handle_command_reply: tid=0 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.447+0000 7f8863dcb740 1 client.26345 shutdown 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 msgr2=0x55fcfac0e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f884c0ed9b0 tx=0x7f884c0bc5b0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 shutdown_connections 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac11330 0x55fcfac0ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfac216e0 0x55fcfac23b00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 --2- 172.21.15.22:0/2411665448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac07e30 0x55fcfac0e2c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 >> 172.21.15.22:0/2411665448 conn(0x55fcfac02560 msgr2=0x55fcfac042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.448+0000 7f883be00640 0 client.26345 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.449+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 shutdown_connections 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.449+0000 7f8863dcb740 1 -- 172.21.15.22:0/2411665448 wait complete. 2024-03-24T06:43:03.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.449+0000 7f8863dcb740 20 client.26345 trim_cache size 0 max 16384 2024-03-24T06:43:03.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.486+0000 7f8863dcb740 1 Processor -- start 2024-03-24T06:43:03.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 -- start start 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 0x55fcfac1ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac08750 0x55fcfac0eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfac293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fcfac1e290 con 0x55fcfac08750 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fcfac03560 con 0x55fcfac11330 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.487+0000 7f8863dcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fcfac1d1f0 con 0x55fcfac07e30 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.488+0000 7f8859e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 0x55fcfac1ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.488+0000 7f885a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfac293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.488+0000 7f885a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfac293b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34698/0 (socket says 172.21.15.22:34698) 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.488+0000 7f885a800640 1 -- 172.21.15.22:0/706602051 learned_addr learned my addr 172.21.15.22:0/706602051 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.489+0000 7f8859400640 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac08750 0x55fcfac0eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 -- 172.21.15.22:0/706602051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 msgr2=0x55fcfac293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfac293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 -- 172.21.15.22:0/706602051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac08750 msgr2=0x55fcfac0eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac08750 0x55fcfac0eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 -- 172.21.15.22:0/706602051 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcfab09ae0 con 0x55fcfac07e30 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.490+0000 7f8859e00640 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 0x55fcfac1ccb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f884c0eda60 tx=0x7f884c003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9089252cf5ad25cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.492+0000 7f8846a00640 1 -- 172.21.15.22:0/706602051 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c0b8ef0 con 0x55fcfac07e30 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.492+0000 7f8846a00640 1 -- 172.21.15.22:0/706602051 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884c0c0da0 con 0x55fcfac07e30 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.492+0000 7f8846a00640 1 -- 172.21.15.22:0/706602051 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c005610 con 0x55fcfac07e30 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 -- 172.21.15.22:0/706602051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 msgr2=0x55fcfac1ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 0x55fcfac1ccb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f884c0eda60 tx=0x7f884c003f90 comp rx=0 tx=0).stop 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 -- 172.21.15.22:0/706602051 shutdown_connections 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfac293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac08750 0x55fcfac0eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 --2- 172.21.15.22:0/706602051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac07e30 0x55fcfac1ccb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 -- 172.21.15.22:0/706602051 >> 172.21.15.22:0/706602051 conn(0x55fcfac02560 msgr2=0x55fcfac015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 -- 172.21.15.22:0/706602051 shutdown_connections 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.493+0000 7f8863dcb740 1 -- 172.21.15.22:0/706602051 wait complete. 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.494+0000 7f8863dcb740 1 -- start start 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 Processor -- start 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac0b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fcfac0be70 con 0x55fcfac07e30 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fcfac0bfe0 con 0x55fcfac11330 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fcfaccaf30 con 0x55fcfac08750 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.495+0000 7f8863dcb740 10 client.0 _ll_register_callbacks cb 0x7f884c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:03.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.496+0000 7f8859400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.496+0000 7f885a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f885a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34710/0 (socket says 172.21.15.22:34710) 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f885a800640 1 -- 172.21.15.22:0/1880968844 learned_addr learned my addr 172.21.15.22:0/1880968844 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f8859e00640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac0b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f885a800640 1 -- 172.21.15.22:0/1880968844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 msgr2=0x55fcfac0b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f885a800640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.497+0000 7f885a800640 1 -- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 msgr2=0x55fcfac0b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.498+0000 7f885a800640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac0b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.498+0000 7f885a800640 1 -- 172.21.15.22:0/1880968844 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcfab6d660 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.499+0000 7f885a800640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8854009900 tx=0x7f8854009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9887db921ca39634 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.499+0000 7f883be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.500+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885401f040 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.501+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f88540173b0 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.502+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f884801aaa0 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.503+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f88540245f0 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.503+0000 7f8863dcb740 10 client.26351 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.503+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fcfab2f6f0 con 0x55fcfac11330 2024-03-24T06:43:03.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.503+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884801aaa0 con 0x55fcfac11330 2024-03-24T06:43:03.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f8854017b00 con 0x55fcfac11330 2024-03-24T06:43:03.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 10 client.26351 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 10 client.26351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 20 client.26351 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfaccc930 0x55fcfacced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 4 client.26351 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.504+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fcfa86e860 con 0x55fcfaccc930 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.505+0000 7f8859e00640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfaccc930 0x55fcfacced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.507+0000 7f8859e00640 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfaccc930 0x55fcfacced50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe2e03a485642dc server_cookie=fbe062052108fa97 in_seq=0 out_seq=0 2024-03-24T06:43:03.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.508+0000 7f883be00640 10 client.26351 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.513+0000 7f883be00640 1 -- 172.21.15.22:0/1880968844 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fcfa86e860 con 0x55fcfaccc930 2024-03-24T06:43:03.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.513+0000 7f883be00640 10 client.26351 handle_command_reply: tid=0 2024-03-24T06:43:03.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.513+0000 7f8863dcb740 1 client.26351 shutdown 2024-03-24T06:43:03.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.513+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 msgr2=0x55fcfacca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.513+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8854009900 tx=0x7f8854009650 comp rx=0 tx=0).stop 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 shutdown_connections 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55fcfaccc930 0x55fcfacced50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fcfac11330 0x55fcfacca9f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcfac08750 0x55fcfac0b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 --2- 172.21.15.22:0/1880968844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fcfac07e30 0x55fcfac0b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 >> 172.21.15.22:0/1880968844 conn(0x55fcfac02560 msgr2=0x55fcfac03720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.514+0000 7f883be00640 0 client.26351 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.515+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 shutdown_connections 2024-03-24T06:43:03.516 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:43:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:43:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d", 2024-03-24T06:43:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:43:03.516 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.515+0000 7f8863dcb740 1 -- 172.21.15.22:0/1880968844 wait complete. 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.515+0000 7f8863dcb740 20 client.26351 trim_cache size 0 max 16384 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.516+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 msgr2=0x7f88400a4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.516+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 0x7f88400a4730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55fcfac036f0 tx=0x7f8854004a40 comp rx=0 tx=0).stop 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.516+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 msgr2=0x7f885c075c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.516+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f884c02c240 tx=0x7f884c004060 comp rx=0 tx=0).stop 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 shutdown_connections 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2270 0x7f88400a4730 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c1098c0 0x7f885c0766a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c104270 0x7f885c076160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 --2- 172.21.15.22:0/1895171501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c1038b0 0x7f885c075c20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 >> 172.21.15.22:0/1895171501 conn(0x7f885c0ff380 msgr2=0x7f885c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 shutdown_connections 2024-03-24T06:43:03.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.517+0000 7f8861000640 1 -- 172.21.15.22:0/1895171501 wait complete. 2024-03-24T06:43:03.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:43:03.548 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:43:03.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.924+0000 7f04bbe00640 1 -- 172.21.15.22:0/4194677530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 msgr2=0x7f04b4110780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.924+0000 7f04bbe00640 1 --2- 172.21.15.22:0/4194677530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b4110780 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ca0 tx=0x7f04ac02c100 comp rx=0 tx=0).stop 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 -- 172.21.15.22:0/4194677530 shutdown_connections 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 --2- 172.21.15.22:0/4194677530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b4110780 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 --2- 172.21.15.22:0/4194677530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41038b0 0x7f04b4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 --2- 172.21.15.22:0/4194677530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41098c0 0x7f04b4109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 -- 172.21.15.22:0/4194677530 >> 172.21.15.22:0/4194677530 conn(0x7f04b40ff360 msgr2=0x7f04b4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 -- 172.21.15.22:0/4194677530 shutdown_connections 2024-03-24T06:43:03.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.925+0000 7f04bbe00640 1 -- 172.21.15.22:0/4194677530 wait complete. 2024-03-24T06:43:03.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.926+0000 7f04bbe00640 1 Processor -- start 2024-03-24T06:43:03.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.927+0000 7f04bbe00640 1 -- start start 2024-03-24T06:43:03.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.927+0000 7f04bbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41038b0 0x7f04b4199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04bbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41098c0 0x7f04b419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04bbe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f04b419ee20 con 0x7f04b41038b0 2024-03-24T06:43:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04bbe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f04b419ef90 con 0x7f04b4104270 2024-03-24T06:43:03.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04bbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f04b419f150 con 0x7f04b41098c0 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04b8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04b8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34736/0 (socket says 172.21.15.22:34736) 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04b8a00640 1 -- 172.21.15.22:0/777244406 learned_addr learned my addr 172.21.15.22:0/777244406 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.928+0000 7f04b9400640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41038b0 0x7f04b4199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.929+0000 7f04b9e00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41098c0 0x7f04b419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 -- 172.21.15.22:0/777244406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41098c0 msgr2=0x7f04b419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41098c0 0x7f04b419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 -- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41038b0 msgr2=0x7f04b4199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41038b0 0x7f04b4199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 -- 172.21.15.22:0/777244406 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac009900 con 0x7f04b4104270 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.930+0000 7f04b8a00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d950 tx=0x7f04a000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.931+0000 7f04b2000640 1 -- 172.21.15.22:0/777244406 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0009c30 con 0x7f04b4104270 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.931+0000 7f04b2000640 1 -- 172.21.15.22:0/777244406 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04a0009d90 con 0x7f04b4104270 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.931+0000 7f04b2000640 1 -- 172.21.15.22:0/777244406 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a000f7e0 con 0x7f04b4104270 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.931+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b419f430 con 0x7f04b4104270 2024-03-24T06:43:03.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.931+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41a6cb0 con 0x7f04b4104270 2024-03-24T06:43:03.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 Processor -- start 2024-03-24T06:43:03.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04b2000640 1 -- 172.21.15.22:0/777244406 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f04a0010ad0 con 0x7f04b4104270 2024-03-24T06:43:03.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 -- start start 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792a8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d179216790 con 0x55d1792b10f0 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d179213cc0 con 0x55d1792a8a60 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.938+0000 7f04beba7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d17920f4a0 con 0x55d1792a7d80 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.939+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.939+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34738/0 (socket says 172.21.15.22:34738) 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.939+0000 7f04b9e00640 1 -- 172.21.15.22:0/1648178346 learned_addr learned my addr 172.21.15.22:0/1648178346 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.939+0000 7f04b9400640 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.939+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b9e00640 1 -- 172.21.15.22:0/1648178346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 msgr2=0x55d1792a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792a8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b9e00640 1 -- 172.21.15.22:0/1648178346 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 msgr2=0x55d1792b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b9e00640 1 -- 172.21.15.22:0/1648178346 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1791d0c20 con 0x55d1792a8a60 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b2000640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 0x7f04980a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.940+0000 7f04b2000640 1 -- 172.21.15.22:0/777244406 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04a00cdf20 con 0x7f04b4104270 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.941+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ca0 tx=0x7f04ac032d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89308578fc55d83e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.941+0000 7f04b9400640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 0x7f04980a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.941+0000 7f04aa600640 1 -- 172.21.15.22:0/1648178346 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0326c0 con 0x55d1792a8a60 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.941+0000 7f04aa600640 1 -- 172.21.15.22:0/1648178346 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04ac032820 con 0x55d1792a8a60 2024-03-24T06:43:03.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.941+0000 7f04aa600640 1 -- 172.21.15.22:0/1648178346 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac003e40 con 0x55d1792a8a60 2024-03-24T06:43:03.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.942+0000 7f04beba7740 1 -- 172.21.15.22:0/1648178346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 msgr2=0x55d1792a77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.942+0000 7f04beba7740 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ca0 tx=0x7f04ac032d80 comp rx=0 tx=0).stop 2024-03-24T06:43:03.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.943+0000 7f04beba7740 1 -- 172.21.15.22:0/1648178346 shutdown_connections 2024-03-24T06:43:03.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.943+0000 7f04beba7740 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.943+0000 7f04beba7740 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.943+0000 7f04beba7740 1 --2- 172.21.15.22:0/1648178346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792a77b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.943+0000 7f04beba7740 1 -- 172.21.15.22:0/1648178346 >> 172.21.15.22:0/1648178346 conn(0x55d17926eb30 msgr2=0x55d1792a27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.944+0000 7f04b9400640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 0x7f04980a48c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55d1792a8fb0 tx=0x7f04a4005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.944+0000 7f04beba7740 1 -- 172.21.15.22:0/1648178346 shutdown_connections 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.944+0000 7f04beba7740 1 -- 172.21.15.22:0/1648178346 wait complete. 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.945+0000 7f04beba7740 1 -- start start 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.945+0000 7f04beba7740 1 Processor -- start 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.947+0000 7f04beba7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04beba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04b9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04b9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57192/0 (socket says 172.21.15.22:57192) 2024-03-24T06:43:03.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04b9400640 1 -- 172.21.15.22:0/1354149552 learned_addr learned my addr 172.21.15.22:0/1354149552 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.948+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.949+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.949+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d1791b3f20 con 0x55d1792b10f0 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.949+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d1791b3bc0 con 0x55d1792a8a60 2024-03-24T06:43:03.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.949+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d178f7d1c0 con 0x55d1792a7d80 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.949+0000 7f04beba7740 10 client.0 _ll_register_callbacks cb 0x55d178f1fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 -- 172.21.15.22:0/1354149552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 msgr2=0x55d1792ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 -- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 msgr2=0x55d1792aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b4111080 con 0x55d1792a7d80 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04b9400640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f04a401ea00 tx=0x7f04a4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b9d2c47c919bdeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f049ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:43:03.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.950+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d179207160 con 0x55d1792a7d80 2024-03-24T06:43:03.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.951+0000 7f049ea00640 1 -- 172.21.15.22:0/1354149552 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4023070 con 0x55d1792a7d80 2024-03-24T06:43:03.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.951+0000 7f049ea00640 1 -- 172.21.15.22:0/1354149552 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04a4009de0 con 0x55d1792a7d80 2024-03-24T06:43:03.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.951+0000 7f049ea00640 1 -- 172.21.15.22:0/1354149552 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f04a4017070 con 0x55d1792a7d80 2024-03-24T06:43:03.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.951+0000 7f04beba7740 10 client.26401 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:03.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.951+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55d17920db20 con 0x55d1792a7d80 2024-03-24T06:43:03.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.952+0000 7f049ea00640 1 -- 172.21.15.22:0/1354149552 <== mon.1 v2:172.21.15.155:3300/0 4 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f04a4029420 con 0x55d1792a7d80 2024-03-24T06:43:03.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.952+0000 7f04beba7740 10 client.26401 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:03.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.952+0000 7f04beba7740 10 client.26401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:03.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.953+0000 7f04beba7740 20 client.26401 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:03.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.953+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d1792d24d0 0x55d1792d4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:03.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.953+0000 7f04beba7740 4 client.26401 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:03.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.953+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d178f0e7e0 con 0x55d1792d24d0 2024-03-24T06:43:03.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.954+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d1792d24d0 0x55d1792d4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:03.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.957+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d1792d24d0 0x55d1792d4990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b2b08bfe006122 server_cookie=8453afa89316537c in_seq=0 out_seq=0 2024-03-24T06:43:03.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.957+0000 7f049ea00640 10 client.26401 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.959+0000 7f049ea00640 1 -- 172.21.15.22:0/1354149552 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d178f0e7e0 con 0x55d1792d24d0 2024-03-24T06:43:03.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.960+0000 7f049ea00640 10 client.26401 handle_command_reply: tid=0 2024-03-24T06:43:03.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.960+0000 7f04beba7740 1 client.26401 shutdown 2024-03-24T06:43:03.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.960+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 msgr2=0x55d1792ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f04a401ea00 tx=0x7f04a4004730 comp rx=0 tx=0).stop 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 shutdown_connections 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792b10f0 0x55d1792aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d1792d24d0 0x55d1792d4990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792a8a60 0x55d1792ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f049ea00640 0 client.26401 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:03.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 --2- 172.21.15.22:0/1354149552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792ae0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:03.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.961+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 >> 172.21.15.22:0/1354149552 conn(0x55d17926eb30 msgr2=0x55d1792a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:03.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.963+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 shutdown_connections 2024-03-24T06:43:03.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.963+0000 7f04beba7740 1 -- 172.21.15.22:0/1354149552 wait complete. 2024-03-24T06:43:03.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:03.963+0000 7f04beba7740 20 client.26401 trim_cache size 0 max 16384 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.018+0000 7f04beba7740 1 Processor -- start 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.018+0000 7f04beba7740 1 -- start start 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 0x55d1792a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d1792af620 con 0x55d1792a8a60 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d178f0e7e0 con 0x55d1792b10f0 2024-03-24T06:43:04.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04beba7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d1792bbbf0 con 0x55d1792a7d80 2024-03-24T06:43:04.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.019+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.020+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34768/0 (socket says 172.21.15.22:34768) 2024-03-24T06:43:04.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.020+0000 7f04b8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 0x55d1792a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.020+0000 7f04b9e00640 1 -- 172.21.15.22:0/2684317298 learned_addr learned my addr 172.21.15.22:0/2684317298 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:04.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.020+0000 7f04b9400640 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b8a00640 1 -- 172.21.15.22:0/2684317298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 msgr2=0x55d1792b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b8a00640 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b8a00640 1 -- 172.21.15.22:0/2684317298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 msgr2=0x55d1792da260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b8a00640 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b8a00640 1 -- 172.21.15.22:0/2684317298 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1791a99a0 con 0x55d1792a8a60 2024-03-24T06:43:04.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.021+0000 7f04b9e00640 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:04.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.022+0000 7f04b8a00640 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 0x55d1792a44b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f04a00cb2b0 tx=0x7f04a00d02b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f07fc7d2ed4f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:04.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.022+0000 7f04a9c00640 1 -- 172.21.15.22:0/2684317298 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0014030 con 0x55d1792a8a60 2024-03-24T06:43:04.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.022+0000 7f04a9c00640 1 -- 172.21.15.22:0/2684317298 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04a009a140 con 0x55d1792a8a60 2024-03-24T06:43:04.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.022+0000 7f04a9c00640 1 -- 172.21.15.22:0/2684317298 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0014030 con 0x55d1792a8a60 2024-03-24T06:43:04.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.024+0000 7f04beba7740 1 -- 172.21.15.22:0/2684317298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 msgr2=0x55d1792a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:04.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.024+0000 7f04beba7740 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 0x55d1792a44b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f04a00cb2b0 tx=0x7f04a00d02b0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 -- 172.21.15.22:0/2684317298 shutdown_connections 2024-03-24T06:43:04.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792da260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a8a60 0x55d1792a44b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 --2- 172.21.15.22:0/2684317298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a7d80 0x55d1792b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 -- 172.21.15.22:0/2684317298 >> 172.21.15.22:0/2684317298 conn(0x55d17926eb30 msgr2=0x55d1792a0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:04.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.026+0000 7f04beba7740 1 -- 172.21.15.22:0/2684317298 shutdown_connections 2024-03-24T06:43:04.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.027+0000 7f04beba7740 1 -- 172.21.15.22:0/2684317298 wait complete. 2024-03-24T06:43:04.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.027+0000 7f04beba7740 1 -- start start 2024-03-24T06:43:04.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.028+0000 7f04beba7740 1 Processor -- start 2024-03-24T06:43:04.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.029+0000 7f04beba7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 0x55d1792ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.030+0000 7f04beba7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 0x55d1792ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.030+0000 7f04beba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.030+0000 7f04beba7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d1792b6af0 con 0x55d1792a7d80 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.030+0000 7f04beba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d1792b6c60 con 0x55d1792b10f0 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.030+0000 7f04beba7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d1792b6e20 con 0x55d1792a8a60 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.031+0000 7f04beba7740 10 client.0 _ll_register_callbacks cb 0x7f04a4009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.031+0000 7f04b9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 0x55d1792ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.031+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.031+0000 7f04b9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34778/0 (socket says 172.21.15.22:34778) 2024-03-24T06:43:04.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.031+0000 7f04b9e00640 1 -- 172.21.15.22:0/1159112395 learned_addr learned my addr 172.21.15.22:0/1159112395 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:04.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.032+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 0x55d1792ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.032+0000 7f04b9e00640 1 -- 172.21.15.22:0/1159112395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 msgr2=0x55d1792ba550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.032+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 0x55d1792ba550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.033+0000 7f04b9e00640 1 -- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 msgr2=0x55d1792ba010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.033+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 0x55d1792ba010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.033+0000 7f04b9e00640 1 -- 172.21.15.22:0/1159112395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1791d0c20 con 0x55d1792b10f0 2024-03-24T06:43:04.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.033+0000 7f04b8a00640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 0x55d1792ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:43:04.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.034+0000 7f04b9400640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 0x55d1792ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:04.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.034+0000 7f04b9e00640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f04ac004af0 tx=0x7f04ac003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34f5acd50675ae31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:04.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.034+0000 7f049ea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:04.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.035+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac050070 con 0x55d1792b10f0 2024-03-24T06:43:04.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.035+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d1792a4b30 con 0x55d1792b10f0 2024-03-24T06:43:04.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.035+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f04ac034650 con 0x55d1792b10f0 2024-03-24T06:43:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f04ac0326c0 con 0x55d1792b10f0 2024-03-24T06:43:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f04beba7740 10 client.26369 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x7f04b41a6cb0 con 0x55d1792b10f0 2024-03-24T06:43:04.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d1792a4b30 con 0x55d1792b10f0 2024-03-24T06:43:04.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7f04ac055420 con 0x55d1792b10f0 2024-03-24T06:43:04.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f04beba7740 10 client.26369 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:04.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f04beba7740 10 client.26369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.036+0000 7f04beba7740 20 client.26369 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.037+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d17934a830 0x55d17934cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.037+0000 7f04beba7740 4 client.26369 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.037+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f04ac032820 con 0x55d17934a830 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.040+0000 7f04b9400640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d17934a830 0x55d17934cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.041+0000 7f04b9400640 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d17934a830 0x55d17934cc50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6d6544227157d86 server_cookie=1b7f046a71fa2424 in_seq=0 out_seq=0 2024-03-24T06:43:04.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.041+0000 7f049ea00640 10 client.26369 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:04.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f049ea00640 1 -- 172.21.15.22:0/1159112395 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7f04ac032820 con 0x55d17934a830 2024-03-24T06:43:04.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f049ea00640 10 client.26369 handle_command_reply: tid=0 2024-03-24T06:43:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f04beba7740 1 client.26369 shutdown 2024-03-24T06:43:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 msgr2=0x55d1792b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f04ac004af0 tx=0x7f04ac003a80 comp rx=0 tx=0).stop 2024-03-24T06:43:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 shutdown_connections 2024-03-24T06:43:04.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.043+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d17934a830 0x55d17934cc50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f049ea00640 0 client.26369 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d1792b10f0 0x55d1792b6420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d1792a8a60 0x55d1792ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 --2- 172.21.15.22:0/1159112395 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d1792a7d80 0x55d1792ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 >> 172.21.15.22:0/1159112395 conn(0x55d17926eb30 msgr2=0x55d1792a0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 shutdown_connections 2024-03-24T06:43:04.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 1 -- 172.21.15.22:0/1159112395 wait complete. 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.045+0000 7f04beba7740 20 client.26369 trim_cache size 0 max 16384 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "scrub active (20 inodes in the stack)", 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d": { 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d", 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:43:04.046 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:43:04.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 msgr2=0x7f04980a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:04.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 0x7f04980a48c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55d1792a8fb0 tx=0x7f04a4005f90 comp rx=0 tx=0).stop 2024-03-24T06:43:04.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 msgr2=0x7f04b419a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:04.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d950 tx=0x7f04a000de10 comp rx=0 tx=0).stop 2024-03-24T06:43:04.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 shutdown_connections 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f04980a2400 0x7f04980a48c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04b41098c0 0x7f04b419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04b4104270 0x7f04b419a4f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 --2- 172.21.15.22:0/777244406 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f04b41038b0 0x7f04b4199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 >> 172.21.15.22:0/777244406 conn(0x7f04b40ff360 msgr2=0x7f04b40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 shutdown_connections 2024-03-24T06:43:04.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:04.047+0000 7f04bbe00640 1 -- 172.21.15.22:0/777244406 wait complete. 2024-03-24T06:43:04.075 INFO:tasks.cephfs.filesystem:scrub status for tag:a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d - {'path': '/', 'tag': 'a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d', 'options': 'recursive,force'} 2024-03-24T06:43:04.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:03 smithi022 ceph-mon[32003]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 81 MiB/s wr, 10.09k op/s 2024-03-24T06:43:04.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:03 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:43:04.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:03 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:04.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:03 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:04.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:03 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:03 smithi143 ceph-mon[36520]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 81 MiB/s wr, 10.09k op/s 2024-03-24T06:43:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:03 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:43:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:03 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:03 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:04.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:03 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:03 smithi155 ceph-mon[36262]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 81 MiB/s wr, 10.09k op/s 2024-03-24T06:43:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:03 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:43:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:03 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:03 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:04.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:03 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:05.148 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:04 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:04 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:05.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:04 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:04 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:04 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:05.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:04 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:04 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:04 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:05.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:04 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:05 smithi022 ceph-mon[32003]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 67 MiB/s wr, 8.16k op/s 2024-03-24T06:43:06.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:05 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:05 smithi143 ceph-mon[36520]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 67 MiB/s wr, 8.16k op/s 2024-03-24T06:43:06.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:05 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:06.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:05 smithi155 ceph-mon[36262]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 67 MiB/s wr, 8.16k op/s 2024-03-24T06:43:06.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:05 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:08.150 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:07 smithi022 ceph-mon[32003]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 72 MiB/s wr, 8.77k op/s 2024-03-24T06:43:08.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:07 smithi143 ceph-mon[36520]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 72 MiB/s wr, 8.77k op/s 2024-03-24T06:43:08.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:07 smithi155 ceph-mon[36262]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 72 MiB/s wr, 8.77k op/s 2024-03-24T06:43:10.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:09 smithi143 ceph-mon[36520]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 76 MiB/s wr, 9.27k op/s 2024-03-24T06:43:10.179 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:09 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:09 smithi155 ceph-mon[36262]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 76 MiB/s wr, 9.27k op/s 2024-03-24T06:43:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:09 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:10.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:09 smithi022 ceph-mon[32003]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 76 MiB/s wr, 9.27k op/s 2024-03-24T06:43:10.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:09 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:12.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:11 smithi143 ceph-mon[36520]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 MiB/s wr, 7.64k op/s 2024-03-24T06:43:12.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:11 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:11 smithi155 ceph-mon[36262]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 MiB/s wr, 7.64k op/s 2024-03-24T06:43:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:11 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:12.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:11 smithi022 ceph-mon[32003]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 MiB/s wr, 7.64k op/s 2024-03-24T06:43:12.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:11 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:13 smithi155 ceph-mon[36262]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 85 MiB/s wr, 10.32k op/s 2024-03-24T06:43:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:13 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:43:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:13 smithi022 ceph-mon[32003]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 85 MiB/s wr, 10.32k op/s 2024-03-24T06:43:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:13 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:43:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:13 smithi143 ceph-mon[36520]: pgmap v99: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 85 MiB/s wr, 10.32k op/s 2024-03-24T06:43:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:13 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:43:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:15 smithi155 ceph-mon[36262]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 MiB/s wr, 8.64k op/s 2024-03-24T06:43:16.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:15 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:43:16.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:15 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:43:16.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:15 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:43:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:15 smithi022 ceph-mon[32003]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 MiB/s wr, 8.64k op/s 2024-03-24T06:43:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:15 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:43:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:15 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:43:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:15 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:43:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:15 smithi143 ceph-mon[36520]: pgmap v100: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 MiB/s wr, 8.64k op/s 2024-03-24T06:43:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:15 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:43:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:15 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:43:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:15 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:43:18.230 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:17 smithi143 ceph-mon[36520]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 72 MiB/s wr, 8.91k op/s 2024-03-24T06:43:18.230 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:17 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:17 smithi155 ceph-mon[36262]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 72 MiB/s wr, 8.91k op/s 2024-03-24T06:43:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:17 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:18.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:17 smithi022 ceph-mon[32003]: pgmap v101: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 72 MiB/s wr, 8.91k op/s 2024-03-24T06:43:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:17 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:20.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:19 smithi155 ceph-mon[36262]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 MiB/s wr, 9.79k op/s 2024-03-24T06:43:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:19 smithi022 ceph-mon[32003]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 MiB/s wr, 9.79k op/s 2024-03-24T06:43:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:19 smithi143 ceph-mon[36520]: pgmap v102: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 77 MiB/s wr, 9.79k op/s 2024-03-24T06:43:22.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:21 smithi155 ceph-mon[36262]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 60 MiB/s wr, 7.75k op/s 2024-03-24T06:43:22.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:21 smithi022 ceph-mon[32003]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 60 MiB/s wr, 7.75k op/s 2024-03-24T06:43:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:21 smithi143 ceph-mon[36520]: pgmap v103: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 60 MiB/s wr, 7.75k op/s 2024-03-24T06:43:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:23 smithi155 ceph-mon[36262]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 71 MiB/s wr, 9.45k op/s 2024-03-24T06:43:24.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:23 smithi022 ceph-mon[32003]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 71 MiB/s wr, 9.45k op/s 2024-03-24T06:43:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:23 smithi143 ceph-mon[36520]: pgmap v104: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 71 MiB/s wr, 9.45k op/s 2024-03-24T06:43:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:25 smithi155 ceph-mon[36262]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 MiB/s wr, 7.70k op/s 2024-03-24T06:43:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:25 smithi022 ceph-mon[32003]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 MiB/s wr, 7.70k op/s 2024-03-24T06:43:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:25 smithi143 ceph-mon[36520]: pgmap v105: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 55 MiB/s wr, 7.70k op/s 2024-03-24T06:43:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:28 smithi155 ceph-mon[36262]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 56 MiB/s wr, 8.04k op/s 2024-03-24T06:43:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:28 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:43:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:28 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:27 smithi022 ceph-mon[32003]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 56 MiB/s wr, 8.04k op/s 2024-03-24T06:43:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:28 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:43:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:28 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:28 smithi143 ceph-mon[36520]: pgmap v106: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 56 MiB/s wr, 8.04k op/s 2024-03-24T06:43:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:28 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:43:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:28 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:30 smithi155 ceph-mon[36262]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 MiB/s wr, 7.86k op/s 2024-03-24T06:43:30.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:30 smithi022 ceph-mon[32003]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 MiB/s wr, 7.86k op/s 2024-03-24T06:43:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:30 smithi143 ceph-mon[36520]: pgmap v107: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 MiB/s wr, 7.86k op/s 2024-03-24T06:43:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:32 smithi155 ceph-mon[36262]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 44 MiB/s wr, 6.44k op/s 2024-03-24T06:43:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:32.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:32 smithi022 ceph-mon[32003]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 44 MiB/s wr, 6.44k op/s 2024-03-24T06:43:32.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:32 smithi143 ceph-mon[36520]: pgmap v108: 129 pgs: 129 active+clean; 14 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 44 MiB/s wr, 6.44k op/s 2024-03-24T06:43:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:43:34.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:43:34.077 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:43:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:34 smithi155 ceph-mon[36262]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 57 MiB/s wr, 8.15k op/s 2024-03-24T06:43:34.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 -- 172.21.15.22:0/4111004660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc620100320 msgr2=0x7fc620100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 --2- 172.21.15.22:0/4111004660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc620100320 0x7fc620100700 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc610009900 tx=0x7fc61002bc40 comp rx=0 tx=0).stop 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 -- 172.21.15.22:0/4111004660 shutdown_connections 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 --2- 172.21.15.22:0/4111004660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc62010ded0 0x7fc6201102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 --2- 172.21.15.22:0/4111004660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100cd0 0x7fc62010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 --2- 172.21.15.22:0/4111004660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc620100320 0x7fc620100700 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 -- 172.21.15.22:0/4111004660 >> 172.21.15.22:0/4111004660 conn(0x7fc6200fbf60 msgr2=0x7fc6200fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 -- 172.21.15.22:0/4111004660 shutdown_connections 2024-03-24T06:43:34.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.374+0000 7fc61fe00640 1 -- 172.21.15.22:0/4111004660 wait complete. 2024-03-24T06:43:34.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 Processor -- start 2024-03-24T06:43:34.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 -- start start 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc620100cd0 0x7fc62019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62010ded0 0x7fc62019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc62019ecb0 con 0x7fc620100320 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc62019ee20 con 0x7fc62010ded0 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.375+0000 7fc61fe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc62019efe0 con 0x7fc620100cd0 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc616000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc620100cd0 0x7fc62019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62010ded0 0x7fc62019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40622/0 (socket says 172.21.15.22:40622) 2024-03-24T06:43:34.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 -- 172.21.15.22:0/1214799415 learned_addr learned my addr 172.21.15.22:0/1214799415 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 -- 172.21.15.22:0/1214799415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc620100cd0 msgr2=0x7fc62019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc620100cd0 0x7fc62019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 -- 172.21.15.22:0/1214799415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62010ded0 msgr2=0x7fc62019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62010ded0 0x7fc62019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.376+0000 7fc61ea00640 1 -- 172.21.15.22:0/1214799415 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc610009560 con 0x7fc620100320 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc61ea00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc610000c00 tx=0x7fc61002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc617e00640 1 -- 172.21.15.22:0/1214799415 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610004390 con 0x7fc620100320 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc62019f260 con 0x7fc620100320 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc617e00640 1 -- 172.21.15.22:0/1214799415 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6100044f0 con 0x7fc620100320 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc617e00640 1 -- 172.21.15.22:0/1214799415 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610003c90 con 0x7fc620100320 2024-03-24T06:43:34.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.377+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6201a6b40 con 0x7fc620100320 2024-03-24T06:43:34.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.380+0000 7fc617e00640 1 -- 172.21.15.22:0/1214799415 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc610004df0 con 0x7fc620100320 2024-03-24T06:43:34.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.381+0000 7fc617e00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 0x7fc6040a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.381+0000 7fc617e00640 1 -- 172.21.15.22:0/1214799415 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc6100f09b0 con 0x7fc620100320 2024-03-24T06:43:34.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.381+0000 7fc628577740 1 Processor -- start 2024-03-24T06:43:34.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc616000640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 0x7fc6040a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 -- start start 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa08fe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa09082d0 0x55afa090a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55afa086d790 con 0x55afa08ffc40 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55afa086acc0 con 0x55afa08fef60 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc628577740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55afa08664a0 con 0x55afa09082d0 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc616000640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 0x7fc6040a4770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6200ff500 tx=0x7fc608005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc616000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa08fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.382+0000 7fc616000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:38814/0 (socket says 172.21.15.22:38814) 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 -- 172.21.15.22:0/3405212427 learned_addr learned my addr 172.21.15.22:0/3405212427 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa08fe990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40630/0 (socket says 172.21.15.22:40630) 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc61f400640 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa09082d0 0x55afa090a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 -- 172.21.15.22:0/3405212427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa09082d0 msgr2=0x55afa090a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa09082d0 0x55afa090a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 -- 172.21.15.22:0/3405212427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 msgr2=0x55afa08fe990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa08fe990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 -- 172.21.15.22:0/3405212427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afa0827c20 con 0x55afa08fef60 2024-03-24T06:43:34.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.383+0000 7fc616000640 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc60800e630 tx=0x7fc608014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7660543302f03169 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc60fe00640 1 -- 172.21.15.22:0/3405212427 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608004020 con 0x55afa08fef60 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc60fe00640 1 -- 172.21.15.22:0/3405212427 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc608021070 con 0x55afa08fef60 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc60fe00640 1 -- 172.21.15.22:0/3405212427 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60801d670 con 0x55afa08fef60 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 -- 172.21.15.22:0/3405212427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 msgr2=0x55afa08ff3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc60800e630 tx=0x7fc608014000 comp rx=0 tx=0).stop 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 -- 172.21.15.22:0/3405212427 shutdown_connections 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa09082d0 0x55afa090a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa08ff3e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 --2- 172.21.15.22:0/3405212427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa08fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 -- 172.21.15.22:0/3405212427 >> 172.21.15.22:0/3405212427 conn(0x55afa08c5b30 msgr2=0x55afa08f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.385+0000 7fc628577740 1 -- 172.21.15.22:0/3405212427 shutdown_connections 2024-03-24T06:43:34.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.386+0000 7fc628577740 1 -- 172.21.15.22:0/3405212427 wait complete. 2024-03-24T06:43:34.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.386+0000 7fc628577740 1 -- start start 2024-03-24T06:43:34.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.386+0000 7fc628577740 1 Processor -- start 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08ffc40 0x55afa0905840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 0x55afa0905d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55afa080af20 con 0x55afa09082d0 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55afa080abc0 con 0x55afa08fef60 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55afa05d41c0 con 0x55afa08ffc40 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc628577740 10 client.0 _ll_register_callbacks cb 0x55afa0576be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:38818/0 (socket says 172.21.15.22:38818) 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61ea00640 1 -- 172.21.15.22:0/3676032943 learned_addr learned my addr 172.21.15.22:0/3676032943 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61f400640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 0x55afa0905d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc616000640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08ffc40 0x55afa0905840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61ea00640 1 -- 172.21.15.22:0/3676032943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08ffc40 msgr2=0x55afa0905840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.387+0000 7fc61ea00640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08ffc40 0x55afa0905840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc61ea00640 1 -- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 msgr2=0x55afa0905d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc61ea00640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 0x55afa0905d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc61ea00640 1 -- 172.21.15.22:0/3676032943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc608017070 con 0x55afa08fef60 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc61f400640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 0x55afa0905d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc61ea00640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc6100b9b50 tx=0x7fc610005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa0df99d6efd5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc60cc00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:43:34.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100c12d0 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc6100c5890 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55afa085e160 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc6100b97f0 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc628577740 10 client.26381 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.388+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55afa0864b20 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55afa085e160 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fc61003c070 con 0x55afa08fef60 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 10 client.26381 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 20 client.26381 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa0918430 0x55afa091a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 4 client.26381 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afa05657e0 con 0x55afa0918430 2024-03-24T06:43:34.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.389+0000 7fc616000640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa0918430 0x55afa091a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.390+0000 7fc616000640 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa0918430 0x55afa091a850 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3023c20fb6a4bba8 server_cookie=42e65abab704f4a1 in_seq=0 out_seq=0 2024-03-24T06:43:34.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.390+0000 7fc60cc00640 10 client.26381 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.390+0000 7fc60cc00640 1 -- 172.21.15.22:0/3676032943 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afa05657e0 con 0x55afa0918430 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.390+0000 7fc60cc00640 10 client.26381 handle_command_reply: tid=0 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 client.26381 shutdown 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 msgr2=0x55afa0905300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc6100b9b50 tx=0x7fc610005670 comp rx=0 tx=0).stop 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 shutdown_connections 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa09082d0 0x55afa0905d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08ffc40 0x55afa0905840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa0918430 0x55afa091a850 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 --2- 172.21.15.22:0/3676032943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa08fef60 0x55afa0905300 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc60cc00640 0 client.26381 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 >> 172.21.15.22:0/3676032943 conn(0x55afa08c5b30 msgr2=0x55afa08fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 shutdown_connections 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 1 -- 172.21.15.22:0/3676032943 wait complete. 2024-03-24T06:43:34.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.391+0000 7fc628577740 20 client.26381 trim_cache size 0 max 16384 2024-03-24T06:43:34.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:34 smithi022 ceph-mon[32003]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 57 MiB/s wr, 8.15k op/s 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.410+0000 7fc628577740 1 Processor -- start 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.410+0000 7fc628577740 1 -- start start 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0913c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa09096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55afa08f55f0 con 0x55afa08ffc40 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55afa091aeb0 con 0x55afa09082d0 2024-03-24T06:43:34.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc628577740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55afa0903890 con 0x55afa08fef60 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40638/0 (socket says 172.21.15.22:40638) 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 -- 172.21.15.22:0/12528341 learned_addr learned my addr 172.21.15.22:0/12528341 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc61f400640 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa09096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc61ea00640 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0913c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 -- 172.21.15.22:0/12528341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 msgr2=0x55afa0913c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0913c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 -- 172.21.15.22:0/12528341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 msgr2=0x55afa09096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa09096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.411+0000 7fc616000640 1 -- 172.21.15.22:0/12528341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afa08009a0 con 0x55afa08ffc40 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.412+0000 7fc616000640 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6080118d0 tx=0x7fc60801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bdcaf8efbc6fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.412+0000 7fc60f400640 1 -- 172.21.15.22:0/12528341 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60802a040 con 0x55afa08ffc40 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.412+0000 7fc60f400640 1 -- 172.21.15.22:0/12528341 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc608034120 con 0x55afa08ffc40 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.412+0000 7fc60f400640 1 -- 172.21.15.22:0/12528341 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608003830 con 0x55afa08ffc40 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 -- 172.21.15.22:0/12528341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 msgr2=0x55afa0920120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6080118d0 tx=0x7fc60801f040 comp rx=0 tx=0).stop 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 -- 172.21.15.22:0/12528341 shutdown_connections 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa09096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0920120 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 --2- 172.21.15.22:0/12528341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0913c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 -- 172.21.15.22:0/12528341 >> 172.21.15.22:0/12528341 conn(0x55afa08c5b30 msgr2=0x55afa08f8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 -- 172.21.15.22:0/12528341 shutdown_connections 2024-03-24T06:43:34.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.413+0000 7fc628577740 1 -- 172.21.15.22:0/12528341 wait complete. 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.414+0000 7fc628577740 1 -- start start 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.414+0000 7fc628577740 1 Processor -- start 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.414+0000 7fc628577740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0902110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.414+0000 7fc628577740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc628577740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa0902b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc628577740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55afa097d8c0 con 0x55afa08ffc40 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc628577740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55afa097da30 con 0x55afa09082d0 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc628577740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55afa097dbf0 con 0x55afa08fef60 2024-03-24T06:43:34.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc628577740 10 client.0 _ll_register_callbacks cb 0x7fc6100bcf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:40654/0 (socket says 172.21.15.22:40654) 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 -- 172.21.15.22:0/3396934794 learned_addr learned my addr 172.21.15.22:0/3396934794 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc61f400640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa0902b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 -- 172.21.15.22:0/3396934794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 msgr2=0x55afa0902110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0902110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 -- 172.21.15.22:0/3396934794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 msgr2=0x55afa0902b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa0902b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 -- 172.21.15.22:0/3396934794 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afa0827c20 con 0x55afa08ffc40 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc616000640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc608009460 tx=0x7fc6080050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7c477b92844184 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.415+0000 7fc60cc00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60802a070 con 0x55afa08ffc40 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc608005280 con 0x55afa08ffc40 2024-03-24T06:43:34.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc608021070 con 0x55afa08ffc40 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc608005440 con 0x55afa08ffc40 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc628577740 10 client.16089 fetch_fsmap learned FSMap version 33 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55afa0826480 con 0x55afa08ffc40 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.416+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc608021070 con 0x55afa08ffc40 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 33) ==== 8394+0+0 (secure 0 0 0) 0x7fc608003ab0 con 0x55afa08ffc40 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 10 client.16089 fetch_fsmap finished waiting for FSMap version 33 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 20 client.16089 populate_metadata read hostname 'smithi022' 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa097f1a0 0x55afa09815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 4 client.16089 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55afa05a2720 con 0x55afa097f1a0 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.417+0000 7fc61ea00640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa097f1a0 0x55afa09815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc61ea00640 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa097f1a0 0x55afa09815c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c34c69898e1b3710 server_cookie=7985d37f1beb2641 in_seq=0 out_seq=0 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc60cc00640 10 client.16089 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc60cc00640 1 -- 172.21.15.22:0/3396934794 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55afa05a2720 con 0x55afa097f1a0 2024-03-24T06:43:34.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc60cc00640 10 client.16089 handle_command_reply: tid=0 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 client.16089 shutdown 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 msgr2=0x55afa0902650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc608009460 tx=0x7fc6080050d0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 shutdown_connections 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55afa097f1a0 0x55afa09815c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55afa09082d0 0x55afa0902b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55afa08ffc40 0x55afa0902650 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 --2- 172.21.15.22:0/3396934794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55afa08fef60 0x55afa0902110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 >> 172.21.15.22:0/3396934794 conn(0x55afa08c5b30 msgr2=0x55afa08fa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc60cc00640 0 client.16089 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:43:34.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 shutdown_connections 2024-03-24T06:43:34.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.418+0000 7fc628577740 1 -- 172.21.15.22:0/3396934794 wait complete. 2024-03-24T06:43:34.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc628577740 20 client.16089 trim_cache size 0 max 16384 2024-03-24T06:43:34.422 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:43:34.422 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: "a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d": { 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d", 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:43:34.423 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 msgr2=0x7fc6040a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 0x7fc6040a4770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6200ff500 tx=0x7fc608005f90 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 msgr2=0x7fc620199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc610000c00 tx=0x7fc61002c640 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 shutdown_connections 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc6040a22b0 0x7fc6040a4770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62010ded0 0x7fc62019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc620100cd0 0x7fc62019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 --2- 172.21.15.22:0/1214799415 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc620100320 0x7fc620199df0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.419+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 >> 172.21.15.22:0/1214799415 conn(0x7fc6200fbf60 msgr2=0x7fc62010f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.420+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 shutdown_connections 2024-03-24T06:43:34.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:43:34.420+0000 7fc61fe00640 1 -- 172.21.15.22:0/1214799415 wait complete. 2024-03-24T06:43:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:34 smithi143 ceph-mon[36520]: pgmap v109: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 57 MiB/s wr, 8.15k op/s 2024-03-24T06:43:34.438 INFO:tasks.cephfs.filesystem:scrub status for tag:a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d - {'path': '/', 'tag': 'a96cdfb5-7d62-4b5e-a43b-3ff314b2e85d', 'options': 'recursive,force'} 2024-03-24T06:43:36.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:36 smithi022 ceph-mon[32003]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 48 MiB/s wr, 6.75k op/s 2024-03-24T06:43:36.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:36 smithi143 ceph-mon[36520]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 48 MiB/s wr, 6.75k op/s 2024-03-24T06:43:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:36 smithi155 ceph-mon[36262]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 48 MiB/s wr, 6.75k op/s 2024-03-24T06:43:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:38 smithi022 ceph-mon[32003]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 7.47k op/s 2024-03-24T06:43:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:38 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:43:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:38 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:43:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:38 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:43:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:38 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:38 smithi143 ceph-mon[36520]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 7.47k op/s 2024-03-24T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:38 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:38 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:38 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:38 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:43:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:38 smithi155 ceph-mon[36262]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 7.47k op/s 2024-03-24T06:43:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:38 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:43:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:38 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:43:38.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:38 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:43:38.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:38 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:43:39.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:39 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:43:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:39 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:43:39.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:39 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:43:40.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:40 smithi022 ceph-mon[32003]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:43:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:40 smithi143 ceph-mon[36520]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:43:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:40 smithi155 ceph-mon[36262]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:43:42.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:42 smithi022 ceph-mon[32003]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 55 MiB/s wr, 7.46k op/s 2024-03-24T06:43:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:42 smithi143 ceph-mon[36520]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 55 MiB/s wr, 7.46k op/s 2024-03-24T06:43:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:42 smithi155 ceph-mon[36262]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 55 MiB/s wr, 7.46k op/s 2024-03-24T06:43:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:44 smithi022 ceph-mon[32003]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 MiB/s wr, 8.99k op/s 2024-03-24T06:43:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:44 smithi143 ceph-mon[36520]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 MiB/s wr, 8.99k op/s 2024-03-24T06:43:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:44 smithi155 ceph-mon[36262]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 MiB/s wr, 8.99k op/s 2024-03-24T06:43:46.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:46 smithi022 ceph-mon[32003]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 7.60k op/s 2024-03-24T06:43:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:46 smithi143 ceph-mon[36520]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 7.60k op/s 2024-03-24T06:43:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:46 smithi155 ceph-mon[36262]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 56 MiB/s wr, 7.60k op/s 2024-03-24T06:43:48.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:48 smithi022 ceph-mon[32003]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 62 MiB/s wr, 8.48k op/s 2024-03-24T06:43:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:48 smithi143 ceph-mon[36520]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 62 MiB/s wr, 8.48k op/s 2024-03-24T06:43:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:48 smithi155 ceph-mon[36262]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 62 MiB/s wr, 8.48k op/s 2024-03-24T06:43:50.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:50 smithi022 ceph-mon[32003]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 MiB/s wr, 8.03k op/s 2024-03-24T06:43:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:50 smithi143 ceph-mon[36520]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 MiB/s wr, 8.03k op/s 2024-03-24T06:43:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:50 smithi155 ceph-mon[36262]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 59 MiB/s wr, 8.03k op/s 2024-03-24T06:43:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:52 smithi143 ceph-mon[36520]: pgmap v118: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 48 MiB/s wr, 6.68k op/s 2024-03-24T06:43:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:52 smithi155 ceph-mon[36262]: pgmap v118: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 48 MiB/s wr, 6.68k op/s 2024-03-24T06:43:52.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:52 smithi022 ceph-mon[32003]: pgmap v118: 129 pgs: 129 active+clean; 15 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 48 MiB/s wr, 6.68k op/s 2024-03-24T06:43:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:54 smithi143 ceph-mon[36520]: pgmap v119: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 MiB/s wr, 9.29k op/s 2024-03-24T06:43:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:54 smithi155 ceph-mon[36262]: pgmap v119: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 MiB/s wr, 9.29k op/s 2024-03-24T06:43:54.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:54 smithi022 ceph-mon[32003]: pgmap v119: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 MiB/s wr, 9.29k op/s 2024-03-24T06:43:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:56 smithi143 ceph-mon[36520]: pgmap v120: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 MiB/s wr, 7.49k op/s 2024-03-24T06:43:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:56 smithi155 ceph-mon[36262]: pgmap v120: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 MiB/s wr, 7.49k op/s 2024-03-24T06:43:56.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:56 smithi022 ceph-mon[32003]: pgmap v120: 129 pgs: 129 active+clean; 15 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 MiB/s wr, 7.49k op/s 2024-03-24T06:43:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:43:58 smithi143 ceph-mon[36520]: pgmap v121: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 61 MiB/s wr, 8.27k op/s 2024-03-24T06:43:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:43:58 smithi155 ceph-mon[36262]: pgmap v121: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 61 MiB/s wr, 8.27k op/s 2024-03-24T06:43:58.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:43:58 smithi022 ceph-mon[32003]: pgmap v121: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 61 MiB/s wr, 8.27k op/s 2024-03-24T06:44:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:00 smithi143 ceph-mon[36520]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 63 MiB/s wr, 8.61k op/s 2024-03-24T06:44:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:00 smithi155 ceph-mon[36262]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 63 MiB/s wr, 8.61k op/s 2024-03-24T06:44:00.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:00 smithi022 ceph-mon[32003]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 63 MiB/s wr, 8.61k op/s 2024-03-24T06:44:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:02 smithi155 ceph-mon[36262]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:44:02.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:02 smithi022 ceph-mon[32003]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:44:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:02 smithi143 ceph-mon[36520]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 58 MiB/s wr, 7.96k op/s 2024-03-24T06:44:04.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:44:04.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:44:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:04 smithi155 ceph-mon[36262]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 MiB/s wr, 10.06k op/s 2024-03-24T06:44:04.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:04 smithi022 ceph-mon[32003]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 MiB/s wr, 10.06k op/s 2024-03-24T06:44:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:04 smithi143 ceph-mon[36520]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 74 MiB/s wr, 10.06k op/s 2024-03-24T06:44:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:06 smithi155 ceph-mon[36262]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-24T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:06 smithi143 ceph-mon[36520]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-24T06:44:07.459 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:06 smithi022 ceph-mon[32003]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-24T06:44:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:08 smithi155 ceph-mon[36262]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 53 MiB/s wr, 7.25k op/s 2024-03-24T06:44:08.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:08 smithi022 ceph-mon[32003]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 53 MiB/s wr, 7.25k op/s 2024-03-24T06:44:08.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:08 smithi143 ceph-mon[36520]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 53 MiB/s wr, 7.25k op/s 2024-03-24T06:44:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:10 smithi155 ceph-mon[36262]: pgmap v127: 129 pgs: 129 active+clean; 16 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.19k op/s 2024-03-24T06:44:10.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:10 smithi022 ceph-mon[32003]: pgmap v127: 129 pgs: 129 active+clean; 16 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.19k op/s 2024-03-24T06:44:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:10 smithi143 ceph-mon[36520]: pgmap v127: 129 pgs: 129 active+clean; 16 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.19k op/s 2024-03-24T06:44:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:12 smithi155 ceph-mon[36262]: pgmap v128: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-24T06:44:12.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:12 smithi022 ceph-mon[32003]: pgmap v128: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-24T06:44:12.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:12 smithi143 ceph-mon[36520]: pgmap v128: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-24T06:44:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:14 smithi155 ceph-mon[36262]: pgmap v129: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.34k op/s 2024-03-24T06:44:14.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:14 smithi022 ceph-mon[32003]: pgmap v129: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.34k op/s 2024-03-24T06:44:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:14 smithi143 ceph-mon[36520]: pgmap v129: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 MiB/s wr, 8.34k op/s 2024-03-24T06:44:15.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:15 smithi155 ceph-mon[36262]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 MiB/s wr, 6.62k op/s 2024-03-24T06:44:15.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:15 smithi022 ceph-mon[32003]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 MiB/s wr, 6.62k op/s 2024-03-24T06:44:15.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:15 smithi143 ceph-mon[36520]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 MiB/s wr, 6.62k op/s 2024-03-24T06:44:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:16 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:44:16.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:16 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:44:16.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:16 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:44:17.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:17 smithi155 ceph-mon[36262]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 MiB/s wr, 6.75k op/s 2024-03-24T06:44:17.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:17 smithi022 ceph-mon[32003]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 MiB/s wr, 6.75k op/s 2024-03-24T06:44:17.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:17 smithi143 ceph-mon[36520]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 MiB/s wr, 6.75k op/s 2024-03-24T06:44:19.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:19 smithi022 ceph-mon[32003]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 56 MiB/s wr, 7.99k op/s 2024-03-24T06:44:19.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:19 smithi143 ceph-mon[36520]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 56 MiB/s wr, 7.99k op/s 2024-03-24T06:44:19.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:19 smithi155 ceph-mon[36262]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 56 MiB/s wr, 7.99k op/s 2024-03-24T06:44:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:21 smithi022 ceph-mon[32003]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 5.65k op/s 2024-03-24T06:44:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:21 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:44:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:21 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:44:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:21 smithi022 ceph-mon[32003]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 MiB/s wr, 5.84k op/s 2024-03-24T06:44:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:21 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:44:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:21 smithi143 ceph-mon[36520]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 5.65k op/s 2024-03-24T06:44:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:21 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:44:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:21 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:44:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:21 smithi143 ceph-mon[36520]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 MiB/s wr, 5.84k op/s 2024-03-24T06:44:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:21 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:44:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:21 smithi155 ceph-mon[36262]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 5.65k op/s 2024-03-24T06:44:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:21 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:44:21.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:21 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:44:21.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:21 smithi155 ceph-mon[36262]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 MiB/s wr, 5.84k op/s 2024-03-24T06:44:21.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:21 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:44:22.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:22 smithi022 ceph-mon[32003]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-24T06:44:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:22 smithi143 ceph-mon[36520]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-24T06:44:22.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:22 smithi155 ceph-mon[36262]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-24T06:44:23.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:23 smithi022 ceph-mon[32003]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 6.06k op/s 2024-03-24T06:44:23.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:23 smithi143 ceph-mon[36520]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 6.06k op/s 2024-03-24T06:44:23.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:23 smithi155 ceph-mon[36262]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 MiB/s wr, 6.06k op/s 2024-03-24T06:44:26.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:25 smithi022 ceph-mon[32003]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 40 MiB/s wr, 5.90k op/s 2024-03-24T06:44:26.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:25 smithi143 ceph-mon[36520]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 40 MiB/s wr, 5.90k op/s 2024-03-24T06:44:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:25 smithi155 ceph-mon[36262]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 40 MiB/s wr, 5.90k op/s 2024-03-24T06:44:28.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:27 smithi022 ceph-mon[32003]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 44 MiB/s wr, 7.06k op/s 2024-03-24T06:44:28.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:27 smithi143 ceph-mon[36520]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 44 MiB/s wr, 7.06k op/s 2024-03-24T06:44:28.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:27 smithi155 ceph-mon[36262]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 44 MiB/s wr, 7.06k op/s 2024-03-24T06:44:30.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:29 smithi143 ceph-mon[36520]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-24T06:44:30.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:29 smithi155 ceph-mon[36262]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-24T06:44:30.402 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:29 smithi022 ceph-mon[32003]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-24T06:44:32.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:31 smithi143 ceph-mon[36520]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.85k op/s 2024-03-24T06:44:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:31 smithi155 ceph-mon[36262]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.85k op/s 2024-03-24T06:44:32.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:31 smithi022 ceph-mon[32003]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 33 MiB/s wr, 5.85k op/s 2024-03-24T06:44:34.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:33 smithi143 ceph-mon[36520]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.66k op/s 2024-03-24T06:44:34.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:33 smithi155 ceph-mon[36262]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.66k op/s 2024-03-24T06:44:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:33 smithi022 ceph-mon[32003]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.66k op/s 2024-03-24T06:44:36.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:35 smithi143 ceph-mon[36520]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-24T06:44:36.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:35 smithi155 ceph-mon[36262]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-24T06:44:36.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:35 smithi022 ceph-mon[32003]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-03-24T06:44:38.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:37 smithi155 ceph-mon[36262]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 MiB/s wr, 5.95k op/s 2024-03-24T06:44:38.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:37 smithi022 ceph-mon[32003]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 MiB/s wr, 5.95k op/s 2024-03-24T06:44:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:37 smithi143 ceph-mon[36520]: pgmap v142: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 MiB/s wr, 5.95k op/s 2024-03-24T06:44:38.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.992+0000 7ff8f5600640 1 -- 172.21.15.22:0/485770297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f0100790 msgr2=0x7ff8f0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:38.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.992+0000 7ff8f5600640 1 --2- 172.21.15.22:0/485770297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f0100790 0x7ff8f0102b80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009ca0 tx=0x7ff8e802c1c0 comp rx=0 tx=0).stop 2024-03-24T06:44:38.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.993+0000 7ff8f5600640 1 -- 172.21.15.22:0/485770297 shutdown_connections 2024-03-24T06:44:38.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.993+0000 7ff8f5600640 1 --2- 172.21.15.22:0/485770297 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff8f010e0e0 0x7ff8f01104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:38.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.993+0000 7ff8f5600640 1 --2- 172.21.15.22:0/485770297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f01030c0 0x7ff8f010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:38.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.993+0000 7ff8f5600640 1 --2- 172.21.15.22:0/485770297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f0100790 0x7ff8f0102b80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:38.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.993+0000 7ff8f5600640 1 -- 172.21.15.22:0/485770297 >> 172.21.15.22:0/485770297 conn(0x7ff8f0077770 msgr2=0x7ff8f0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:38.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.994+0000 7ff8f5600640 1 -- 172.21.15.22:0/485770297 shutdown_connections 2024-03-24T06:44:38.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.994+0000 7ff8f5600640 1 -- 172.21.15.22:0/485770297 wait complete. 2024-03-24T06:44:38.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.995+0000 7ff8f5600640 1 Processor -- start 2024-03-24T06:44:38.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.996+0000 7ff8f5600640 1 -- start start 2024-03-24T06:44:38.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.998+0000 7ff8f5600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff8f0100790 0x7ff8f0072d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:38.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.999+0000 7ff8f5600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f01030c0 0x7ff8f00732a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:38.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:38.999+0000 7ff8f5600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8f5600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f006c5f0 con 0x7ff8f0100790 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8f5600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f006c760 con 0x7ff8f01030c0 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8f5600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f006c920 con 0x7ff8f010e0e0 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50272/0 (socket says 172.21.15.22:50272) 2024-03-24T06:44:39.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8ef400640 1 -- 172.21.15.22:0/2574805992 learned_addr learned my addr 172.21.15.22:0/2574805992 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:39.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.000+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f01030c0 0x7ff8f00732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.001+0000 7ff8ef400640 1 -- 172.21.15.22:0/2574805992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f01030c0 msgr2=0x7ff8f00732a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.001+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f01030c0 0x7ff8f00732a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.001+0000 7ff8ef400640 1 -- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff8f0100790 msgr2=0x7ff8f0072d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.001+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff8f0100790 0x7ff8f0072d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.001+0000 7ff8ef400640 1 -- 172.21.15.22:0/2574805992 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e8009900 con 0x7ff8f010e0e0 2024-03-24T06:44:39.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.002+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c7c0 tx=0x7ff8e000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.002+0000 7ff8e7400640 1 -- 172.21.15.22:0/2574805992 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00077a0 con 0x7ff8f010e0e0 2024-03-24T06:44:39.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.003+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f006cc00 con 0x7ff8f010e0e0 2024-03-24T06:44:39.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.003+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f0070ba0 con 0x7ff8f010e0e0 2024-03-24T06:44:39.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.003+0000 7ff8e7400640 1 -- 172.21.15.22:0/2574805992 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e0007de0 con 0x7ff8f010e0e0 2024-03-24T06:44:39.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.003+0000 7ff8e7400640 1 -- 172.21.15.22:0/2574805992 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00077a0 con 0x7ff8f010e0e0 2024-03-24T06:44:39.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.015+0000 7ff8e7400640 1 -- 172.21.15.22:0/2574805992 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7ff8e001dc30 con 0x7ff8f010e0e0 2024-03-24T06:44:39.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.019+0000 7ff8e7400640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 0x7ff8d40a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.019+0000 7ff8e7400640 1 -- 172.21.15.22:0/2574805992 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff8e00d5f00 con 0x7ff8f010e0e0 2024-03-24T06:44:39.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.021+0000 7ff8eea00640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 0x7ff8d40a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.029+0000 7ff8eea00640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 0x7ff8d40a4880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006010 tx=0x7ff8e8009870 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:38 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:44:39.306 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:38 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:44:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:38 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:44:39.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.493+0000 7ff8f8409740 1 Processor -- start 2024-03-24T06:44:39.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.493+0000 7ff8f8409740 1 -- start start 2024-03-24T06:44:39.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c91786c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178784e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c9177db790 con 0x55c91786da60 2024-03-24T06:44:39.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c9177d8cc0 con 0x55c9178760f0 2024-03-24T06:44:39.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.494+0000 7ff8f8409740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c9177d44a0 con 0x55c91786cd80 2024-03-24T06:44:39.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178784e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178784e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:56876/0 (socket says 172.21.15.22:56876) 2024-03-24T06:44:39.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8ee000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8ee000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50286/0 (socket says 172.21.15.22:50286) 2024-03-24T06:44:39.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8ee000640 1 -- 172.21.15.22:0/3512326103 learned_addr learned my addr 172.21.15.22:0/3512326103 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:39.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.495+0000 7ff8eea00640 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c91786c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8eea00640 1 -- 172.21.15.22:0/3512326103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 msgr2=0x55c91786d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8eea00640 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8eea00640 1 -- 172.21.15.22:0/3512326103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 msgr2=0x55c9178784e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8eea00640 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178784e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8eea00640 1 -- 172.21.15.22:0/3512326103 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c917795c20 con 0x55c91786da60 2024-03-24T06:44:39.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.497+0000 7ff8ee000640 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:39.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.498+0000 7ff8eea00640 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c91786c7b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8037040 tx=0x7ff8e803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28d0ed12b5def57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.500+0000 7ff8db400640 1 -- 172.21.15.22:0/3512326103 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80348c0 con 0x55c91786da60 2024-03-24T06:44:39.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.501+0000 7ff8db400640 1 -- 172.21.15.22:0/3512326103 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e8032bd0 con 0x55c91786da60 2024-03-24T06:44:39.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.501+0000 7ff8db400640 1 -- 172.21.15.22:0/3512326103 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e803fc90 con 0x55c91786da60 2024-03-24T06:44:39.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.503+0000 7ff8f8409740 1 -- 172.21.15.22:0/3512326103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 msgr2=0x55c91786c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.503+0000 7ff8f8409740 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c91786c7b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8037040 tx=0x7ff8e803c000 comp rx=0 tx=0).stop 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.505+0000 7ff8f8409740 1 -- 172.21.15.22:0/3512326103 shutdown_connections 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.505+0000 7ff8f8409740 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.505+0000 7ff8f8409740 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.505+0000 7ff8f8409740 1 --2- 172.21.15.22:0/3512326103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c91786c7b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.506+0000 7ff8f8409740 1 -- 172.21.15.22:0/3512326103 >> 172.21.15.22:0/3512326103 conn(0x55c917833b30 msgr2=0x55c9178677d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:39.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.506+0000 7ff8f8409740 1 -- 172.21.15.22:0/3512326103 shutdown_connections 2024-03-24T06:44:39.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.506+0000 7ff8f8409740 1 -- 172.21.15.22:0/3512326103 wait complete. 2024-03-24T06:44:39.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.507+0000 7ff8f8409740 1 -- start start 2024-03-24T06:44:39.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.507+0000 7ff8f8409740 1 Processor -- start 2024-03-24T06:44:39.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.510+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.513+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 0x55c917873550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.513+0000 7ff8eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.513+0000 7ff8eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:56890/0 (socket says 172.21.15.22:56890) 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.513+0000 7ff8eea00640 1 -- 172.21.15.22:0/2432811104 learned_addr learned my addr 172.21.15.22:0/2432811104 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.513+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c9178760f0 0x55c917873a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.514+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c917778f20 con 0x55c9178760f0 2024-03-24T06:44:39.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.514+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c917778bc0 con 0x55c91786cd80 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.514+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c9175421c0 con 0x55c91786da60 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.514+0000 7ff8f8409740 10 client.0 _ll_register_callbacks cb 0x55c9174e4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.514+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c9178760f0 0x55c917873a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 0x55c917873550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8eea00640 1 -- 172.21.15.22:0/2432811104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 msgr2=0x55c917873550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8eea00640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 0x55c917873550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8eea00640 1 -- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c9178760f0 msgr2=0x55c917873a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8eea00640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c9178760f0 0x55c917873a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.515+0000 7ff8eea00640 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f0110fc0 con 0x55c91786cd80 2024-03-24T06:44:39.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.516+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 0x55c917873550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:44:39.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.517+0000 7ff8eea00640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8032f70 tx=0x7ff8e8032fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a56a8814d4708a4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.517+0000 7ff8cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:39.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.517+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e803f9d0 con 0x55c91786cd80 2024-03-24T06:44:39.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.518+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff8e8044040 con 0x55c91786cd80 2024-03-24T06:44:39.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.519+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c9177cc160 con 0x55c91786cd80 2024-03-24T06:44:39.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.520+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff8e803fb30 con 0x55c91786cd80 2024-03-24T06:44:39.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.520+0000 7ff8f8409740 10 client.26399 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:39.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.520+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c9177d2b20 con 0x55c91786cd80 2024-03-24T06:44:39.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c9177cc160 con 0x55c91786cd80 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7ff8e8062070 con 0x55c91786cd80 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 10 client.26399 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 10 client.26399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 20 client.26399 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178860f0 0x55c917888510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 4 client.26399 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:39.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.521+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9174d37e0 con 0x55c9178860f0 2024-03-24T06:44:39.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.524+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178860f0 0x55c917888510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.525+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178860f0 0x55c917888510 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c45c20e7e7d971 server_cookie=e4c7a1ce4984ec5a in_seq=0 out_seq=0 2024-03-24T06:44:39.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.525+0000 7ff8cfe00640 10 client.26399 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:39.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.528+0000 7ff8cfe00640 1 -- 172.21.15.22:0/2432811104 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c9174d37e0 con 0x55c9178860f0 2024-03-24T06:44:39.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.528+0000 7ff8cfe00640 10 client.26399 handle_command_reply: tid=0 2024-03-24T06:44:39.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.528+0000 7ff8f8409740 1 client.26399 shutdown 2024-03-24T06:44:39.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.530+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 msgr2=0x55c917873010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.531+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8032f70 tx=0x7ff8e8032fa0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 shutdown_connections 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c9178760f0 0x55c917873a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786da60 0x55c917873550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178860f0 0x55c917888510 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2432811104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917873010 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8cfe00640 0 client.26399 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:39.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 >> 172.21.15.22:0/2432811104 conn(0x55c917833b30 msgr2=0x55c917869520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:39.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 shutdown_connections 2024-03-24T06:44:39.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.533+0000 7ff8f8409740 1 -- 172.21.15.22:0/2432811104 wait complete. 2024-03-24T06:44:39.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.534+0000 7ff8f8409740 20 client.26399 trim_cache size 0 max 16384 2024-03-24T06:44:39.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.571+0000 7ff8f8409740 1 Processor -- start 2024-03-24T06:44:39.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.571+0000 7ff8f8409740 1 -- start start 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917875740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917871f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c91786de40 con 0x55c91786da60 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c917869b30 con 0x55c91786cd80 2024-03-24T06:44:39.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.572+0000 7ff8f8409740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c91786d6c0 con 0x55c9178760f0 2024-03-24T06:44:39.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917875740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8eea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917875740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:56902/0 (socket says 172.21.15.22:56902) 2024-03-24T06:44:39.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50304/0 (socket says 172.21.15.22:50304) 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8ef400640 1 -- 172.21.15.22:0/2057708171 learned_addr learned my addr 172.21.15.22:0/2057708171 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.573+0000 7ff8ee000640 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917871f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.574+0000 7ff8ef400640 1 -- 172.21.15.22:0/2057708171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 msgr2=0x55c917875740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.574+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917875740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.574+0000 7ff8ef400640 1 -- 172.21.15.22:0/2057708171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 msgr2=0x55c917871f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.574+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917871f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.574+0000 7ff8ef400640 1 -- 172.21.15.22:0/2057708171 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c91776e9a0 con 0x55c9178760f0 2024-03-24T06:44:39.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.575+0000 7ff8ef400640 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00a22a0 tx=0x7ff8e008d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6955b4ce4e4d61d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.576+0000 7ff8daa00640 1 -- 172.21.15.22:0/2057708171 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00a25a0 con 0x55c9178760f0 2024-03-24T06:44:39.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.576+0000 7ff8daa00640 1 -- 172.21.15.22:0/2057708171 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e00a66f0 con 0x55c9178760f0 2024-03-24T06:44:39.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.576+0000 7ff8daa00640 1 -- 172.21.15.22:0/2057708171 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00aa8b0 con 0x55c9178760f0 2024-03-24T06:44:39.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 -- 172.21.15.22:0/2057708171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 msgr2=0x55c917867300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00a22a0 tx=0x7ff8e008d040 comp rx=0 tx=0).stop 2024-03-24T06:44:39.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 -- 172.21.15.22:0/2057708171 shutdown_connections 2024-03-24T06:44:39.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9178760f0 0x55c917867300 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917871f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.577+0000 7ff8f8409740 1 --2- 172.21.15.22:0/2057708171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c91786cd80 0x55c917875740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.578+0000 7ff8f8409740 1 -- 172.21.15.22:0/2057708171 >> 172.21.15.22:0/2057708171 conn(0x55c917833b30 msgr2=0x55c917869420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:39.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.581+0000 7ff8f8409740 1 -- 172.21.15.22:0/2057708171 shutdown_connections 2024-03-24T06:44:39.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.582+0000 7ff8f8409740 1 -- 172.21.15.22:0/2057708171 wait complete. 2024-03-24T06:44:39.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.582+0000 7ff8f8409740 1 -- start start 2024-03-24T06:44:39.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.583+0000 7ff8f8409740 1 Processor -- start 2024-03-24T06:44:39.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.584+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.584+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917870420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.584+0000 7ff8f8409740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8f8409740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c9178eb610 con 0x55c91786da60 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8f8409740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c9178eb780 con 0x55c9178760f0 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8f8409740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c9178eb940 con 0x55c91786cd80 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8ef400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:56906/0 (socket says 172.21.15.22:56906) 2024-03-24T06:44:39.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8ef400640 1 -- 172.21.15.22:0/441041990 learned_addr learned my addr 172.21.15.22:0/441041990 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:39.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8eea00640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8f8409740 10 client.0 _ll_register_callbacks cb 0x7ff8e001df00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:39.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.585+0000 7ff8ef400640 1 -- 172.21.15.22:0/441041990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 msgr2=0x55c91786fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8ef400640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8ef400640 1 -- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 msgr2=0x55c917870420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:39.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8ef400640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917870420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8ef400640 1 -- 172.21.15.22:0/441041990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f0070ba0 con 0x55c9178760f0 2024-03-24T06:44:39.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8eea00640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:44:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8ef400640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00c2010 tx=0x7ff8e00076c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97694c65268a6cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.586+0000 7ff8cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.587+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00a25a0 con 0x55c9178760f0 2024-03-24T06:44:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.588+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ff8e00c20b0 con 0x55c9178760f0 2024-03-24T06:44:39.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.588+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c917867960 con 0x55c9178760f0 2024-03-24T06:44:39.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.588+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff8e00aa8b0 con 0x55c9178760f0 2024-03-24T06:44:39.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.588+0000 7ff8f8409740 10 client.26405 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:39.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.589+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c917794480 con 0x55c9178760f0 2024-03-24T06:44:39.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.589+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e008b020 con 0x55c9178760f0 2024-03-24T06:44:39.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.589+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7ff8e00a19f0 con 0x55c9178760f0 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 10 client.26405 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 20 client.26405 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178ecce0 0x55c9178ef120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 4 client.26405 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff8e8032bd0 con 0x55c9178ecce0 2024-03-24T06:44:39.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.591+0000 7ff8eea00640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178ecce0 0x55c9178ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:39.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.592+0000 7ff8eea00640 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178ecce0 0x55c9178ef120 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b70e6975ca24a712 server_cookie=836cc845a7efc6f1 in_seq=0 out_seq=0 2024-03-24T06:44:39.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.592+0000 7ff8cfe00640 10 client.26405 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:39.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.593+0000 7ff8cfe00640 1 -- 172.21.15.22:0/441041990 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7ff8e8032bd0 con 0x55c9178ecce0 2024-03-24T06:44:39.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.593+0000 7ff8cfe00640 10 client.26405 handle_command_reply: tid=0 2024-03-24T06:44:39.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.593+0000 7ff8f8409740 1 client.26405 shutdown 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.593+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 msgr2=0x55c9178709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.593+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00c2010 tx=0x7ff8e00076c0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 shutdown_connections 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c9178ecce0 0x55c9178ef120 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c9178760f0 0x55c9178709a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c91786da60 0x55c917870420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 --2- 172.21.15.22:0/441041990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c91786cd80 0x55c91786fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 >> 172.21.15.22:0/441041990 conn(0x55c917833b30 msgr2=0x55c917868840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:39.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.594+0000 7ff8cfe00640 0 client.26405 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:39.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.595+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 shutdown_connections 2024-03-24T06:44:39.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.595+0000 7ff8f8409740 1 -- 172.21.15.22:0/441041990 wait complete. 2024-03-24T06:44:39.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.595+0000 7ff8f8409740 20 client.26405 trim_cache size 0 max 16384 2024-03-24T06:44:39.598 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:44:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:44:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:44:39.598 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 msgr2=0x7ff8d40a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 0x7ff8d40a4880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006010 tx=0x7ff8e8009870 comp rx=0 tx=0).stop 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 msgr2=0x7ff8f006be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000c7c0 tx=0x7ff8e000cc80 comp rx=0 tx=0).stop 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 shutdown_connections 2024-03-24T06:44:39.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff8d40a23c0 0x7ff8d40a4880 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0e0 0x7ff8f006be60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f01030c0 0x7ff8f00732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 --2- 172.21.15.22:0/2574805992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff8f0100790 0x7ff8f0072d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.597+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 >> 172.21.15.22:0/2574805992 conn(0x7ff8f0077770 msgr2=0x7ff8f0100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.598+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 shutdown_connections 2024-03-24T06:44:39.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:39.598+0000 7ff8f5600640 1 -- 172.21.15.22:0/2574805992 wait complete. 2024-03-24T06:44:40.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:39 smithi155 ceph-mon[36262]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 4.90k op/s 2024-03-24T06:44:40.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:39 smithi022 ceph-mon[32003]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 4.90k op/s 2024-03-24T06:44:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:39 smithi143 ceph-mon[36520]: pgmap v143: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 4.90k op/s 2024-03-24T06:44:41.141 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:44:41.142 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:44:42.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:41 smithi155 ceph-mon[36262]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 MiB/s wr, 4.88k op/s 2024-03-24T06:44:42.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:41 smithi022 ceph-mon[32003]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 MiB/s wr, 4.88k op/s 2024-03-24T06:44:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:41 smithi143 ceph-mon[36520]: pgmap v144: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 MiB/s wr, 4.88k op/s 2024-03-24T06:44:44.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:44 smithi155 ceph-mon[36262]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 MiB/s wr, 6.02k op/s 2024-03-24T06:44:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:44 smithi022 ceph-mon[32003]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 MiB/s wr, 6.02k op/s 2024-03-24T06:44:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:44 smithi143 ceph-mon[36520]: pgmap v145: 129 pgs: 129 active+clean; 16 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 MiB/s wr, 6.02k op/s 2024-03-24T06:44:46.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:46 smithi022 ceph-mon[32003]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 5.01k op/s 2024-03-24T06:44:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:46 smithi143 ceph-mon[36520]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 5.01k op/s 2024-03-24T06:44:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:46 smithi155 ceph-mon[36262]: pgmap v146: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 25 MiB/s wr, 5.01k op/s 2024-03-24T06:44:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:48 smithi022 ceph-mon[32003]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 5.45k op/s 2024-03-24T06:44:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:48 smithi143 ceph-mon[36520]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 5.45k op/s 2024-03-24T06:44:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:48 smithi155 ceph-mon[36262]: pgmap v147: 129 pgs: 129 active+clean; 16 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 MiB/s wr, 5.45k op/s 2024-03-24T06:44:48.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.711+0000 7f4b97200640 1 -- 172.21.15.22:0/3698279312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90112540 msgr2=0x7f4b90114930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:48.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.711+0000 7f4b97200640 1 --2- 172.21.15.22:0/3698279312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90112540 0x7f4b90114930 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009cd0 tx=0x7f4b8802c0e0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 -- 172.21.15.22:0/3698279312 shutdown_connections 2024-03-24T06:44:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 --2- 172.21.15.22:0/3698279312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90112540 0x7f4b90114930 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 --2- 172.21.15.22:0/3698279312 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90076560 0x7f4b90112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 --2- 172.21.15.22:0/3698279312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90075c40 0x7f4b90076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 -- 172.21.15.22:0/3698279312 >> 172.21.15.22:0/3698279312 conn(0x7f4b900ff370 msgr2=0x7f4b90101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 -- 172.21.15.22:0/3698279312 shutdown_connections 2024-03-24T06:44:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.712+0000 7f4b97200640 1 -- 172.21.15.22:0/3698279312 wait complete. 2024-03-24T06:44:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.714+0000 7f4b97200640 1 Processor -- start 2024-03-24T06:44:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.714+0000 7f4b97200640 1 -- start start 2024-03-24T06:44:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.715+0000 7f4b97200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 0x7f4b90072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.715+0000 7f4b97200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49202/0 (socket says 172.21.15.22:49202) 2024-03-24T06:44:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b8fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 0x7f4b90072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b97200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90112540 0x7f4b9006c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b8fe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 0x7f4b90072e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47772/0 (socket says 172.21.15.22:47772) 2024-03-24T06:44:48.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.716+0000 7f4b8f400640 1 -- 172.21.15.22:0/3815876537 learned_addr learned my addr 172.21.15.22:0/3815876537 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:48.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.717+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9006c7e0 con 0x7f4b90075c40 2024-03-24T06:44:48.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.717+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9006c950 con 0x7f4b90112540 2024-03-24T06:44:48.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.717+0000 7f4b95200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90112540 0x7f4b9006c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.717+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9006cb10 con 0x7f4b90076560 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.718+0000 7f4b8f400640 1 -- 172.21.15.22:0/3815876537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90112540 msgr2=0x7f4b9006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.718+0000 7f4b8f400640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90112540 0x7f4b9006c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.718+0000 7f4b8f400640 1 -- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 msgr2=0x7f4b90072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.718+0000 7f4b8f400640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 0x7f4b90072e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:48.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.718+0000 7f4b8f400640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c009490 con 0x7f4b90076560 2024-03-24T06:44:48.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.719+0000 7f4b8f400640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00ea60 tx=0x7f4b7c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:48.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.720+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c00cde0 con 0x7f4b90076560 2024-03-24T06:44:48.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.721+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b7c0035f0 con 0x7f4b90076560 2024-03-24T06:44:48.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.721+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c010690 con 0x7f4b90076560 2024-03-24T06:44:48.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.721+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b88009900 con 0x7f4b90076560 2024-03-24T06:44:48.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.721+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b90070b90 con 0x7f4b90076560 2024-03-24T06:44:48.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.731+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b90077380 con 0x7f4b90076560 2024-03-24T06:44:48.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.734+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f4b7c0107f0 con 0x7f4b90076560 2024-03-24T06:44:48.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.738+0000 7f4b8cc00640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 0x7f4b740a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:48.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.740+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4b7c014070 con 0x7f4b90076560 2024-03-24T06:44:48.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.740+0000 7f4b8fe00640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 0x7f4b740a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:48.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.742+0000 7f4b8fe00640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 0x7f4b740a47f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005d00 tx=0x7f4b8000b410 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:48.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:48.744+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b7c097110 con 0x7f4b90076560 2024-03-24T06:44:49.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.262+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b900715d0 con 0x7f4b90076560 2024-03-24T06:44:49.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.264+0000 7f4b8cc00640 1 -- 172.21.15.22:0/3815876537 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11947 (secure 0 0 0) 0x7f4b7c097110 con 0x7f4b90076560 2024-03-24T06:44:49.268 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:44:49.268 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:44:37.321535+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24601,24584,24532,24565,24544]},"id":1}]} 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 34 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.268+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 msgr2=0x7f4b740a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.268+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 0x7f4b740a47f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005d00 tx=0x7f4b8000b410 comp rx=0 tx=0).stop 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.268+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 msgr2=0x7f4b90073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.268+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00ea60 tx=0x7f4b7c00ef20 comp rx=0 tx=0).stop 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 shutdown_connections 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4b740a2330 0x7f4b740a47f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b90112540 0x7f4b9006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b90076560 0x7f4b90073370 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 --2- 172.21.15.22:0/3815876537 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b90075c40 0x7f4b90072e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.269+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 >> 172.21.15.22:0/3815876537 conn(0x7f4b900ff370 msgr2=0x7f4b90113bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.270+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 shutdown_connections 2024-03-24T06:44:49.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.270+0000 7f4b97200640 1 -- 172.21.15.22:0/3815876537 wait complete. 2024-03-24T06:44:49.303 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.706+0000 7f69a9e00640 1 -- 172.21.15.22:0/2749941748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4076560 msgr2=0x7f69a4112010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.706+0000 7f69a9e00640 1 --2- 172.21.15.22:0/2749941748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4076560 0x7f69a4112010 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6994009930 tx=0x7f699402bc30 comp rx=0 tx=0).stop 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.708+0000 7f69a9e00640 1 -- 172.21.15.22:0/2749941748 shutdown_connections 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.708+0000 7f69a9e00640 1 --2- 172.21.15.22:0/2749941748 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69a4112550 0x7f69a4114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.708+0000 7f69a9e00640 1 --2- 172.21.15.22:0/2749941748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4076560 0x7f69a4112010 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.708+0000 7f69a9e00640 1 --2- 172.21.15.22:0/2749941748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.708+0000 7f69a9e00640 1 -- 172.21.15.22:0/2749941748 >> 172.21.15.22:0/2749941748 conn(0x7f69a40ff380 msgr2=0x7f69a41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.709+0000 7f69a9e00640 1 -- 172.21.15.22:0/2749941748 shutdown_connections 2024-03-24T06:44:49.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.709+0000 7f69a9e00640 1 -- 172.21.15.22:0/2749941748 wait complete. 2024-03-24T06:44:49.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.710+0000 7f69a9e00640 1 Processor -- start 2024-03-24T06:44:49.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.710+0000 7f69a9e00640 1 -- start start 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69a4076560 0x7f69a4111f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 0x7f69a410aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f69a410b330 con 0x7f69a4076560 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f69a410b4a0 con 0x7f69a4075c40 2024-03-24T06:44:49.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a9e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f69a410b660 con 0x7f69a4112550 2024-03-24T06:44:49.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48862/0 (socket says 172.21.15.22:48862) 2024-03-24T06:44:49.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.711+0000 7f69a2a00640 1 -- 172.21.15.22:0/3157381421 learned_addr learned my addr 172.21.15.22:0/3157381421 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:49.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.712+0000 7f69a3400640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 0x7f69a410aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 -- 172.21.15.22:0/3157381421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 msgr2=0x7f69a4111a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 -- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69a4076560 msgr2=0x7f69a4111f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69a4076560 0x7f69a4111f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 -- 172.21.15.22:0/3157381421 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6994009560 con 0x7f69a4112550 2024-03-24T06:44:49.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:44:49.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.713+0000 7f69a3400640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 0x7f69a410aba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f699c00bdf0 tx=0x7f699c00bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.717+0000 7f69a8a00640 1 -- 172.21.15.22:0/3157381421 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c00ce70 con 0x7f69a4112550 2024-03-24T06:44:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.717+0000 7f69a8a00640 1 -- 172.21.15.22:0/3157381421 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699c014920 con 0x7f69a4112550 2024-03-24T06:44:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.717+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a410f520 con 0x7f69a4112550 2024-03-24T06:44:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.717+0000 7f69a8a00640 1 -- 172.21.15.22:0/3157381421 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c0129a0 con 0x7f69a4112550 2024-03-24T06:44:49.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.717+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a410fa30 con 0x7f69a4112550 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.729+0000 7f69a8a00640 1 -- 172.21.15.22:0/3157381421 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f699c014a90 con 0x7f69a4112550 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.732+0000 7f69a8a00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 0x7f69880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.732+0000 7f69a8a00640 1 -- 172.21.15.22:0/3157381421 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f699c0cde70 con 0x7f69a4112550 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.732+0000 7f69acb94740 1 Processor -- start 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 -- start start 2024-03-24T06:44:49.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 0x7f69880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5430240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5430780 0x559dd5432b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559dd53956e0 con 0x559dd5427ba0 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559dd5392c10 con 0x559dd5430780 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.733+0000 7f69acb94740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559dd538e3f0 con 0x559dd5427280 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.734+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 0x7f69880a4770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f699000aca0 tx=0x7f699000a680 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.734+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.734+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49240/0 (socket says 172.21.15.22:49240) 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.734+0000 7f69a2a00640 1 -- 172.21.15.22:0/2340456016 learned_addr learned my addr 172.21.15.22:0/2340456016 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2000640 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5430240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 -- 172.21.15.22:0/2340456016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5430780 msgr2=0x559dd5432b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5430780 0x559dd5432b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 -- 172.21.15.22:0/2340456016 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 msgr2=0x559dd5430240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5430240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 -- 172.21.15.22:0/2340456016 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd534fe40 con 0x559dd5427280 2024-03-24T06:44:49.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.735+0000 7f69a2a00640 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f699000af70 tx=0x7f6990011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c2bc253e9455d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.736+0000 7f698fe00640 1 -- 172.21.15.22:0/2340456016 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699001d860 con 0x559dd5427280 2024-03-24T06:44:49.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.736+0000 7f698fe00640 1 -- 172.21.15.22:0/2340456016 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699000f3c0 con 0x559dd5427280 2024-03-24T06:44:49.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.736+0000 7f698fe00640 1 -- 172.21.15.22:0/2340456016 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699001dea0 con 0x559dd5427280 2024-03-24T06:44:49.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.737+0000 7f69acb94740 1 -- 172.21.15.22:0/2340456016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 msgr2=0x559dd5427660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.737+0000 7f69acb94740 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f699000af70 tx=0x7f6990011000 comp rx=0 tx=0).stop 2024-03-24T06:44:49.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.738+0000 7f69acb94740 1 -- 172.21.15.22:0/2340456016 shutdown_connections 2024-03-24T06:44:49.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.738+0000 7f69acb94740 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5430780 0x559dd5432b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.738+0000 7f69acb94740 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5430240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.738+0000 7f69acb94740 1 --2- 172.21.15.22:0/2340456016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427280 0x559dd5427660 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.738+0000 7f69acb94740 1 -- 172.21.15.22:0/2340456016 >> 172.21.15.22:0/2340456016 conn(0x559dd54219b0 msgr2=0x559dd5421d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.740+0000 7f69acb94740 1 -- 172.21.15.22:0/2340456016 shutdown_connections 2024-03-24T06:44:49.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.740+0000 7f69acb94740 1 -- 172.21.15.22:0/2340456016 wait complete. 2024-03-24T06:44:49.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.741+0000 7f69acb94740 1 -- start start 2024-03-24T06:44:49.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.741+0000 7f69acb94740 1 Processor -- start 2024-03-24T06:44:49.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.742+0000 7f69acb94740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd542d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.742+0000 7f69acb94740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd542de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.742+0000 7f69acb94740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd542d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd542d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48892/0 (socket says 172.21.15.22:48892) 2024-03-24T06:44:49.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69a2000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd542de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69a2a00640 1 -- 172.21.15.22:0/1067150382 learned_addr learned my addr 172.21.15.22:0/1067150382 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559dd5333060 con 0x559dd5427ba0 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559dd5332d00 con 0x559dd5427280 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559dd50fc1c0 con 0x559dd5430780 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69a3400640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.744+0000 7f69a3400640 1 -- 172.21.15.22:0/1067150382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 msgr2=0x559dd542d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.744+0000 7f69a3400640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd542d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.744+0000 7f69a3400640 1 -- 172.21.15.22:0/1067150382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 msgr2=0x559dd542de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.743+0000 7f69acb94740 10 client.0 _ll_register_callbacks cb 0x559dd509ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.744+0000 7f69a3400640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd542de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.744+0000 7f69a3400640 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f699000acd0 con 0x559dd5430780 2024-03-24T06:44:49.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.745+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559dd508d7e0 con 0x559dd5430780 2024-03-24T06:44:49.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.745+0000 7f69a3400640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e360 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f699c0cb310 tx=0x7f699c099fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3d72c9598062960 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.745+0000 7f698cc00640 10 client.26452 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:44:49.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.745+0000 7f698cc00640 1 -- 172.21.15.22:0/1067150382 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c09e700 con 0x559dd5430780 2024-03-24T06:44:49.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.747+0000 7f698cc00640 1 -- 172.21.15.22:0/1067150382 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f699c014e40 con 0x559dd5430780 2024-03-24T06:44:49.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.748+0000 7f698cc00640 1 -- 172.21.15.22:0/1067150382 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f699c0a2d50 con 0x559dd5430780 2024-03-24T06:44:49.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.748+0000 7f69acb94740 10 client.26452 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:49.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.748+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({fsmap=34}) -- 0x559dd538ca70 con 0x559dd5430780 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f698cc00640 1 -- 172.21.15.22:0/1067150382 <== mon.1 v2:172.21.15.155:3300/0 4 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f699c0b8d20 con 0x559dd5430780 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 10 client.26452 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 10 client.26452 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 20 client.26452 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd5462b40 0x559dd5464f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 4 client.26452 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dd50ca720 con 0x559dd5462b40 2024-03-24T06:44:49.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.750+0000 7f69a2a00640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd5462b40 0x559dd5464f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.751+0000 7f69a2a00640 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd5462b40 0x559dd5464f30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9f82d37a8e0b81b8 server_cookie=4f3426db94f6e5ca in_seq=0 out_seq=0 2024-03-24T06:44:49.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.751+0000 7f698cc00640 10 client.26452 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:44:49.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f698cc00640 1 -- 172.21.15.22:0/1067150382 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559dd50ca720 con 0x559dd5462b40 2024-03-24T06:44:49.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f698cc00640 10 client.26452 handle_command_reply: tid=0 2024-03-24T06:44:49.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 client.26452 shutdown 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 msgr2=0x559dd542e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e360 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f699c0cb310 tx=0x7f699c099fd0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 shutdown_connections 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd5462b40 0x559dd5464f30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.753+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e360 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd542de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f698cc00640 0 client.26452 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 1 --2- 172.21.15.22:0/1067150382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd542d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 >> 172.21.15.22:0/1067150382 conn(0x559dd54219b0 msgr2=0x559dd5423700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 shutdown_connections 2024-03-24T06:44:49.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 1 -- 172.21.15.22:0/1067150382 wait complete. 2024-03-24T06:44:49.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.754+0000 7f69acb94740 20 client.26452 trim_cache size 0 max 16384 2024-03-24T06:44:49.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 Processor -- start 2024-03-24T06:44:49.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 -- start start 2024-03-24T06:44:49.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd545e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5423510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559dd54265a0 con 0x559dd5427ba0 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559dd5422be0 con 0x559dd5427280 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.794+0000 7f69acb94740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559dd54628f0 con 0x559dd5430780 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.795+0000 7f69a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.795+0000 7f69a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49252/0 (socket says 172.21.15.22:49252) 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.795+0000 7f69a3400640 1 -- 172.21.15.22:0/3985614985 learned_addr learned my addr 172.21.15.22:0/3985614985 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.795+0000 7f69a2000640 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5423510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.795+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd545e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 -- 172.21.15.22:0/3985614985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 msgr2=0x559dd542e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 -- 172.21.15.22:0/3985614985 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 msgr2=0x559dd5423510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5423510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 -- 172.21.15.22:0/3985614985 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd5328b00 con 0x559dd5427280 2024-03-24T06:44:49.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.796+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd545e020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x559dd54657a0 tx=0x7f6990014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4331cd63e8be621 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.797+0000 7f698f400640 1 -- 172.21.15.22:0/3985614985 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990033da0 con 0x559dd5427280 2024-03-24T06:44:49.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.797+0000 7f698f400640 1 -- 172.21.15.22:0/3985614985 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699002d040 con 0x559dd5427280 2024-03-24T06:44:49.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.797+0000 7f698f400640 1 -- 172.21.15.22:0/3985614985 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990004310 con 0x559dd5427280 2024-03-24T06:44:49.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.798+0000 7f69acb94740 1 -- 172.21.15.22:0/3985614985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 msgr2=0x559dd545e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.798+0000 7f69acb94740 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd545e020 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x559dd54657a0 tx=0x7f6990014000 comp rx=0 tx=0).stop 2024-03-24T06:44:49.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.801+0000 7f69acb94740 1 -- 172.21.15.22:0/3985614985 shutdown_connections 2024-03-24T06:44:49.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.801+0000 7f69acb94740 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5430780 0x559dd542e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.801+0000 7f69acb94740 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5427ba0 0x559dd5423510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.801+0000 7f69acb94740 1 --2- 172.21.15.22:0/3985614985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd545e020 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.801+0000 7f69acb94740 1 -- 172.21.15.22:0/3985614985 >> 172.21.15.22:0/3985614985 conn(0x559dd54219b0 msgr2=0x559dd5422e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.802+0000 7f69acb94740 1 -- 172.21.15.22:0/3985614985 shutdown_connections 2024-03-24T06:44:49.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.802+0000 7f69acb94740 1 -- 172.21.15.22:0/3985614985 wait complete. 2024-03-24T06:44:49.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.802+0000 7f69acb94740 1 -- start start 2024-03-24T06:44:49.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.803+0000 7f69acb94740 1 Processor -- start 2024-03-24T06:44:49.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.815+0000 7f69acb94740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd5456050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 0x559dd5456590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 0x559dd5456ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559dd54c7e30 con 0x559dd5430780 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559dd54c7f70 con 0x559dd5427280 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559dd54c8130 con 0x559dd5427ba0 2024-03-24T06:44:49.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.820+0000 7f69acb94740 10 client.0 _ll_register_callbacks cb 0x7f699c014d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.824+0000 7f69a2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 0x559dd5456590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.824+0000 7f69a2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd5456050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.824+0000 7f69a2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 0x559dd5456590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49258/0 (socket says 172.21.15.22:49258) 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.824+0000 7f69a2000640 1 -- 172.21.15.22:0/3381468522 learned_addr learned my addr 172.21.15.22:0/3381468522 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.824+0000 7f69a3400640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 0x559dd5456ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.825+0000 7f69a3400640 1 -- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 msgr2=0x559dd5456590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.825+0000 7f69a3400640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 0x559dd5456590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.825+0000 7f69a3400640 1 -- 172.21.15.22:0/3381468522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 msgr2=0x559dd5456050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.825+0000 7f69a3400640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd5456050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.825+0000 7f69a3400640 1 -- 172.21.15.22:0/3381468522 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dd534fe40 con 0x559dd5430780 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.826+0000 7f69a3400640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 0x559dd5456ad0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f699c096f80 tx=0x7f699c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35f9982c53434a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.826+0000 7f698cc00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.828+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c09e700 con 0x559dd5430780 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.828+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f699c014e40 con 0x559dd5430780 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.828+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f699000f3c0 con 0x559dd5430780 2024-03-24T06:44:49.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.828+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f699c09a380 con 0x559dd5430780 2024-03-24T06:44:49.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.832+0000 7f69acb94740 10 client.16113 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:49.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.832+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559dd534e6a0 con 0x559dd5430780 2024-03-24T06:44:49.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.833+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f699000f3c0 con 0x559dd5430780 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.833+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f699c0b8d20 con 0x559dd5430780 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.833+0000 7f69acb94740 10 client.16113 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.833+0000 7f69acb94740 10 client.16113 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.833+0000 7f69acb94740 20 client.16113 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.834+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd54c94e0 0x559dd54cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.834+0000 7f69acb94740 4 client.16113 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:44:49.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.834+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559dd5119890 con 0x559dd54c94e0 2024-03-24T06:44:49.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.839+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd54c94e0 0x559dd54cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:49.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.840+0000 7f69a2a00640 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd54c94e0 0x559dd54cb8d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=83cbecb37314991f server_cookie=181086bf69bb4c4 in_seq=0 out_seq=0 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f698cc00640 10 client.16113 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f698cc00640 1 -- 172.21.15.22:0/3381468522 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559dd5119890 con 0x559dd54c94e0 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f698cc00640 10 client.16113 handle_command_reply: tid=0 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 client.16113 shutdown 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 msgr2=0x559dd5456ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 0x559dd5456ad0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f699c096f80 tx=0x7f699c004a40 comp rx=0 tx=0).stop 2024-03-24T06:44:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 shutdown_connections 2024-03-24T06:44:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x559dd54c94e0 0x559dd54cb8d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559dd5430780 0x559dd5456ad0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559dd5427ba0 0x559dd5456590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f698cc00640 0 client.16113 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 --2- 172.21.15.22:0/3381468522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559dd5427280 0x559dd5456050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.843+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 >> 172.21.15.22:0/3381468522 conn(0x559dd54219b0 msgr2=0x559dd5423640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.844+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 shutdown_connections 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.844+0000 7f69acb94740 1 -- 172.21.15.22:0/3381468522 wait complete. 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.844+0000 7f69acb94740 20 client.16113 trim_cache size 0 max 16384 2024-03-24T06:44:49.845 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:44:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 msgr2=0x7f69880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 0x7f69880a4770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f699000aca0 tx=0x7f699000a680 comp rx=0 tx=0).stop 2024-03-24T06:44:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 msgr2=0x7f69a410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:49.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 0x7f69a410aba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f699c00bdf0 tx=0x7f699c00bee0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 shutdown_connections 2024-03-24T06:44:49.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f69880a22b0 0x7f69880a4770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69a4112550 0x7f69a410aba0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69a4076560 0x7f69a4111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 --2- 172.21.15.22:0/3157381421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a4075c40 0x7f69a4111a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.846+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 >> 172.21.15.22:0/3157381421 conn(0x7f69a40ff380 msgr2=0x7f69a4113eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.849+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 shutdown_connections 2024-03-24T06:44:49.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:49.849+0000 7f69a9e00640 1 -- 172.21.15.22:0/3157381421 wait complete. 2024-03-24T06:44:49.888 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:44:50.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:50 smithi022 ceph-mon[32003]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 4.35k op/s 2024-03-24T06:44:50.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:50 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3815876537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:44:50.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 -- 172.21.15.22:0/786278404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 msgr2=0x7f9408101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 --2- 172.21.15.22:0/786278404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408101b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009890 tx=0x7f93f402bb70 comp rx=0 tx=0).stop 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 -- 172.21.15.22:0/786278404 shutdown_connections 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 --2- 172.21.15.22:0/786278404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9408102060 0x7f940810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 --2- 172.21.15.22:0/786278404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408101b20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 --2- 172.21.15.22:0/786278404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94081076b0 0x7f9408107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 -- 172.21.15.22:0/786278404 >> 172.21.15.22:0/786278404 conn(0x7f94080fd150 msgr2=0x7f94080ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 -- 172.21.15.22:0/786278404 shutdown_connections 2024-03-24T06:44:50.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.225+0000 7f940ea00640 1 -- 172.21.15.22:0/786278404 wait complete. 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.226+0000 7f940ea00640 1 Processor -- start 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.226+0000 7f940ea00640 1 -- start start 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.226+0000 7f940ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.226+0000 7f940ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 0x7f940819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 0x7f940819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ea00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f940819ee50 con 0x7f94081016a0 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ea00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f940819efc0 con 0x7f9408102060 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f940819f180 con 0x7f94081076b0 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 0x7f940819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 0x7f940819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47856/0 (socket says 172.21.15.22:47856) 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f940ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 0x7f940819e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49274/0 (socket says 172.21.15.22:49274) 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 0x7f940819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48918/0 (socket says 172.21.15.22:48918) 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 -- 172.21.15.22:0/549293194 learned_addr learned my addr 172.21.15.22:0/549293194 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 -- 172.21.15.22:0/549293194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 msgr2=0x7f940819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 0x7f940819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 -- 172.21.15.22:0/549293194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 msgr2=0x7f940819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 0x7f940819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.227+0000 7f9407e00640 1 -- 172.21.15.22:0/549293194 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f8009560 con 0x7f94081016a0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f9407e00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93f800e9c0 tx=0x7f93f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f9404c00640 1 -- 172.21.15.22:0/549293194 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f800cde0 con 0x7f94081016a0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f4009490 con 0x7f94081016a0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f9404c00640 1 -- 172.21.15.22:0/549293194 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93f80035f0 con 0x7f94081016a0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f9404c00640 1 -- 172.21.15.22:0/549293194 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8010670 con 0x7f94081016a0 2024-03-24T06:44:50.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.228+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94081a6b00 con 0x7f94081016a0 2024-03-24T06:44:50.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.231+0000 7f941166e740 1 Processor -- start 2024-03-24T06:44:50.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.231+0000 7f9404c00640 1 -- 172.21.15.22:0/549293194 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f93f8003ad0 con 0x7f94081016a0 2024-03-24T06:44:50.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.231+0000 7f941166e740 1 -- start start 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a23a8780 0x55e2a23aab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2a230d6e0 con 0x55e2a239f280 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2a230ac10 con 0x55e2a239fba0 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f941166e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2a23063f0 con 0x55e2a23a8780 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47860/0 (socket says 172.21.15.22:47860) 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 -- 172.21.15.22:0/4275357614 learned_addr learned my addr 172.21.15.22:0/4275357614 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f9407e00640 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f9407400640 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a23a8780 0x55e2a23aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 -- 172.21.15.22:0/4275357614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a23a8780 msgr2=0x55e2a23aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a23a8780 0x55e2a23aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 -- 172.21.15.22:0/4275357614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 msgr2=0x55e2a23a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.232+0000 7f940ca00640 1 -- 172.21.15.22:0/4275357614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2a22c7e40 con 0x55e2a239f280 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f940ca00640 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9400009fb0 tx=0x7f94000118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f397dfb7a1655d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f9404c00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 0x7f93ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f9404c00640 1 -- 172.21.15.22:0/549293194 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f93f8014070 con 0x7f94081016a0 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f9407e00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 0x7f93ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f93fd200640 1 -- 172.21.15.22:0/4275357614 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400004d20 con 0x55e2a239f280 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f93fd200640 1 -- 172.21.15.22:0/4275357614 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9400004e80 con 0x55e2a239f280 2024-03-24T06:44:50.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.233+0000 7f93fd200640 1 -- 172.21.15.22:0/4275357614 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400005280 con 0x55e2a239f280 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f9407e00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 0x7f93ec0a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93f80cbc70 tx=0x7f93f80d4550 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 -- 172.21.15.22:0/4275357614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 msgr2=0x55e2a239f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9400009fb0 tx=0x7f94000118d0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 -- 172.21.15.22:0/4275357614 shutdown_connections 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a23a8780 0x55e2a23aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 --2- 172.21.15.22:0/4275357614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a239f660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 -- 172.21.15.22:0/4275357614 >> 172.21.15.22:0/4275357614 conn(0x55e2a23999b0 msgr2=0x55e2a2399d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 -- 172.21.15.22:0/4275357614 shutdown_connections 2024-03-24T06:44:50.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.234+0000 7f941166e740 1 -- 172.21.15.22:0/4275357614 wait complete. 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.235+0000 7f941166e740 1 -- start start 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.235+0000 7f941166e740 1 Processor -- start 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.235+0000 7f941166e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 0x55e2a23a5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2a22ab060 con 0x55e2a23a8780 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2a22aad00 con 0x55e2a239f280 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2a20741c0 con 0x55e2a239fba0 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f941166e740 10 client.0 _ll_register_callbacks cb 0x55e2a2016be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f940ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 0x55e2a23a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f940ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 0x55e2a23a5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48938/0 (socket says 172.21.15.22:48938) 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f940ca00640 1 -- 172.21.15.22:0/2079526049 learned_addr learned my addr 172.21.15.22:0/2079526049 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f940ca00640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 -- 172.21.15.22:0/2079526049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 msgr2=0x55e2a23a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 0x55e2a23a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 -- 172.21.15.22:0/2079526049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 msgr2=0x55e2a23a6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f9407400640 1 -- 172.21.15.22:0/2079526049 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9400009c60 con 0x55e2a239fba0 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.236+0000 7f940ca00640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.237+0000 7f9407400640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a5cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f93f40057c0 tx=0x7f93f4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64fe87fe712c3ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.237+0000 7f93f1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.237+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f4034640 con 0x55e2a239fba0 2024-03-24T06:44:50.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.237+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2a2042720 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.238+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f93f40347a0 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.238+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f93f40349c0 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.238+0000 7f941166e740 10 client.26470 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.238+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e2a2304a70 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.238+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e2a2042720 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f93f4032600 con 0x55e2a239fba0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 10 client.26470 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 10 client.26470 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 20 client.26470 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a23b8b40 0x55e2a23baf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 4 client.26470 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2a2091890 con 0x55e2a23b8b40 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.239+0000 7f940ca00640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a23b8b40 0x55e2a23baf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.240+0000 7f940ca00640 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a23b8b40 0x55e2a23baf60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66b3180c5514c90e server_cookie=561786796c9e6a87 in_seq=0 out_seq=0 2024-03-24T06:44:50.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.240+0000 7f93f1600640 10 client.26470 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f93f1600640 1 -- 172.21.15.22:0/2079526049 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2a2091890 con 0x55e2a23b8b40 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f93f1600640 10 client.26470 handle_command_reply: tid=0 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 client.26470 shutdown 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 msgr2=0x55e2a23a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a5cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f93f40057c0 tx=0x7f93f4004270 comp rx=0 tx=0).stop 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 shutdown_connections 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a23b8b40 0x55e2a23baf60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a5cc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 --2- 172.21.15.22:0/2079526049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239f280 0x55e2a23a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f93f1600640 0 client.26470 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:44:50.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 >> 172.21.15.22:0/2079526049 conn(0x55e2a23999b0 msgr2=0x55e2a239b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 shutdown_connections 2024-03-24T06:44:50.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 1 -- 172.21.15.22:0/2079526049 wait complete. 2024-03-24T06:44:50.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.241+0000 7f941166e740 20 client.26470 trim_cache size 0 max 16384 2024-03-24T06:44:50.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.269+0000 7f941166e740 1 Processor -- start 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.269+0000 7f941166e740 1 -- start start 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a23a8780 0x55e2a23988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2a23b4550 con 0x55e2a239f280 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2a23bb590 con 0x55e2a23a8780 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f941166e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2a2395270 con 0x55e2a239fba0 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f940ca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f940ca00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47870/0 (socket says 172.21.15.22:47870) 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f940ca00640 1 -- 172.21.15.22:0/4200372610 learned_addr learned my addr 172.21.15.22:0/4200372610 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f940ca00640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a23a8780 0x55e2a23988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f9407400640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.270+0000 7f940ca00640 1 -- 172.21.15.22:0/4200372610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 msgr2=0x55e2a23a6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f940ca00640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f940ca00640 1 -- 172.21.15.22:0/4200372610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a23a8780 msgr2=0x55e2a23988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f940ca00640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a23a8780 0x55e2a23988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f940ca00640 1 -- 172.21.15.22:0/4200372610 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2a22a0b00 con 0x55e2a239f280 2024-03-24T06:44:50.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f9407400640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f940ca00640 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55e2a239a8f0 tx=0x7f9400009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3ad18238381ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f93f3e00640 1 -- 172.21.15.22:0/4200372610 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940002ea30 con 0x55e2a239f280 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f93f3e00640 1 -- 172.21.15.22:0/4200372610 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940002e160 con 0x55e2a239f280 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.271+0000 7f93f3e00640 1 -- 172.21.15.22:0/4200372610 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940000f740 con 0x55e2a239f280 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- 172.21.15.22:0/4200372610 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 msgr2=0x55e2a23b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55e2a239a8f0 tx=0x7f9400009500 comp rx=0 tx=0).stop 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- 172.21.15.22:0/4200372610 shutdown_connections 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a23a8780 0x55e2a23988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239fba0 0x55e2a23a6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 --2- 172.21.15.22:0/4200372610 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a239f280 0x55e2a23b4010 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- 172.21.15.22:0/4200372610 >> 172.21.15.22:0/4200372610 conn(0x55e2a23999b0 msgr2=0x55e2a23981e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- 172.21.15.22:0/4200372610 shutdown_connections 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- 172.21.15.22:0/4200372610 wait complete. 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.272+0000 7f941166e740 1 -- start start 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 Processor -- start 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239f280 0x55e2a23a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2a241de30 con 0x55e2a23a8780 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2a241df70 con 0x55e2a239fba0 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2a241e0b0 con 0x55e2a239f280 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f941166e740 10 client.0 _ll_register_callbacks cb 0x7f94000093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f9407400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f9407400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48946/0 (socket says 172.21.15.22:48946) 2024-03-24T06:44:50.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f9407400640 1 -- 172.21.15.22:0/15391188 learned_addr learned my addr 172.21.15.22:0/15391188 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f940ca00640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239f280 0x55e2a23a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.273+0000 7f940ca00640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 -- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239f280 msgr2=0x55e2a23a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239f280 0x55e2a23a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 -- 172.21.15.22:0/15391188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 msgr2=0x55e2a23a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 -- 172.21.15.22:0/15391188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2a22c7e40 con 0x55e2a239fba0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f940ca00640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f9407400640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f93f402bbb0 tx=0x7f93f4032e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac7b49d8f2883cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f93f1600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f402cb30 con 0x55e2a239fba0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2a20057e0 con 0x55e2a239fba0 2024-03-24T06:44:50.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.274+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f93f4000e40 con 0x55e2a239fba0 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f93f40042e0 con 0x55e2a239fba0 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 10 client.26441 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e2a22c66a0 con 0x55e2a239fba0 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e2a20057e0 con 0x55e2a239fba0 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f93f4003900 con 0x55e2a239fba0 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 10 client.26441 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 10 client.26441 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 20 client.26441 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a241f420 0x55e2a2421840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 4 client.26441 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.275+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2a1fb1700 con 0x55e2a241f420 2024-03-24T06:44:50.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f940ca00640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a241f420 0x55e2a2421840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f940ca00640 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a241f420 0x55e2a2421840 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bcd0e75b7ac981d1 server_cookie=7ba1a76cf568f391 in_seq=0 out_seq=0 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f93f1600640 10 client.26441 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f93f1600640 1 -- 172.21.15.22:0/15391188 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2a1fb1700 con 0x55e2a241f420 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f93f1600640 10 client.26441 handle_command_reply: tid=0 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.276+0000 7f941166e740 1 client.26441 shutdown 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 msgr2=0x55e2a23a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f93f402bbb0 tx=0x7f93f4032e30 comp rx=0 tx=0).stop 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 shutdown_connections 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e2a241f420 0x55e2a2421840 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2a23a8780 0x55e2a23a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2a239fba0 0x55e2a23a2830 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f93f1600640 0 client.26441 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 --2- 172.21.15.22:0/15391188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2a239f280 0x55e2a23a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 >> 172.21.15.22:0/15391188 conn(0x55e2a23999b0 msgr2=0x55e2a2398a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 shutdown_connections 2024-03-24T06:44:50.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 1 -- 172.21.15.22:0/15391188 wait complete. 2024-03-24T06:44:50.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.277+0000 7f941166e740 20 client.26441 trim_cache size 0 max 16384 2024-03-24T06:44:50.282 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 msgr2=0x7f93ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 0x7f93ec0a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f93f80cbc70 tx=0x7f93f80d4550 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 msgr2=0x7f9408199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f93f800e9c0 tx=0x7f93f800ee80 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 shutdown_connections 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93ec0a2230 0x7f93ec0a46f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94081076b0 0x7f940819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9408102060 0x7f940819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 --2- 172.21.15.22:0/549293194 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94081016a0 0x7f9408199f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 >> 172.21.15.22:0/549293194 conn(0x7f94080fd150 msgr2=0x7f940810e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 shutdown_connections 2024-03-24T06:44:50.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.278+0000 7f940ea00640 1 -- 172.21.15.22:0/549293194 wait complete. 2024-03-24T06:44:50.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:44:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:50 smithi143 ceph-mon[36520]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 4.35k op/s 2024-03-24T06:44:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:50 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3815876537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 -- 172.21.15.22:0/3538558923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 msgr2=0x7fca08114320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 --2- 172.21.15.22:0/3538558923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca08114320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8009900 tx=0x7fc9f802bc40 comp rx=0 tx=0).stop 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 -- 172.21.15.22:0/3538558923 shutdown_connections 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 --2- 172.21.15.22:0/3538558923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 0x7fca08116c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 --2- 172.21.15.22:0/3538558923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca08114320 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 --2- 172.21.15.22:0/3538558923 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca08108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 -- 172.21.15.22:0/3538558923 >> 172.21.15.22:0/3538558923 conn(0x7fca0806c7f0 msgr2=0x7fca0806cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.499+0000 7fca0ea00640 1 -- 172.21.15.22:0/3538558923 shutdown_connections 2024-03-24T06:44:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.500+0000 7fca0ea00640 1 -- 172.21.15.22:0/3538558923 wait complete. 2024-03-24T06:44:50.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.501+0000 7fca0ea00640 1 Processor -- start 2024-03-24T06:44:50.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.502+0000 7fca0ea00640 1 -- start start 2024-03-24T06:44:50.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca081a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 0x7fca081a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fca081a2da0 con 0x7fca081088f0 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fca081a2f10 con 0x7fca08109210 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca0ea00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fca081a30d0 con 0x7fca08114860 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca07400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca07400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48964/0 (socket says 172.21.15.22:48964) 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.503+0000 7fca07400640 1 -- 172.21.15.22:0/1258195306 learned_addr learned my addr 172.21.15.22:0/1258195306 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07e00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca081a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca0ca00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 0x7fca081a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07400640 1 -- 172.21.15.22:0/1258195306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 msgr2=0x7fca081a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07400640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 0x7fca081a2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07400640 1 -- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 msgr2=0x7fca081a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07400640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca081a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07400640 1 -- 172.21.15.22:0/1258195306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f8009560 con 0x7fca08109210 2024-03-24T06:44:50.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.504+0000 7fca07e00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca081a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.505+0000 7fca07400640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8000c00 tx=0x7fc9f802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.506+0000 7fca04c00640 1 -- 172.21.15.22:0/1258195306 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8004390 con 0x7fca08109210 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.506+0000 7fca04c00640 1 -- 172.21.15.22:0/1258195306 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f80044f0 con 0x7fca08109210 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.506+0000 7fca04c00640 1 -- 172.21.15.22:0/1258195306 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8003d00 con 0x7fca08109210 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.506+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca081a32f0 con 0x7fca08109210 2024-03-24T06:44:50.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.506+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca081196d0 con 0x7fca08109210 2024-03-24T06:44:50.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.512+0000 7fca11789740 1 Processor -- start 2024-03-24T06:44:50.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.513+0000 7fca04c00640 1 -- 172.21.15.22:0/1258195306 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc9f8004df0 con 0x7fca08109210 2024-03-24T06:44:50.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.513+0000 7fca11789740 1 -- start start 2024-03-24T06:44:50.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.513+0000 7fca11789740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48980/0 (socket says 172.21.15.22:48980) 2024-03-24T06:44:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca11789740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 0x5636b53d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca0ca00640 1 -- 172.21.15.22:0/505917439 learned_addr learned my addr 172.21.15.22:0/505917439 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca04c00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 0x7fc9ec0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca11789740 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53d2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636b53356e0 con 0x5636b53c7ba0 2024-03-24T06:44:50.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636b5332c10 con 0x5636b53c7280 2024-03-24T06:44:50.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636b532e3f0 con 0x5636b53d0780 2024-03-24T06:44:50.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.515+0000 7fca04c00640 1 -- 172.21.15.22:0/1258195306 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc9f80f0a60 con 0x7fca08109210 2024-03-24T06:44:50.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.516+0000 7fca07e00640 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 0x5636b53d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca07e00640 1 -- 172.21.15.22:0/505917439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 msgr2=0x5636b53d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca07e00640 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53d2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca07e00640 1 -- 172.21.15.22:0/505917439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 msgr2=0x5636b53c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca07e00640 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca07e00640 1 -- 172.21.15.22:0/505917439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636b52efe40 con 0x5636b53c7ba0 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.517+0000 7fca0ca00640 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:50.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.518+0000 7fca07400640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 0x7fc9ec0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.518+0000 7fca07e00640 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 0x5636b53d0240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009c30 tx=0x7fc9f40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6981c84f690649f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.519+0000 7fca07400640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 0x7fc9ec0a4770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80b9c00 tx=0x7fc9f8005e20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.520+0000 7fc9fd200640 1 -- 172.21.15.22:0/505917439 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4012450 con 0x5636b53c7ba0 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.520+0000 7fc9fd200640 1 -- 172.21.15.22:0/505917439 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f4012a90 con 0x5636b53c7ba0 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.520+0000 7fc9fd200640 1 -- 172.21.15.22:0/505917439 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f400fae0 con 0x5636b53c7ba0 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 msgr2=0x5636b53d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 0x5636b53d0240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009c30 tx=0x7fc9f40117b0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 shutdown_connections 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53d2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7ba0 0x5636b53d0240 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009c30 tx=0x7fc9f40117b0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 --2- 172.21.15.22:0/505917439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.521+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 >> 172.21.15.22:0/505917439 conn(0x5636b53c19b0 msgr2=0x5636b53c1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.522+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 shutdown_connections 2024-03-24T06:44:50.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.523+0000 7fca11789740 1 -- 172.21.15.22:0/505917439 wait complete. 2024-03-24T06:44:50.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.523+0000 7fca11789740 1 -- start start 2024-03-24T06:44:50.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.524+0000 7fca11789740 1 Processor -- start 2024-03-24T06:44:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:50 smithi155 ceph-mon[36262]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 MiB/s wr, 4.35k op/s 2024-03-24T06:44:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:50 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3815876537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:44:50.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7280 0x5636b53cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 0x5636b53dc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636b52d3060 con 0x5636b53c7280 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636b52d2d00 con 0x5636b53ce3d0 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636b509c1c0 con 0x5636b53d0780 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.526+0000 7fca11789740 10 client.0 _ll_register_callbacks cb 0x5636b503ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:50.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.544+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 0x5636b53dc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.544+0000 7fca07e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.544+0000 7fca07e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49338/0 (socket says 172.21.15.22:49338) 2024-03-24T06:44:50.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.544+0000 7fca07e00640 1 -- 172.21.15.22:0/3745374152 learned_addr learned my addr 172.21.15.22:0/3745374152 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.544+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7280 0x5636b53cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.545+0000 7fca0ca00640 1 -- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 msgr2=0x5636b53cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.545+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.545+0000 7fca0ca00640 1 -- 172.21.15.22:0/3745374152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7280 msgr2=0x5636b53cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.545+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7280 0x5636b53cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.545+0000 7fca0ca00640 1 -- 172.21.15.22:0/3745374152 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f4009860 con 0x5636b53ce3d0 2024-03-24T06:44:50.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.546+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 0x5636b53dc6e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5636b53c2b40 tx=0x7fca00009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b8d5a6cac54bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.546+0000 7fc9f1600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:50.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.547+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00013e40 con 0x5636b53ce3d0 2024-03-24T06:44:50.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.547+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fca00013110 con 0x5636b53ce3d0 2024-03-24T06:44:50.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.547+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5636b502d7e0 con 0x5636b53ce3d0 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.548+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fca000132d0 con 0x5636b53ce3d0 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.548+0000 7fca11789740 10 client.26453 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.548+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5636b532ca70 con 0x5636b53ce3d0 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca00019070 con 0x5636b53ce3d0 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fca00011440 con 0x5636b53ce3d0 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 10 client.26453 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:50.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 10 client.26453 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:44:50.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 20 client.26453 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:50.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b53e1260 0x5636b53e3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 4 client.26453 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:50.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.549+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636b502d7e0 con 0x5636b53e1260 2024-03-24T06:44:50.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.550+0000 7fca07e00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b53e1260 0x5636b53e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.551+0000 7fca07e00640 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b53e1260 0x5636b53e3680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3005e45ee5cdda1 server_cookie=14ed442bf6788fd6 in_seq=0 out_seq=0 2024-03-24T06:44:50.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.551+0000 7fc9f1600640 10 client.26453 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:44:50.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.552+0000 7fc9f1600640 1 -- 172.21.15.22:0/3745374152 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636b502d7e0 con 0x5636b53e1260 2024-03-24T06:44:50.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.552+0000 7fc9f1600640 10 client.26453 handle_command_reply: tid=0 2024-03-24T06:44:50.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.553+0000 7fca11789740 1 client.26453 shutdown 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.553+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 msgr2=0x5636b53dc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.553+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 0x5636b53dc6e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5636b53c2b40 tx=0x7fca00009520 comp rx=0 tx=0).stop 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 shutdown_connections 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b53e1260 0x5636b53e3680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53ce3d0 0x5636b53dc6e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b53cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 --2- 172.21.15.22:0/3745374152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53c7280 0x5636b53cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 >> 172.21.15.22:0/3745374152 conn(0x5636b53c19b0 msgr2=0x5636b53c3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.554+0000 7fc9f1600640 0 client.26453 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:44:50.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.555+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 shutdown_connections 2024-03-24T06:44:50.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.555+0000 7fca11789740 1 -- 172.21.15.22:0/3745374152 wait complete. 2024-03-24T06:44:50.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.556+0000 7fca11789740 20 client.26453 trim_cache size 0 max 16384 2024-03-24T06:44:50.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 Processor -- start 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 -- start start 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 0x5636b53e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53dc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53d0780 0x5636b53dc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636b53cd740 con 0x5636b53ce3d0 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636b53c79c0 con 0x5636b53d0780 2024-03-24T06:44:50.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.609+0000 7fca11789740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636b53dca90 con 0x5636b53c7280 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.611+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 0x5636b53e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.611+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 0x5636b53e8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49340/0 (socket says 172.21.15.22:49340) 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.611+0000 7fca0ca00640 1 -- 172.21.15.22:0/658451368 learned_addr learned my addr 172.21.15.22:0/658451368 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.611+0000 7fca0ca00640 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53d0780 0x5636b53dc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.612+0000 7fca07e00640 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 -- 172.21.15.22:0/658451368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 msgr2=0x5636b53e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 0x5636b53e8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 -- 172.21.15.22:0/658451368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53d0780 msgr2=0x5636b53dc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53d0780 0x5636b53dc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 -- 172.21.15.22:0/658451368 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636b52c8b00 con 0x5636b53ce3d0 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.613+0000 7fca07e00640 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53dc010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009570 tx=0x7fc9f4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15900df8f2caccbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.614+0000 7fc9f3e00640 1 -- 172.21.15.22:0/658451368 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4012450 con 0x5636b53ce3d0 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.615+0000 7fc9f3e00640 1 -- 172.21.15.22:0/658451368 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f4005040 con 0x5636b53ce3d0 2024-03-24T06:44:50.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.615+0000 7fc9f3e00640 1 -- 172.21.15.22:0/658451368 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f402a5a0 con 0x5636b53ce3d0 2024-03-24T06:44:50.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.616+0000 7fca11789740 1 -- 172.21.15.22:0/658451368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 msgr2=0x5636b53dc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.616+0000 7fca11789740 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53dc010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009570 tx=0x7fc9f4005f10 comp rx=0 tx=0).stop 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.616+0000 7fca11789740 1 -- 172.21.15.22:0/658451368 shutdown_connections 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.616+0000 7fca11789740 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53d0780 0x5636b53dc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.616+0000 7fca11789740 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53dc010 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.617+0000 7fca11789740 1 --2- 172.21.15.22:0/658451368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53c7280 0x5636b53e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.617+0000 7fca11789740 1 -- 172.21.15.22:0/658451368 >> 172.21.15.22:0/658451368 conn(0x5636b53c19b0 msgr2=0x5636b53c8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.617+0000 7fca11789740 1 -- 172.21.15.22:0/658451368 shutdown_connections 2024-03-24T06:44:50.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.617+0000 7fca11789740 1 -- 172.21.15.22:0/658451368 wait complete. 2024-03-24T06:44:50.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.618+0000 7fca11789740 1 -- start start 2024-03-24T06:44:50.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.618+0000 7fca11789740 1 Processor -- start 2024-03-24T06:44:50.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53d4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b5446580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636b53d5170 con 0x5636b53ce3d0 2024-03-24T06:44:50.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636b5446ac0 con 0x5636b53c7280 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.620+0000 7fca11789740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636b5446c00 con 0x5636b53d0780 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca11789740 10 client.0 _ll_register_callbacks cb 0x7fca00009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca0ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49006/0 (socket says 172.21.15.22:49006) 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca0ca00640 1 -- 172.21.15.22:0/3245593334 learned_addr learned my addr 172.21.15.22:0/3245593334 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b5446580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.621+0000 7fca07e00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53d4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 -- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 msgr2=0x5636b5446580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b5446580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 -- 172.21.15.22:0/3245593334 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 msgr2=0x5636b53d4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53d4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 -- 172.21.15.22:0/3245593334 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636b52efe40 con 0x5636b53c7280 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fca0ca00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fca00013040 tx=0x7fca00009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c2791e38351b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fc9f1600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.622+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00013110 con 0x5636b53c7280 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.623+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fca00013270 con 0x5636b53c7280 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.623+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9f4012a90 con 0x5636b53c7280 2024-03-24T06:44:50.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.623+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fca00011440 con 0x5636b53c7280 2024-03-24T06:44:50.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.623+0000 7fca11789740 10 client.26459 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:50.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.623+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5636b52ee6a0 con 0x5636b53c7280 2024-03-24T06:44:50.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.624+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca00019070 con 0x5636b53c7280 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.624+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fca000289f0 con 0x5636b53c7280 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 10 client.26459 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 10 client.26459 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 20 client.26459 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b5448470 0x5636b544a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 4 client.26459 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:44:50.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc9f4012a90 con 0x5636b5448470 2024-03-24T06:44:50.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.626+0000 7fca07e00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b5448470 0x5636b544a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.627+0000 7fca07e00640 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b5448470 0x5636b544a860 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f92f5fd4c5bf844f server_cookie=a5e31b202a89ee99 in_seq=0 out_seq=0 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.627+0000 7fc9f1600640 10 client.26459 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fc9f1600640 1 -- 172.21.15.22:0/3245593334 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc9f4012a90 con 0x5636b5448470 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fc9f1600640 10 client.26459 handle_command_reply: tid=0 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 client.26459 shutdown 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 msgr2=0x5636b53d4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fca00013040 tx=0x7fca00009520 comp rx=0 tx=0).stop 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 shutdown_connections 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5636b5448470 0x5636b544a860 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636b53d0780 0x5636b5446580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636b53ce3d0 0x5636b53d4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 --2- 172.21.15.22:0/3245593334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636b53c7280 0x5636b53d4580 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.628+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 >> 172.21.15.22:0/3245593334 conn(0x5636b53c19b0 msgr2=0x5636b53c01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.630+0000 7fc9f1600640 0 client.26459 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:44:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.631+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 shutdown_connections 2024-03-24T06:44:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.631+0000 7fca11789740 1 -- 172.21.15.22:0/3245593334 wait complete. 2024-03-24T06:44:50.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.631+0000 7fca11789740 20 client.26459 trim_cache size 0 max 16384 2024-03-24T06:44:50.632 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:44:50.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.632+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 msgr2=0x7fc9ec0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.632+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 0x7fc9ec0a4770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80b9c00 tx=0x7fc9f8005e20 comp rx=0 tx=0).stop 2024-03-24T06:44:50.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.632+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 msgr2=0x7fca081a89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:50.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.632+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8000c00 tx=0x7fc9f802c640 comp rx=0 tx=0).stop 2024-03-24T06:44:50.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 shutdown_connections 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9ec0a22b0 0x7fc9ec0a4770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca08114860 0x7fca081a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca08109210 0x7fca081a89c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 --2- 172.21.15.22:0/1258195306 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fca081088f0 0x7fca081a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 >> 172.21.15.22:0/1258195306 conn(0x7fca0806c7f0 msgr2=0x7fca0810b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 shutdown_connections 2024-03-24T06:44:50.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:50.633+0000 7fca0ea00640 1 -- 172.21.15.22:0/1258195306 wait complete. 2024-03-24T06:44:50.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:44:51.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.594+0000 7fab0d800640 1 -- 172.21.15.22:0/818107485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab08102060 msgr2=0x7fab0810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.594+0000 7fab0d800640 1 --2- 172.21.15.22:0/818107485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab08102060 0x7fab0810e570 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faaf4013080 tx=0x7faaf4033ee0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 -- 172.21.15.22:0/818107485 shutdown_connections 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 --2- 172.21.15.22:0/818107485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab08102060 0x7fab0810e570 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 --2- 172.21.15.22:0/818107485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab081016a0 0x7fab08101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 --2- 172.21.15.22:0/818107485 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab08107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 -- 172.21.15.22:0/818107485 >> 172.21.15.22:0/818107485 conn(0x7fab08078b10 msgr2=0x7fab08078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.595+0000 7fab0d800640 1 -- 172.21.15.22:0/818107485 shutdown_connections 2024-03-24T06:44:51.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.597+0000 7fab0d800640 1 -- 172.21.15.22:0/818107485 wait complete. 2024-03-24T06:44:51.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.598+0000 7fab0d800640 1 Processor -- start 2024-03-24T06:44:51.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.598+0000 7fab0d800640 1 -- start start 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 0x7fab0819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 0x7fab0819bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab081958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fab08195ef0 con 0x7fab081076b0 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fab08196060 con 0x7fab08102060 2024-03-24T06:44:51.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab0d800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fab08196220 con 0x7fab081016a0 2024-03-24T06:44:51.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab06000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 0x7fab0819bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab06a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 0x7fab0819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab06000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 0x7fab0819bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49016/0 (socket says 172.21.15.22:49016) 2024-03-24T06:44:51.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab06a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 0x7fab0819b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49362/0 (socket says 172.21.15.22:49362) 2024-03-24T06:44:51.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.599+0000 7fab06000640 1 -- 172.21.15.22:0/2657597112 learned_addr learned my addr 172.21.15.22:0/2657597112 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab081958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 -- 172.21.15.22:0/2657597112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 msgr2=0x7fab0819b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 0x7fab0819b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 -- 172.21.15.22:0/2657597112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 msgr2=0x7fab0819bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 0x7fab0819bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.602+0000 7fab07400640 1 -- 172.21.15.22:0/2657597112 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf4011ca0 con 0x7fab081076b0 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.603+0000 7fab07400640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab081958a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faaf403e040 tx=0x7faaf4011300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.603+0000 7faaff400640 1 -- 172.21.15.22:0/2657597112 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004250 con 0x7fab081076b0 2024-03-24T06:44:51.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.604+0000 7faaff400640 1 -- 172.21.15.22:0/2657597112 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf40043b0 con 0x7fab081076b0 2024-03-24T06:44:51.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.604+0000 7faaff400640 1 -- 172.21.15.22:0/2657597112 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4046860 con 0x7fab081076b0 2024-03-24T06:44:51.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.605+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab081964a0 con 0x7fab081076b0 2024-03-24T06:44:51.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.605+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab081b37a0 con 0x7fab081076b0 2024-03-24T06:44:51.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.612+0000 7fab10563740 1 Processor -- start 2024-03-24T06:44:51.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.612+0000 7fab10563740 1 -- start start 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f74a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f753240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f755b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a96f6b86e0 con 0x55a96f753780 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a96f6b5c10 con 0x55a96f74aba0 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab10563740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a96f6b13f0 con 0x55a96f74a280 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab06a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f753240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab06a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f753240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49018/0 (socket says 172.21.15.22:49018) 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.613+0000 7fab06a00640 1 -- 172.21.15.22:0/2722061079 learned_addr learned my addr 172.21.15.22:0/2722061079 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:51.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.614+0000 7fab06000640 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f755b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.614+0000 7fab06000640 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f74a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.616+0000 7fab06000640 1 -- 172.21.15.22:0/2722061079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 msgr2=0x55a96f753240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.616+0000 7fab06000640 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f753240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.616+0000 7fab06000640 1 -- 172.21.15.22:0/2722061079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 msgr2=0x55a96f755b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.616+0000 7fab06000640 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f755b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.616+0000 7fab06000640 1 -- 172.21.15.22:0/2722061079 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a96f672e40 con 0x55a96f74a280 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.617+0000 7faaff400640 1 -- 172.21.15.22:0/2657597112 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7faaf4003b60 con 0x7fab081076b0 2024-03-24T06:44:51.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.617+0000 7fab06000640 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f74a660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab08197090 tx=0x7faaf8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86bde01a1cc4f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.619+0000 7faaf3e00640 1 -- 172.21.15.22:0/2722061079 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8003e50 con 0x55a96f74a280 2024-03-24T06:44:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.619+0000 7faaff400640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 0x7faaec0a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.619+0000 7faaf3e00640 1 -- 172.21.15.22:0/2722061079 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf8013940 con 0x55a96f74a280 2024-03-24T06:44:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.619+0000 7faaf3e00640 1 -- 172.21.15.22:0/2722061079 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf800f950 con 0x55a96f74a280 2024-03-24T06:44:51.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.621+0000 7fab10563740 1 -- 172.21.15.22:0/2722061079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 msgr2=0x55a96f74a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.621+0000 7fab10563740 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f74a660 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fab08197090 tx=0x7faaf8004730 comp rx=0 tx=0).stop 2024-03-24T06:44:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.622+0000 7fab10563740 1 -- 172.21.15.22:0/2722061079 shutdown_connections 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.622+0000 7fab10563740 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f755b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.622+0000 7fab10563740 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f753240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.622+0000 7fab10563740 1 --2- 172.21.15.22:0/2722061079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f74a660 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.622+0000 7fab10563740 1 -- 172.21.15.22:0/2722061079 >> 172.21.15.22:0/2722061079 conn(0x55a96f7449b0 msgr2=0x55a96f744d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.623+0000 7faaff400640 1 -- 172.21.15.22:0/2657597112 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faaf40ef470 con 0x7fab081076b0 2024-03-24T06:44:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.623+0000 7fab10563740 1 -- 172.21.15.22:0/2722061079 shutdown_connections 2024-03-24T06:44:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.623+0000 7fab10563740 1 -- 172.21.15.22:0/2722061079 wait complete. 2024-03-24T06:44:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.624+0000 7fab10563740 1 -- start start 2024-03-24T06:44:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.624+0000 7fab06a00640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 0x7faaec0a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.624+0000 7fab10563740 1 Processor -- start 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f750740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f753780 0x55a96f7511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a96f656060 con 0x55a96f74aba0 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a96f655d00 con 0x55a96f753780 2024-03-24T06:44:51.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a96f41f1c0 con 0x55a96f74a280 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.625+0000 7fab10563740 10 client.0 _ll_register_callbacks cb 0x55a96f3c1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.626+0000 7fab06000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.626+0000 7fab06000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:47980/0 (socket says 172.21.15.22:47980) 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.626+0000 7fab06000640 1 -- 172.21.15.22:0/2788024256 learned_addr learned my addr 172.21.15.22:0/2788024256 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.626+0000 7fab06a00640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f750740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06000640 1 -- 172.21.15.22:0/2788024256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 msgr2=0x55a96f750740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06000640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f750740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06000640 1 -- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f753780 msgr2=0x55a96f7511c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06000640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f753780 0x55a96f7511c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06000640 1 -- 172.21.15.22:0/2788024256 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf801a040 con 0x55a96f74aba0 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06a00640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f750740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:44:51.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.627+0000 7fab06a00640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 0x7faaec0a4840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fab00005c40 tx=0x7fab0000a590 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.629+0000 7fab06000640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004700 tx=0x7faaf8013710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685605212d7f11ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.630+0000 7faaf0c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:44:51.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.630+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf800b480 con 0x55a96f74aba0 2024-03-24T06:44:51.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.630+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faaf8003e50 con 0x55a96f74aba0 2024-03-24T06:44:51.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.630+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a96f3b07e0 con 0x55a96f74aba0 2024-03-24T06:44:51.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.631+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faaf8013cc0 con 0x55a96f74aba0 2024-03-24T06:44:51.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.632+0000 7fab10563740 10 client.16155 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:51.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.632+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a96f6afa70 con 0x55a96f74aba0 2024-03-24T06:44:51.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.633+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a96f3b07e0 con 0x55a96f74aba0 2024-03-24T06:44:51.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7faaf8004a50 con 0x55a96f74aba0 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 10 client.16155 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 10 client.16155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 20 client.16155 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f763b80 0x55a96f765fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 4 client.16155 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:51.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.634+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a96f3ed720 con 0x55a96f763b80 2024-03-24T06:44:51.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.635+0000 7fab06a00640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f763b80 0x55a96f765fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.637+0000 7faaf0c00640 10 client.16155 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:44:51.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.637+0000 7fab06a00640 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f763b80 0x55a96f765fa0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a66e49908b88ed2d server_cookie=3190dce8f4ebcc7e in_seq=0 out_seq=0 2024-03-24T06:44:51.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.638+0000 7faaf0c00640 1 -- 172.21.15.22:0/2788024256 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a96f3ed720 con 0x55a96f763b80 2024-03-24T06:44:51.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.638+0000 7faaf0c00640 10 client.16155 handle_command_reply: tid=0 2024-03-24T06:44:51.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.638+0000 7fab10563740 1 client.16155 shutdown 2024-03-24T06:44:51.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.639+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 msgr2=0x55a96f750c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.639+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004700 tx=0x7faaf8013710 comp rx=0 tx=0).stop 2024-03-24T06:44:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 shutdown_connections 2024-03-24T06:44:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f763b80 0x55a96f765fa0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f753780 0x55a96f7511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f74aba0 0x55a96f750c80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7faaf0c00640 0 client.16155 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 --2- 172.21.15.22:0/2788024256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f750740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 >> 172.21.15.22:0/2788024256 conn(0x55a96f7449b0 msgr2=0x55a96f743a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 shutdown_connections 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 1 -- 172.21.15.22:0/2788024256 wait complete. 2024-03-24T06:44:51.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.640+0000 7fab10563740 20 client.16155 trim_cache size 0 max 16384 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.669+0000 7fab10563740 1 Processor -- start 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 -- start start 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74a280 0x55a96f752e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f7438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a96f7400e0 con 0x55a96f753780 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a96f749f40 con 0x55a96f74a280 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.670+0000 7fab10563740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a96f745700 con 0x55a96f74aba0 2024-03-24T06:44:51.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49392/0 (socket says 172.21.15.22:49392) 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 -- 172.21.15.22:0/3244064043 learned_addr learned my addr 172.21.15.22:0/3244064043 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab07400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f7438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06a00640 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74a280 0x55a96f752e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 -- 172.21.15.22:0/3244064043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74a280 msgr2=0x55a96f752e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74a280 0x55a96f752e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 -- 172.21.15.22:0/3244064043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 msgr2=0x55a96f7438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f7438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.671+0000 7fab06000640 1 -- 172.21.15.22:0/3244064043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a96f64bb00 con 0x55a96f74aba0 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.672+0000 7fab06000640 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004de0 tx=0x7faaf800b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b41a634f909951e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.672+0000 7faaf3400640 1 -- 172.21.15.22:0/3244064043 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8013780 con 0x55a96f74aba0 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.672+0000 7faaf3400640 1 -- 172.21.15.22:0/3244064043 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf800b480 con 0x55a96f74aba0 2024-03-24T06:44:51.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.673+0000 7fab10563740 1 -- 172.21.15.22:0/3244064043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 msgr2=0x55a96f75f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.673+0000 7faaf3400640 1 -- 172.21.15.22:0/3244064043 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8004730 con 0x55a96f74aba0 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaf8004de0 tx=0x7faaf800b890 comp rx=0 tx=0).stop 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 -- 172.21.15.22:0/3244064043 shutdown_connections 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f7438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74aba0 0x55a96f75f070 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 --2- 172.21.15.22:0/3244064043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74a280 0x55a96f752e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 -- 172.21.15.22:0/3244064043 >> 172.21.15.22:0/3244064043 conn(0x55a96f7449b0 msgr2=0x55a96f7431e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 -- 172.21.15.22:0/3244064043 shutdown_connections 2024-03-24T06:44:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.674+0000 7fab10563740 1 -- 172.21.15.22:0/3244064043 wait complete. 2024-03-24T06:44:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.675+0000 7fab10563740 1 -- start start 2024-03-24T06:44:51.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.675+0000 7fab10563740 1 Processor -- start 2024-03-24T06:44:51.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.676+0000 7fab10563740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f75b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.676+0000 7fab10563740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.676+0000 7fab10563740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f74d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.676+0000 7fab06000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49050/0 (socket says 172.21.15.22:49050) 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 -- 172.21.15.22:0/2358489587 learned_addr learned my addr 172.21.15.22:0/2358489587 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a96f74dc00 con 0x55a96f753780 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a96f74dd70 con 0x55a96f74aba0 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a96f7c8ec0 con 0x55a96f74a280 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab10563740 10 client.0 _ll_register_callbacks cb 0x7faaf800b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 -- 172.21.15.22:0/2358489587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 msgr2=0x55a96f75b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f75b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 -- 172.21.15.22:0/2358489587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 msgr2=0x55a96f74d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f74d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.677+0000 7fab06000640 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a96f6afa70 con 0x55a96f74aba0 2024-03-24T06:44:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.678+0000 7fab06000640 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faaf800be10 tx=0x7faaf800f530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e827a5eb656138d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.678+0000 7faaf0c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.678+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf800fa40 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.678+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaf8013940 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.678+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7faaf800fba0 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.679+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faaf800fd60 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.679+0000 7fab10563740 10 client.26474 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.679+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a96f6716a0 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf8013940 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7faaf8004810 con 0x55a96f74aba0 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 10 client.26474 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:51.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 10 client.26474 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:44:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 20 client.26474 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f7ca200 0x55a96f7cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 4 client.26474 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:44:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a96f43c890 con 0x55a96f7ca200 2024-03-24T06:44:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.680+0000 7fab06a00640 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f7ca200 0x55a96f7cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.682+0000 7fab06a00640 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f7ca200 0x55a96f7cc620 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=33d52e7b12691905 server_cookie=68f240114cd0ebce in_seq=0 out_seq=0 2024-03-24T06:44:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.682+0000 7faaf0c00640 10 client.26474 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:44:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.682+0000 7faaf0c00640 1 -- 172.21.15.22:0/2358489587 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a96f43c890 con 0x55a96f7ca200 2024-03-24T06:44:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.682+0000 7faaf0c00640 10 client.26474 handle_command_reply: tid=0 2024-03-24T06:44:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.682+0000 7fab10563740 1 client.26474 shutdown 2024-03-24T06:44:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 msgr2=0x55a96f74cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faaf800be10 tx=0x7faaf800f530 comp rx=0 tx=0).stop 2024-03-24T06:44:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 shutdown_connections 2024-03-24T06:44:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a96f753780 0x55a96f74d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a96f7ca200 0x55a96f7cc620 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a96f74aba0 0x55a96f74cff0 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faaf800be10 tx=0x7faaf800f530 comp rx=0 tx=0).stop 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 --2- 172.21.15.22:0/2358489587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a96f74a280 0x55a96f75b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 >> 172.21.15.22:0/2358489587 conn(0x55a96f7449b0 msgr2=0x55a96f743a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.683+0000 7faaf0c00640 0 client.26474 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.684+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 shutdown_connections 2024-03-24T06:44:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.684+0000 7fab10563740 1 -- 172.21.15.22:0/2358489587 wait complete. 2024-03-24T06:44:51.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.684+0000 7fab10563740 20 client.26474 trim_cache size 0 max 16384 2024-03-24T06:44:51.686 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:44:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.686+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 msgr2=0x7faaec0a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.686+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 0x7faaec0a4840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fab00005c40 tx=0x7fab0000a590 comp rx=0 tx=0).stop 2024-03-24T06:44:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.686+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 msgr2=0x7fab081958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:51.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.686+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab081958a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faaf403e040 tx=0x7faaf4011300 comp rx=0 tx=0).stop 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 shutdown_connections 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7faaec0a2380 0x7faaec0a4840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab081076b0 0x7fab081958a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab08102060 0x7fab0819bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 --2- 172.21.15.22:0/2657597112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab081016a0 0x7fab0819b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.687+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 >> 172.21.15.22:0/2657597112 conn(0x7fab08078b10 msgr2=0x7fab08103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.689+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 shutdown_connections 2024-03-24T06:44:51.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:51.690+0000 7fab0d800640 1 -- 172.21.15.22:0/2657597112 wait complete. 2024-03-24T06:44:51.720 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:44:52.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 -- 172.21.15.22:0/1191843223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481088d0 msgr2=0x7f404810c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 --2- 172.21.15.22:0/1191843223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481088d0 0x7f404810c910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4034009900 tx=0x7f403402bc60 comp rx=0 tx=0).stop 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 -- 172.21.15.22:0/1191843223 shutdown_connections 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 --2- 172.21.15.22:0/1191843223 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f404810cff0 0x7f4048114bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 --2- 172.21.15.22:0/1191843223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481088d0 0x7f404810c910 secure :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4034009900 tx=0x7f403402bc60 comp rx=0 tx=0).stop 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 --2- 172.21.15.22:0/1191843223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f4048108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.062+0000 7f404d400640 1 -- 172.21.15.22:0/1191843223 >> 172.21.15.22:0/1191843223 conn(0x7f4048075c50 msgr2=0x7f4048078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.063+0000 7f404d400640 1 -- 172.21.15.22:0/1191843223 shutdown_connections 2024-03-24T06:44:52.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.064+0000 7f404d400640 1 -- 172.21.15.22:0/1191843223 wait complete. 2024-03-24T06:44:52.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.065+0000 7f404d400640 1 Processor -- start 2024-03-24T06:44:52.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.066+0000 7f404d400640 1 -- start start 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f404819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f404810cff0 0x7f404819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 0x7f40481a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f40481a3800 con 0x7f404810cff0 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f40481a3970 con 0x7f4048107f20 2024-03-24T06:44:52.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f404d400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f40481a3ae0 con 0x7f40481a2e10 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f4047200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 0x7f40481a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f4047200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 0x7f40481a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49424/0 (socket says 172.21.15.22:49424) 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f4047200640 1 -- 172.21.15.22:0/1498970749 learned_addr learned my addr 172.21.15.22:0/1498970749 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.067+0000 7f4046800640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f404819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.068+0000 7f4046800640 1 -- 172.21.15.22:0/1498970749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 msgr2=0x7f40481a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.068+0000 7f4046800640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 0x7f40481a32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.068+0000 7f4046800640 1 -- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f404810cff0 msgr2=0x7f404819ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.068+0000 7f4046800640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f404810cff0 0x7f404819ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.068+0000 7f4046800640 1 -- 172.21.15.22:0/1498970749 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4034009560 con 0x7f4048107f20 2024-03-24T06:44:52.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.069+0000 7f4046800640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f404819e550 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f403801fa00 tx=0x7f403801fec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.070+0000 7f403f400640 1 -- 172.21.15.22:0/1498970749 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403800ccc0 con 0x7f4048107f20 2024-03-24T06:44:52.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.071+0000 7f403f400640 1 -- 172.21.15.22:0/1498970749 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f403800ce20 con 0x7f4048107f20 2024-03-24T06:44:52.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.071+0000 7f403f400640 1 -- 172.21.15.22:0/1498970749 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380298e0 con 0x7f4048107f20 2024-03-24T06:44:52.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.071+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4048079f60 con 0x7f4048107f20 2024-03-24T06:44:52.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.071+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404807a410 con 0x7f4048107f20 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.083+0000 7f40501d1740 1 Processor -- start 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.083+0000 7f403f400640 1 -- 172.21.15.22:0/1498970749 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f4038029a40 con 0x7f4048107f20 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.083+0000 7f40501d1740 1 -- start start 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9a8f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aabb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ada99fd6e0 con 0x55ada9aa9790 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ada99fac10 con 0x55ada9a8f890 2024-03-24T06:44:52.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.084+0000 7f40501d1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ada99f63f0 con 0x55ada9a8ef70 2024-03-24T06:44:52.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.085+0000 7f4046800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.085+0000 7f4046800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49072/0 (socket says 172.21.15.22:49072) 2024-03-24T06:44:52.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.085+0000 7f4046800640 1 -- 172.21.15.22:0/1128237040 learned_addr learned my addr 172.21.15.22:0/1128237040 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f4046800640 1 -- 172.21.15.22:0/1128237040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 msgr2=0x55ada9a8f350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f4046800640 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9a8f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f4046800640 1 -- 172.21.15.22:0/1128237040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 msgr2=0x55ada9aabb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f4046800640 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aabb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f4046800640 1 -- 172.21.15.22:0/1128237040 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada99b7e40 con 0x55ada9a8f890 2024-03-24T06:44:52.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.086+0000 7f403f400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 0x7f402c0a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.087+0000 7f403f400640 1 -- 172.21.15.22:0/1498970749 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4038025070 con 0x7f4048107f20 2024-03-24T06:44:52.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.088+0000 7f4046800640 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f40380dcc00 tx=0x7f40380e5ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88853c6a24700c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.089+0000 7f4047200640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 0x7f402c0a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.089+0000 7f4033400640 1 -- 172.21.15.22:0/1128237040 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380a8200 con 0x55ada9a8f890 2024-03-24T06:44:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.089+0000 7f4033400640 1 -- 172.21.15.22:0/1128237040 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40380abd30 con 0x55ada9a8f890 2024-03-24T06:44:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.090+0000 7f40501d1740 1 -- 172.21.15.22:0/1128237040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 msgr2=0x55ada9aa9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.090+0000 7f4033400640 1 -- 172.21.15.22:0/1128237040 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40380abf60 con 0x55ada9a8f890 2024-03-24T06:44:52.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.090+0000 7f40501d1740 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f40380dcc00 tx=0x7f40380e5ef0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.091+0000 7f40501d1740 1 -- 172.21.15.22:0/1128237040 shutdown_connections 2024-03-24T06:44:52.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.092+0000 7f4047200640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 0x7f402c0a4840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4040009e10 tx=0x7f40400094b0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aabb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa9250 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 --2- 172.21.15.22:0/1128237040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9a8f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 -- 172.21.15.22:0/1128237040 >> 172.21.15.22:0/1128237040 conn(0x55ada9a899b0 msgr2=0x55ada9a89d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 -- 172.21.15.22:0/1128237040 shutdown_connections 2024-03-24T06:44:52.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 -- 172.21.15.22:0/1128237040 wait complete. 2024-03-24T06:44:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.094+0000 7f40501d1740 1 -- start start 2024-03-24T06:44:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.095+0000 7f40501d1740 1 Processor -- start 2024-03-24T06:44:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.095+0000 7f40501d1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8ef70 0x55ada9aa66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.095+0000 7f40501d1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f40501d1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f4045e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f4045e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49076/0 (socket says 172.21.15.22:49076) 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f4045e00640 1 -- 172.21.15.22:0/3962419229 learned_addr learned my addr 172.21.15.22:0/3962419229 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ada999b060 con 0x55ada9a8ef70 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ada999ad00 con 0x55ada9a8f890 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ada97641c0 con 0x55ada9aa9790 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.096+0000 7f4047200640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.097+0000 7f4047200640 1 -- 172.21.15.22:0/3962419229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 msgr2=0x55ada9aa6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.097+0000 7f4047200640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.097+0000 7f4047200640 1 -- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8ef70 msgr2=0x55ada9aa66c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.097+0000 7f40501d1740 10 client.0 _ll_register_callbacks cb 0x55ada9706be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.098+0000 7f4047200640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8ef70 0x55ada9aa66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.098+0000 7f4047200640 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404807a410 con 0x55ada9aa9790 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.098+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ada96f57e0 con 0x55ada9aa9790 2024-03-24T06:44:52.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.098+0000 7f4047200640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa7140 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4040009a60 tx=0x7f4040004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a467ca16a410f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.098+0000 7f4027e00640 10 client.26512 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:44:52.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.099+0000 7f4027e00640 1 -- 172.21.15.22:0/3962419229 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4040018e10 con 0x55ada9aa9790 2024-03-24T06:44:52.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.099+0000 7f4027e00640 1 -- 172.21.15.22:0/3962419229 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f40400180d0 con 0x55ada9aa9790 2024-03-24T06:44:52.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.099+0000 7f4027e00640 1 -- 172.21.15.22:0/3962419229 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4040018e10 con 0x55ada9aa9790 2024-03-24T06:44:52.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.099+0000 7f40501d1740 10 client.26512 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:52.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.099+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({fsmap=34}) -- 0x55ada99f4a70 con 0x55ada9aa9790 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f4027e00640 1 -- 172.21.15.22:0/3962419229 <== mon.1 v2:172.21.15.155:3300/0 4 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f4040043420 con 0x55ada9aa9790 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 10 client.26512 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 10 client.26512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 20 client.26512 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9a91800 0x55ada9a93cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 4 client.26512 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:52.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.101+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ada9732720 con 0x55ada9a91800 2024-03-24T06:44:52.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.103+0000 7f4046800640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9a91800 0x55ada9a93cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.106+0000 7f4046800640 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9a91800 0x55ada9a93cc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6f81ef63b938d91 server_cookie=ed43ff106130f1c8 in_seq=0 out_seq=0 2024-03-24T06:44:52.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.106+0000 7f4027e00640 10 client.26512 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:52.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.109+0000 7f4027e00640 1 -- 172.21.15.22:0/3962419229 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ada9732720 con 0x55ada9a91800 2024-03-24T06:44:52.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.109+0000 7f4027e00640 10 client.26512 handle_command_reply: tid=0 2024-03-24T06:44:52.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.109+0000 7f40501d1740 1 client.26512 shutdown 2024-03-24T06:44:52.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.109+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 msgr2=0x55ada9aa7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.109+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa7140 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4040009a60 tx=0x7f4040004b50 comp rx=0 tx=0).stop 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.110+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 shutdown_connections 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa7140 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9a91800 0x55ada9a93cc0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aa6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f40501d1740 1 --2- 172.21.15.22:0/3962419229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8ef70 0x55ada9aa66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f4027e00640 0 client.26512 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:52.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.111+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 >> 172.21.15.22:0/3962419229 conn(0x55ada9a899b0 msgr2=0x55ada9a8b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.112+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 shutdown_connections 2024-03-24T06:44:52.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.112+0000 7f40501d1740 1 -- 172.21.15.22:0/3962419229 wait complete. 2024-03-24T06:44:52.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.113+0000 7f40501d1740 20 client.26512 trim_cache size 0 max 16384 2024-03-24T06:44:52.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.149+0000 7f40501d1740 1 Processor -- start 2024-03-24T06:44:52.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.149+0000 7f40501d1740 1 -- start start 2024-03-24T06:44:52.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.150+0000 7f40501d1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 0x55ada9a8b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.150+0000 7f40501d1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 0x55ada9ab5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f4046800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 0x55ada9a8b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f4046800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 0x55ada9a8b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49084/0 (socket says 172.21.15.22:49084) 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f40501d1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f40501d1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ada9aa7620 con 0x55ada9a8f890 2024-03-24T06:44:52.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f40501d1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ada9aa2280 con 0x55ada9a8ef70 2024-03-24T06:44:52.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ada9a891f0 con 0x55ada9aa9790 2024-03-24T06:44:52.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f4046800640 1 -- 172.21.15.22:0/2385113449 learned_addr learned my addr 172.21.15.22:0/2385113449 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:52.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.151+0000 7f4045e00640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 0x55ada9ab5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.152+0000 7f4047200640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4045e00640 1 -- 172.21.15.22:0/2385113449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 msgr2=0x55ada9aa5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:52.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4045e00640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4045e00640 1 -- 172.21.15.22:0/2385113449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 msgr2=0x55ada9a8b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4045e00640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 0x55ada9a8b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4045e00640 1 -- 172.21.15.22:0/2385113449 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada9990b00 con 0x55ada9a8f890 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.153+0000 7f4047200640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.154+0000 7f4045e00640 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 0x55ada9ab5100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4034009230 tx=0x7f4034004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16c37a269c3a3c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.154+0000 7f4032a00640 1 -- 172.21.15.22:0/2385113449 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034004230 con 0x55ada9a8f890 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.154+0000 7f4032a00640 1 -- 172.21.15.22:0/2385113449 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4034005360 con 0x55ada9a8f890 2024-03-24T06:44:52.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.154+0000 7f4032a00640 1 -- 172.21.15.22:0/2385113449 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403402c710 con 0x55ada9a8f890 2024-03-24T06:44:52.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.155+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 msgr2=0x55ada9ab5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.155+0000 7f40501d1740 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 0x55ada9ab5100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4034009230 tx=0x7f4034004c60 comp rx=0 tx=0).stop 2024-03-24T06:44:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 shutdown_connections 2024-03-24T06:44:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9aa9790 0x55ada9aa5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9a8f890 0x55ada9ab5100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 --2- 172.21.15.22:0/2385113449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8ef70 0x55ada9a8b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 >> 172.21.15.22:0/2385113449 conn(0x55ada9a899b0 msgr2=0x55ada9aa6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 shutdown_connections 2024-03-24T06:44:52.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.156+0000 7f40501d1740 1 -- 172.21.15.22:0/2385113449 wait complete. 2024-03-24T06:44:52.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.157+0000 7f40501d1740 1 -- start start 2024-03-24T06:44:52.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.157+0000 7f40501d1740 1 Processor -- start 2024-03-24T06:44:52.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.158+0000 7f40501d1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9ab1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.158+0000 7f40501d1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.158+0000 7f40501d1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aacc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.158+0000 7f4046800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9ab1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.158+0000 7f4045e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f40501d1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ada9ab1c80 con 0x55ada9aa9790 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f40501d1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ada9ab1df0 con 0x55ada9a8f890 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f40501d1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ada9ab1f60 con 0x55ada9a8ef70 2024-03-24T06:44:52.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f4045e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49100/0 (socket says 172.21.15.22:49100) 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f4045e00640 1 -- 172.21.15.22:0/1651999301 learned_addr learned my addr 172.21.15.22:0/1651999301 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f40501d1740 10 client.0 _ll_register_callbacks cb 0x7f4040009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f4047200640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aacc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.159+0000 7f4045e00640 1 -- 172.21.15.22:0/1651999301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 msgr2=0x55ada9ab1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.160+0000 7f4045e00640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9ab1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.160+0000 7f4045e00640 1 -- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 msgr2=0x55ada9aacc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:52.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.160+0000 7f4045e00640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aacc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.161+0000 7f4045e00640 1 -- 172.21.15.22:0/1651999301 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4034038040 con 0x55ada9a8f890 2024-03-24T06:44:52.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.161+0000 7f4046800640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9ab1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:44:52.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.161+0000 7f4047200640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aacc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.162+0000 7f4045e00640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f40340092a0 tx=0x7f4034004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b8010e47bb0d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.162+0000 7f4027e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.163+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034004800 con 0x55ada9a8f890 2024-03-24T06:44:52.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.163+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4034005090 con 0x55ada9a8f890 2024-03-24T06:44:52.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.163+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40380abd30 con 0x55ada9a8f890 2024-03-24T06:44:52.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.164+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f403402c600 con 0x55ada9a8f890 2024-03-24T06:44:52.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.164+0000 7f40501d1740 10 client.26492 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:52.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.164+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ada99b66a0 con 0x55ada9a8f890 2024-03-24T06:44:52.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.165+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40380abd30 con 0x55ada9a8f890 2024-03-24T06:44:52.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f4034048910 con 0x55ada9a8f890 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 10 client.26492 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 10 client.26492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 20 client.26492 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9b0f130 0x55ada9b11520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 4 client.26492 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:44:52.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.166+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ada9781890 con 0x55ada9b0f130 2024-03-24T06:44:52.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.170+0000 7f4046800640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9b0f130 0x55ada9b11520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:52.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.171+0000 7f4046800640 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9b0f130 0x55ada9b11520 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a44f1837fa246c server_cookie=1a07789358dcb17d in_seq=0 out_seq=0 2024-03-24T06:44:52.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.171+0000 7f4027e00640 10 client.26492 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:52.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.173+0000 7f4027e00640 1 -- 172.21.15.22:0/1651999301 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ada9781890 con 0x55ada9b0f130 2024-03-24T06:44:52.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.173+0000 7f4027e00640 10 client.26492 handle_command_reply: tid=0 2024-03-24T06:44:52.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.173+0000 7f40501d1740 1 client.26492 shutdown 2024-03-24T06:44:52.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 msgr2=0x55ada9aac6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f40340092a0 tx=0x7f4034004590 comp rx=0 tx=0).stop 2024-03-24T06:44:52.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 shutdown_connections 2024-03-24T06:44:52.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ada9aa9790 0x55ada9aacc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ada9b0f130 0x55ada9b11520 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ada9a8f890 0x55ada9aac6f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 --2- 172.21.15.22:0/1651999301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ada9a8ef70 0x55ada9ab1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f4027e00640 0 client.26492 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 >> 172.21.15.22:0/1651999301 conn(0x55ada9a899b0 msgr2=0x55ada9a881e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.174+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 shutdown_connections 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.175+0000 7f40501d1740 1 -- 172.21.15.22:0/1651999301 wait complete. 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.175+0000 7f40501d1740 20 client.26492 trim_cache size 0 max 16384 2024-03-24T06:44:52.175 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:44:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.176+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 msgr2=0x7f402c0a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.176+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 0x7f402c0a4840 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4040009e10 tx=0x7f40400094b0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.176+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 msgr2=0x7f404819e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:52.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.176+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f404819e550 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f403801fa00 tx=0x7f403801fec0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 shutdown_connections 2024-03-24T06:44:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f402c0a2380 0x7f402c0a4840 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481a2e10 0x7f40481a32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f404810cff0 0x7f404819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 --2- 172.21.15.22:0/1498970749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4048107f20 0x7f404819e550 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.177+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 >> 172.21.15.22:0/1498970749 conn(0x7f4048075c50 msgr2=0x7f4048076490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.178+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 shutdown_connections 2024-03-24T06:44:52.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:52.178+0000 7f404d400640 1 -- 172.21.15.22:0/1498970749 wait complete. 2024-03-24T06:44:52.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:52 smithi022 ceph-mon[32003]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 MiB/s wr, 4.19k op/s 2024-03-24T06:44:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:52 smithi143 ceph-mon[36520]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 MiB/s wr, 4.19k op/s 2024-03-24T06:44:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:52 smithi155 ceph-mon[36262]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 MiB/s wr, 4.19k op/s 2024-03-24T06:44:53.212 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:44:53.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:44:53.213 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:44:54.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:54 smithi022 ceph-mon[32003]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 5.73k op/s 2024-03-24T06:44:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:54 smithi143 ceph-mon[36520]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 5.73k op/s 2024-03-24T06:44:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:54 smithi155 ceph-mon[36262]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 5.73k op/s 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.911+0000 7ffb76200640 1 -- 172.21.15.22:0/1312508678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 msgr2=0x7ffb70100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.911+0000 7ffb76200640 1 --2- 172.21.15.22:0/1312508678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 0x7ffb70100370 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000aa10 tx=0x7ffb6002cf70 comp rx=0 tx=0).stop 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 -- 172.21.15.22:0/1312508678 shutdown_connections 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 --2- 172.21.15.22:0/1312508678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb7010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 --2- 172.21.15.22:0/1312508678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffb70100940 0x7ffb70104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 --2- 172.21.15.22:0/1312508678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 0x7ffb70100370 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 -- 172.21.15.22:0/1312508678 >> 172.21.15.22:0/1312508678 conn(0x7ffb70076b30 msgr2=0x7ffb700fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 -- 172.21.15.22:0/1312508678 shutdown_connections 2024-03-24T06:44:54.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.912+0000 7ffb76200640 1 -- 172.21.15.22:0/1312508678 wait complete. 2024-03-24T06:44:54.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.914+0000 7ffb76200640 1 Processor -- start 2024-03-24T06:44:54.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.914+0000 7ffb76200640 1 -- start start 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 0x7ffb7006db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffb70072ad0 0x7ffb70072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ffb701aedf0 con 0x7ffb70072ad0 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffb701aef60 con 0x7ffb70105060 2024-03-24T06:44:54.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb76200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffb701af120 con 0x7ffb700fff90 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59228/0 (socket says 172.21.15.22:59228) 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.915+0000 7ffb66a00640 1 -- 172.21.15.22:0/3653079990 learned_addr learned my addr 172.21.15.22:0/3653079990 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.916+0000 7ffb66a00640 1 -- 172.21.15.22:0/3653079990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 msgr2=0x7ffb7006db10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.916+0000 7ffb66a00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 0x7ffb7006db10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.916+0000 7ffb66a00640 1 -- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffb70072ad0 msgr2=0x7ffb70072f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.917+0000 7ffb66a00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffb70072ad0 0x7ffb70072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.917+0000 7ffb66a00640 1 -- 172.21.15.22:0/3653079990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6000a670 con 0x7ffb70105060 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.917+0000 7ffb66a00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800eb90 tx=0x7ffb5800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.918+0000 7ffb6cc00640 1 -- 172.21.15.22:0/3653079990 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb580099b0 con 0x7ffb70105060 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.918+0000 7ffb6cc00640 1 -- 172.21.15.22:0/3653079990 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb5800ee90 con 0x7ffb70105060 2024-03-24T06:44:54.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.918+0000 7ffb6cc00640 1 -- 172.21.15.22:0/3653079990 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58018b20 con 0x7ffb70105060 2024-03-24T06:44:54.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.918+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb701af400 con 0x7ffb70105060 2024-03-24T06:44:54.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.918+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb701af910 con 0x7ffb70105060 2024-03-24T06:44:54.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.926+0000 7ffb6cc00640 1 -- 172.21.15.22:0/3653079990 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7ffb58018c80 con 0x7ffb70105060 2024-03-24T06:44:54.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.928+0000 7ffb78f7d740 1 Processor -- start 2024-03-24T06:44:54.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.928+0000 7ffb78f7d740 1 -- start start 2024-03-24T06:44:54.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955c0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f19551d2d0 con 0x55f1955c0b40 2024-03-24T06:44:54.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f19551a800 con 0x55f1955af5f0 2024-03-24T06:44:54.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f195515fe0 con 0x55f1955aec40 2024-03-24T06:44:54.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.929+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.930+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52196/0 (socket says 172.21.15.22:52196) 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.930+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.930+0000 7ffb6fe00640 1 -- 172.21.15.22:0/1256715500 learned_addr learned my addr 172.21.15.22:0/1256715500 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.930+0000 7ffb6cc00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 0x7ffb500a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.931+0000 7ffb6cc00640 1 -- 172.21.15.22:0/3653079990 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffb58014070 con 0x7ffb70105060 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.931+0000 7ffb6fe00640 1 -- 172.21.15.22:0/1256715500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 msgr2=0x55f1955c0600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.931+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955c0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955c0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6fe00640 1 -- 172.21.15.22:0/1256715500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 msgr2=0x55f1955c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6fe00640 1 -- 172.21.15.22:0/1256715500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1954d7e90 con 0x55f1955aec40 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955c0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:44:54.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.932+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 0x7ffb500a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.933+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009d10 tx=0x7ffb68010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58af77676cfe2c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:54.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.935+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 0x7ffb500a4840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb68011010 tx=0x7ffb68019000 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:54.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.935+0000 7ffb5fe00640 1 -- 172.21.15.22:0/1256715500 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68009830 con 0x55f1955aec40 2024-03-24T06:44:54.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.935+0000 7ffb5fe00640 1 -- 172.21.15.22:0/1256715500 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb68012be0 con 0x55f1955aec40 2024-03-24T06:44:54.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.935+0000 7ffb5fe00640 1 -- 172.21.15.22:0/1256715500 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68009830 con 0x55f1955aec40 2024-03-24T06:44:54.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.936+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1256715500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 msgr2=0x55f1955af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:54.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.936+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009d10 tx=0x7ffb68010740 comp rx=0 tx=0).stop 2024-03-24T06:44:54.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.937+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1256715500 shutdown_connections 2024-03-24T06:44:54.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.937+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.937+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955c0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.937+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1256715500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955af020 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.937+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1256715500 >> 172.21.15.22:0/1256715500 conn(0x55f1955a9560 msgr2=0x55f1955a9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:54.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.938+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1256715500 shutdown_connections 2024-03-24T06:44:54.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.938+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1256715500 wait complete. 2024-03-24T06:44:54.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.939+0000 7ffb78f7d740 1 -- start start 2024-03-24T06:44:54.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.939+0000 7ffb78f7d740 1 Processor -- start 2024-03-24T06:44:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955af5f0 0x55f1955be120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955be660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f1954bafc0 con 0x55f1955c0b40 2024-03-24T06:44:54.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1954bac60 con 0x55f1955aec40 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f19525e0c0 con 0x55f1955af5f0 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.940+0000 7ffb78f7d740 10 client.0 _ll_register_callbacks cb 0x55f1954d2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.941+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.941+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59236/0 (socket says 172.21.15.22:59236) 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.941+0000 7ffb6f400640 1 -- 172.21.15.22:0/1517235362 learned_addr learned my addr 172.21.15.22:0/1517235362 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.941+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955af5f0 0x55f1955be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 -- 172.21.15.22:0/1517235362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955af5f0 msgr2=0x55f1955be120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955af5f0 0x55f1955be120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 -- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 msgr2=0x55f1955be660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 -- 172.21.15.22:0/1517235362 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68009990 con 0x55f1955aec40 2024-03-24T06:44:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.942+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb6002edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c412e8685e32a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.943+0000 7ffb5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:54.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.943+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60004250 con 0x55f1955aec40 2024-03-24T06:44:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.943+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffb600043b0 con 0x55f1955aec40 2024-03-24T06:44:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.943+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f19553ae90 con 0x55f1955aec40 2024-03-24T06:44:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.945+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ffb60004250 con 0x55f1955aec40 2024-03-24T06:44:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.945+0000 7ffb78f7d740 10 client.26501 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:54.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.945+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f195514660 con 0x55f1955aec40 2024-03-24T06:44:54.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.946+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb60004ba0 con 0x55f1955aec40 2024-03-24T06:44:54.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.946+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7ffb6003d070 con 0x55f1955aec40 2024-03-24T06:44:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.947+0000 7ffb78f7d740 10 client.26501 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.947+0000 7ffb78f7d740 10 client.26501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.947+0000 7ffb78f7d740 20 client.26501 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:54.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.947+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f1955b1370 0x55f1955b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:54.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.948+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f1955b1370 0x55f1955b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:54.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.950+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f1955b1370 0x55f1955b3800 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acdca3bfdf955401 server_cookie=1acf99b6fbab710d in_seq=0 out_seq=0 2024-03-24T06:44:54.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.950+0000 7ffb5cc00640 10 client.26501 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:54.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.951+0000 7ffb78f7d740 4 client.26501 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:54.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.951+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f19553ae90 con 0x55f1955b1370 2024-03-24T06:44:54.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.957+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1517235362 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f19553ae90 con 0x55f1955b1370 2024-03-24T06:44:54.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.957+0000 7ffb5cc00640 10 client.26501 handle_command_reply: tid=0 2024-03-24T06:44:54.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.957+0000 7ffb78f7d740 1 client.26501 shutdown 2024-03-24T06:44:54.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.958+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 msgr2=0x55f1955bdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:54.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.958+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb6002edd0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.959+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 shutdown_connections 2024-03-24T06:44:54.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.959+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.959+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955af5f0 0x55f1955be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.959+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f1955b1370 0x55f1955b3800 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.961+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1517235362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955bdbe0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.961+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 >> 172.21.15.22:0/1517235362 conn(0x55f1955a9560 msgr2=0x55f1955ab2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.961+0000 7ffb5cc00640 0 client.26501 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.961+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 shutdown_connections 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.961+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1517235362 wait complete. 2024-03-24T06:44:54.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:54.962+0000 7ffb78f7d740 20 client.26501 trim_cache size 0 max 16384 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.018+0000 7ffb78f7d740 1 Processor -- start 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.018+0000 7ffb78f7d740 1 -- start start 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.019+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.019+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955af5f0 0x55f1955a8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.019+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.019+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59252/0 (socket says 172.21.15.22:59252) 2024-03-24T06:44:55.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.019+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955c0b40 0x55f1955d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb66a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955af5f0 0x55f1955a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb6f400640 1 -- 172.21.15.22:0/559356398 learned_addr learned my addr 172.21.15.22:0/559356398 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f1955b15a0 con 0x55f1955af5f0 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb6fe00640 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955c0b40 0x55f1955d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1955bd9d0 con 0x55f1955aec40 2024-03-24T06:44:55.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.020+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f1955b97b0 con 0x55f1955c0b40 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.021+0000 7ffb6f400640 1 -- 172.21.15.22:0/559356398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955c0b40 msgr2=0x55f1955d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.021+0000 7ffb6f400640 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955c0b40 0x55f1955d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.021+0000 7ffb6f400640 1 -- 172.21.15.22:0/559356398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955af5f0 msgr2=0x55f1955a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.021+0000 7ffb6f400640 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955af5f0 0x55f1955a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.021+0000 7ffb6f400640 1 -- 172.21.15.22:0/559356398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1954b0ae0 con 0x55f1955aec40 2024-03-24T06:44:55.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.022+0000 7ffb6f400640 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb60000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c610dece62e72d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.023+0000 7ffb5f400640 1 -- 172.21.15.22:0/559356398 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60057070 con 0x55f1955aec40 2024-03-24T06:44:55.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.023+0000 7ffb5f400640 1 -- 172.21.15.22:0/559356398 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb60005d50 con 0x55f1955aec40 2024-03-24T06:44:55.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.023+0000 7ffb5f400640 1 -- 172.21.15.22:0/559356398 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60004250 con 0x55f1955aec40 2024-03-24T06:44:55.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.024+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 msgr2=0x55f1955c0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005ab0 tx=0x7ffb60000bc0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 shutdown_connections 2024-03-24T06:44:55.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955c0b40 0x55f1955d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955af5f0 0x55f1955a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/559356398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955aec40 0x55f1955c0310 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.025+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 >> 172.21.15.22:0/559356398 conn(0x55f1955a9560 msgr2=0x55f1955a7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.029+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 shutdown_connections 2024-03-24T06:44:55.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.029+0000 7ffb78f7d740 1 -- 172.21.15.22:0/559356398 wait complete. 2024-03-24T06:44:55.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.030+0000 7ffb78f7d740 1 -- start start 2024-03-24T06:44:55.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.030+0000 7ffb78f7d740 1 Processor -- start 2024-03-24T06:44:55.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.032+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.032+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.032+0000 7ffb78f7d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.032+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.032+0000 7ffb6f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52218/0 (socket says 172.21.15.22:52218) 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb6f400640 1 -- 172.21.15.22:0/1491810501 learned_addr learned my addr 172.21.15.22:0/1491810501 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f19562e0a0 con 0x55f1955c0b40 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f19562e210 con 0x55f1955af5f0 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f19562e3d0 con 0x55f1955aec40 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.033+0000 7ffb78f7d740 10 client.0 _ll_register_callbacks cb 0x7ffb580093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.034+0000 7ffb6f400640 1 -- 172.21.15.22:0/1491810501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 msgr2=0x55f1955d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.034+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.034+0000 7ffb6f400640 1 -- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 msgr2=0x55f1955d2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.034+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.034+0000 7ffb6f400640 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb60003e70 con 0x55f1955aec40 2024-03-24T06:44:55.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.035+0000 7ffb6f400640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000aa10 tx=0x7ffb600476f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57c73fb2d5326361 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.035+0000 7ffb5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:44:55.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.037+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60057070 con 0x55f1955aec40 2024-03-24T06:44:55.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.037+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb68012be0 con 0x55f1955aec40 2024-03-24T06:44:55.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.038+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7ffb60000e40 con 0x55f1955aec40 2024-03-24T06:44:55.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.038+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ffb60057070 con 0x55f1955aec40 2024-03-24T06:44:55.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.038+0000 7ffb78f7d740 10 client.26527 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:55.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.038+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55f1954d66f0 con 0x55f1955aec40 2024-03-24T06:44:55.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.039+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb68012be0 con 0x55f1955aec40 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.039+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7ffb60063020 con 0x55f1955aec40 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 10 client.26527 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 20 client.26527 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f19562f860 0x55f195631c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 4 client.26527 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f19550c210 con 0x55f19562f860 2024-03-24T06:44:55.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.040+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f19562f860 0x55f195631c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.041+0000 7ffb66a00640 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f19562f860 0x55f195631c90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4205c02806ff4de2 server_cookie=3baf42d5efdd9461 in_seq=0 out_seq=0 2024-03-24T06:44:55.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.041+0000 7ffb5cc00640 10 client.26527 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.043+0000 7ffb5cc00640 1 -- 172.21.15.22:0/1491810501 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f19550c210 con 0x55f19562f860 2024-03-24T06:44:55.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.043+0000 7ffb5cc00640 10 client.26527 handle_command_reply: tid=0 2024-03-24T06:44:55.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.043+0000 7ffb78f7d740 1 client.26527 shutdown 2024-03-24T06:44:55.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 msgr2=0x55f1955d24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000aa10 tx=0x7ffb600476f0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 shutdown_connections 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f1955c0b40 0x55f1955d2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55f19562f860 0x55f195631c90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1955af5f0 0x55f1955d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 --2- 172.21.15.22:0/1491810501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f1955aec40 0x55f1955d24f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb5cc00640 0 client.26527 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.044+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 >> 172.21.15.22:0/1491810501 conn(0x55f1955a9560 msgr2=0x55f1955a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.045+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 shutdown_connections 2024-03-24T06:44:55.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.045+0000 7ffb78f7d740 1 -- 172.21.15.22:0/1491810501 wait complete. 2024-03-24T06:44:55.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.045+0000 7ffb78f7d740 20 client.26527 trim_cache size 0 max 16384 2024-03-24T06:44:55.046 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:44:55.047 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:44:55.047 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "2bfb7f80-b67e-4b84-9742-1162332769f9", 2024-03-24T06:44:55.047 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:44:55.047 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 msgr2=0x7ffb500a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 0x7ffb500a4840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb68011010 tx=0x7ffb68019000 comp rx=0 tx=0).stop 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 msgr2=0x7ffb70072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800eb90 tx=0x7ffb5800c6a0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 shutdown_connections 2024-03-24T06:44:55.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffb500a2380 0x7ffb500a4840 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffb70072ad0 0x7ffb70072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb70105060 0x7ffb70072590 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 --2- 172.21.15.22:0/3653079990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb700fff90 0x7ffb7006db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 >> 172.21.15.22:0/3653079990 conn(0x7ffb70076b30 msgr2=0x7ffb701031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 shutdown_connections 2024-03-24T06:44:55.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.050+0000 7ffb76200640 1 -- 172.21.15.22:0/3653079990 wait complete. 2024-03-24T06:44:55.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:44:55.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 -- 172.21.15.22:0/428178900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581038b0 msgr2=0x7f7f58103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 --2- 172.21.15.22:0/428178900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581038b0 0x7f7f58103d30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009860 tx=0x7f7f4802bbb0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 -- 172.21.15.22:0/428178900 shutdown_connections 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 --2- 172.21.15.22:0/428178900 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f58110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 --2- 172.21.15.22:0/428178900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581038b0 0x7f7f58103d30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 --2- 172.21.15.22:0/428178900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581098c0 0x7f7f58109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.429+0000 7f7f60800640 1 -- 172.21.15.22:0/428178900 >> 172.21.15.22:0/428178900 conn(0x7f7f580ff360 msgr2=0x7f7f58101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.430+0000 7f7f60800640 1 -- 172.21.15.22:0/428178900 shutdown_connections 2024-03-24T06:44:55.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.430+0000 7f7f60800640 1 -- 172.21.15.22:0/428178900 wait complete. 2024-03-24T06:44:55.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.432+0000 7f7f60800640 1 Processor -- start 2024-03-24T06:44:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.433+0000 7f7f60800640 1 -- start start 2024-03-24T06:44:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.433+0000 7f7f60800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 0x7f7f58119c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.433+0000 7f7f60800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.433+0000 7f7f60800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581098c0 0x7f7f5810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.433+0000 7f7f5d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f5d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52236/0 (socket says 172.21.15.22:52236) 2024-03-24T06:44:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f5d400640 1 -- 172.21.15.22:0/387415230 learned_addr learned my addr 172.21.15.22:0/387415230 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581103e0 con 0x7f7f581038b0 2024-03-24T06:44:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58110550 con 0x7f7f581098c0 2024-03-24T06:44:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f58110710 con 0x7f7f58104270 2024-03-24T06:44:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f5de00640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 0x7f7f58119c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.434+0000 7f7f5e800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581098c0 0x7f7f5810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5d400640 1 -- 172.21.15.22:0/387415230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581098c0 msgr2=0x7f7f5810fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5d400640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581098c0 0x7f7f5810fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5d400640 1 -- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 msgr2=0x7f7f58119c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5d400640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 0x7f7f58119c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5d400640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f48009490 con 0x7f7f58104270 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.435+0000 7f7f5de00640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 0x7f7f58119c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:44:55.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.436+0000 7f7f5d400640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7f480057c0 tx=0x7f7f48034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.437+0000 7f7f52a00640 1 -- 172.21.15.22:0/387415230 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48034840 con 0x7f7f58104270 2024-03-24T06:44:55.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.438+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f58115650 con 0x7f7f58104270 2024-03-24T06:44:55.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.438+0000 7f7f52a00640 1 -- 172.21.15.22:0/387415230 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f48005500 con 0x7f7f58104270 2024-03-24T06:44:55.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.438+0000 7f7f52a00640 1 -- 172.21.15.22:0/387415230 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004920 con 0x7f7f58104270 2024-03-24T06:44:55.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.438+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f58115b50 con 0x7f7f58104270 2024-03-24T06:44:55.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.444+0000 7f7f63580740 1 Processor -- start 2024-03-24T06:44:55.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.445+0000 7f7f52a00640 1 -- 172.21.15.22:0/387415230 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7f4802c550 con 0x7f7f58104270 2024-03-24T06:44:55.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.445+0000 7f7f63580740 1 -- start start 2024-03-24T06:44:55.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.446+0000 7f7f63580740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.447+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54682/0 (socket says 172.21.15.22:54682) 2024-03-24T06:44:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f63580740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18953200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f63580740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e1895e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f63580740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e188c1790 con 0x559e18953a60 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f63580740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e188becc0 con 0x559e1895c0f0 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f63580740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e188ba4a0 con 0x559e18952d80 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f5e800640 1 -- 172.21.15.22:0/1343360850 learned_addr learned my addr 172.21.15.22:0/1343360850 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f52a00640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 0x7f7f400a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.449+0000 7f7f5de00640 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18953200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f5e800640 1 -- 172.21.15.22:0/1343360850 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 msgr2=0x559e18953200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f5e800640 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18953200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f5e800640 1 -- 172.21.15.22:0/1343360850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 msgr2=0x559e1895e4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:44:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f5e800640 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e1895e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f5e800640 1 -- 172.21.15.22:0/1343360850 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e1887bc20 con 0x559e18953a60 2024-03-24T06:44:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.450+0000 7f7f52a00640 1 -- 172.21.15.22:0/387415230 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7f480f0560 con 0x7f7f58104270 2024-03-24T06:44:55.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.451+0000 7f7f5e800640 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400f450 tx=0x7f7f5400f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a7a93b4212d537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.451+0000 7f7f5e800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 0x7f7f400a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.452+0000 7f7f47000640 1 -- 172.21.15.22:0/1343360850 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54003b70 con 0x559e18953a60 2024-03-24T06:44:55.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.452+0000 7f7f47000640 1 -- 172.21.15.22:0/1343360850 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f54003cd0 con 0x559e18953a60 2024-03-24T06:44:55.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.453+0000 7f7f63580740 1 -- 172.21.15.22:0/1343360850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 msgr2=0x559e189527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.453+0000 7f7f63580740 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400f450 tx=0x7f7f5400f910 comp rx=0 tx=0).stop 2024-03-24T06:44:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.455+0000 7f7f63580740 1 -- 172.21.15.22:0/1343360850 shutdown_connections 2024-03-24T06:44:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.456+0000 7f7f63580740 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e1895e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.456+0000 7f7f63580740 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18953200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.456+0000 7f7f63580740 1 --2- 172.21.15.22:0/1343360850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e189527b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.456+0000 7f7f63580740 1 -- 172.21.15.22:0/1343360850 >> 172.21.15.22:0/1343360850 conn(0x559e18919b30 msgr2=0x559e1894d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.457+0000 7f7f5e800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 0x7f7f400a4770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400f450 tx=0x7f7f5400f910 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.458+0000 7f7f63580740 1 -- 172.21.15.22:0/1343360850 shutdown_connections 2024-03-24T06:44:55.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.458+0000 7f7f63580740 1 -- 172.21.15.22:0/1343360850 wait complete. 2024-03-24T06:44:55.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.459+0000 7f7f63580740 1 -- start start 2024-03-24T06:44:55.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.459+0000 7f7f63580740 1 Processor -- start 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18959010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e18959550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18959a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e1885ef20 con 0x559e1895c0f0 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e1885ebc0 con 0x559e18953a60 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e186281c0 con 0x559e18952d80 2024-03-24T06:44:55.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.460+0000 7f7f63580740 10 client.0 _ll_register_callbacks cb 0x559e185cabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:55.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.462+0000 7f7f5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18959010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.462+0000 7f7f5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18959010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52248/0 (socket says 172.21.15.22:52248) 2024-03-24T06:44:55.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.462+0000 7f7f5de00640 1 -- 172.21.15.22:0/1190487349 learned_addr learned my addr 172.21.15.22:0/1190487349 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.462+0000 7f7f5d400640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e18959550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5e800640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18959a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5d400640 1 -- 172.21.15.22:0/1190487349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 msgr2=0x559e18959010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5d400640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18959010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5d400640 1 -- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 msgr2=0x559e18959a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5d400640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18959a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.463+0000 7f7f5d400640 1 -- 172.21.15.22:0/1190487349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f54009c40 con 0x559e18953a60 2024-03-24T06:44:55.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.464+0000 7f7f5d400640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e18959550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7f480eda60 tx=0x7f7f480bc900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ffd00323670574 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.464+0000 7f7f3b400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:55.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.465+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f480f03e0 con 0x559e18953a60 2024-03-24T06:44:55.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.465+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7f480bcd00 con 0x559e18953a60 2024-03-24T06:44:55.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.466+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559e188b2160 con 0x559e18953a60 2024-03-24T06:44:55.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.466+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7f480c0e50 con 0x559e18953a60 2024-03-24T06:44:55.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.466+0000 7f7f63580740 10 client.26516 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:55.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.466+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559e188b8b20 con 0x559e18953a60 2024-03-24T06:44:55.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.467+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559e188b2160 con 0x559e18953a60 2024-03-24T06:44:55.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.469+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f7f480c5010 con 0x559e18953a60 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 10 client.26516 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 20 client.26516 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e1896c3b0 0x559e1896e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 4 client.26516 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:44:55.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.470+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e185b97e0 con 0x559e1896c3b0 2024-03-24T06:44:55.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.471+0000 7f7f5de00640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e1896c3b0 0x559e1896e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.474+0000 7f7f3b400640 10 client.26516 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.474+0000 7f7f5de00640 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e1896c3b0 0x559e1896e7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce06cd0888e601c7 server_cookie=5ebaec5608f9fc46 in_seq=0 out_seq=0 2024-03-24T06:44:55.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.478+0000 7f7f3b400640 1 -- 172.21.15.22:0/1190487349 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e185b97e0 con 0x559e1896c3b0 2024-03-24T06:44:55.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.478+0000 7f7f3b400640 10 client.26516 handle_command_reply: tid=0 2024-03-24T06:44:55.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.478+0000 7f7f63580740 1 client.26516 shutdown 2024-03-24T06:44:55.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.479+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 msgr2=0x559e18959550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.479+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e18959550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7f480eda60 tx=0x7f7f480bc900 comp rx=0 tx=0).stop 2024-03-24T06:44:55.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 shutdown_connections 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18959a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e1896c3b0 0x559e1896e7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e18959550 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 --2- 172.21.15.22:0/1190487349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18959010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 >> 172.21.15.22:0/1190487349 conn(0x559e18919b30 msgr2=0x559e1894f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f3b400640 0 client.26516 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 shutdown_connections 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.480+0000 7f7f63580740 1 -- 172.21.15.22:0/1190487349 wait complete. 2024-03-24T06:44:55.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.481+0000 7f7f63580740 20 client.26516 trim_cache size 0 max 16384 2024-03-24T06:44:55.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.518+0000 7f7f63580740 1 Processor -- start 2024-03-24T06:44:55.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.518+0000 7f7f63580740 1 -- start start 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e1894d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18967b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e188b2160 con 0x559e1895c0f0 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e189525f0 con 0x559e18953a60 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f63580740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e1895bae0 con 0x559e18952d80 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f5de00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52260/0 (socket says 172.21.15.22:52260) 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.519+0000 7f7f5de00640 1 -- 172.21.15.22:0/808793956 learned_addr learned my addr 172.21.15.22:0/808793956 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.520+0000 7f7f5de00640 1 -- 172.21.15.22:0/808793956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 msgr2=0x559e1894d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.520+0000 7f7f5de00640 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e1894d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.520+0000 7f7f5de00640 1 -- 172.21.15.22:0/808793956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 msgr2=0x559e18967b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.520+0000 7f7f5de00640 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18967b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.520+0000 7f7f5de00640 1 -- 172.21.15.22:0/808793956 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e188549a0 con 0x559e18952d80 2024-03-24T06:44:55.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.521+0000 7f7f5de00640 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x559e1896f010 tx=0x7f7f4c013910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6f83550e70aa549 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.523+0000 7f7f46600640 1 -- 172.21.15.22:0/808793956 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017040 con 0x559e18952d80 2024-03-24T06:44:55.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.523+0000 7f7f46600640 1 -- 172.21.15.22:0/808793956 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c019040 con 0x559e18952d80 2024-03-24T06:44:55.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.523+0000 7f7f46600640 1 -- 172.21.15.22:0/808793956 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0039e0 con 0x559e18952d80 2024-03-24T06:44:55.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.524+0000 7f7f63580740 1 -- 172.21.15.22:0/808793956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 msgr2=0x559e18957f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.524+0000 7f7f63580740 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x559e1896f010 tx=0x7f7f4c013910 comp rx=0 tx=0).stop 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 -- 172.21.15.22:0/808793956 shutdown_connections 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e1895c0f0 0x559e18967b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e18953a60 0x559e1894d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 --2- 172.21.15.22:0/808793956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e18957f50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 -- 172.21.15.22:0/808793956 >> 172.21.15.22:0/808793956 conn(0x559e18919b30 msgr2=0x559e1894f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.525+0000 7f7f63580740 1 -- 172.21.15.22:0/808793956 shutdown_connections 2024-03-24T06:44:55.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.526+0000 7f7f63580740 1 -- 172.21.15.22:0/808793956 wait complete. 2024-03-24T06:44:55.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.527+0000 7f7f63580740 1 -- start start 2024-03-24T06:44:55.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.527+0000 7f7f63580740 1 Processor -- start 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e189561d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e18956710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e18956e10 con 0x559e18953a60 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e189d1c10 con 0x559e1895c0f0 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e189d1d50 con 0x559e18952d80 2024-03-24T06:44:55.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f63580740 10 client.0 _ll_register_callbacks cb 0x7f7f480b9070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f5e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59340/0 (socket says 172.21.15.22:59340) 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.528+0000 7f7f5e800640 1 -- 172.21.15.22:0/2055507374 learned_addr learned my addr 172.21.15.22:0/2055507374 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.529+0000 7f7f5e800640 1 -- 172.21.15.22:0/2055507374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 msgr2=0x559e189561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.529+0000 7f7f5e800640 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e189561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.529+0000 7f7f5e800640 1 -- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 msgr2=0x559e18956710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.529+0000 7f7f5e800640 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e18956710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.529+0000 7f7f5e800640 1 -- 172.21.15.22:0/2055507374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4c00aca0 con 0x559e1895c0f0 2024-03-24T06:44:55.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.533+0000 7f7f5e800640 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ba0 tx=0x7f7f5401a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e78c842dd64efbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:44:55.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.533+0000 7f7f3b400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:44:55.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.534+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54028040 con 0x559e1895c0f0 2024-03-24T06:44:55.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.534+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f7f54004e40 con 0x559e1895c0f0 2024-03-24T06:44:55.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.534+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559e1894d960 con 0x559e1895c0f0 2024-03-24T06:44:55.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.535+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7f5401adb0 con 0x559e1895c0f0 2024-03-24T06:44:55.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.535+0000 7f7f63580740 10 client.26519 fetch_fsmap learned FSMap version 34 2024-03-24T06:44:55.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.535+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559e1887a480 con 0x559e1895c0f0 2024-03-24T06:44:55.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.536+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559e1894d960 con 0x559e1895c0f0 2024-03-24T06:44:55.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.536+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7f7f5402e040 con 0x559e1895c0f0 2024-03-24T06:44:55.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 10 client.26519 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 10 client.26519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 20 client.26519 populate_metadata read hostname 'smithi022' 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e189d3560 0x559e189d59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 4 client.26519 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.537+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7f54003cd0 con 0x559e189d3560 2024-03-24T06:44:55.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.538+0000 7f7f5de00640 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e189d3560 0x559e189d59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:44:55.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.540+0000 7f7f5de00640 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e189d3560 0x559e189d59a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2eac5ce7b19319e server_cookie=ce156d6a8932842a in_seq=0 out_seq=0 2024-03-24T06:44:55.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.540+0000 7f7f3b400640 10 client.26519 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.544+0000 7f7f3b400640 1 -- 172.21.15.22:0/2055507374 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7f7f54003cd0 con 0x559e189d3560 2024-03-24T06:44:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.544+0000 7f7f3b400640 10 client.26519 handle_command_reply: tid=0 2024-03-24T06:44:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.545+0000 7f7f63580740 1 client.26519 shutdown 2024-03-24T06:44:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.545+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 msgr2=0x559e189d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.545+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009ba0 tx=0x7f7f5401a430 comp rx=0 tx=0).stop 2024-03-24T06:44:55.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 shutdown_connections 2024-03-24T06:44:55.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e189d3560 0x559e189d59a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e1895c0f0 0x559e189d16d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e18953a60 0x559e18956710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 --2- 172.21.15.22:0/2055507374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e18952d80 0x559e189561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 >> 172.21.15.22:0/2055507374 conn(0x559e18919b30 msgr2=0x559e1895a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.546+0000 7f7f3b400640 0 client.26519 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:44:55.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.547+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 shutdown_connections 2024-03-24T06:44:55.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.547+0000 7f7f63580740 1 -- 172.21.15.22:0/2055507374 wait complete. 2024-03-24T06:44:55.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.547+0000 7f7f63580740 20 client.26519 trim_cache size 0 max 16384 2024-03-24T06:44:55.548 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "scrub active (23 inodes in the stack)", 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "2bfb7f80-b67e-4b84-9742-1162332769f9": { 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "2bfb7f80-b67e-4b84-9742-1162332769f9", 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:44:55.549 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 msgr2=0x7f7f400a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 0x7f7f400a4770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400f450 tx=0x7f7f5400f910 comp rx=0 tx=0).stop 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 msgr2=0x7f7f5811a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7f480057c0 tx=0x7f7f48034440 comp rx=0 tx=0).stop 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 shutdown_connections 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f400a22b0 0x7f7f400a4770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f581098c0 0x7f7f5810fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f58104270 0x7f7f5811a180 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 --2- 172.21.15.22:0/387415230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f581038b0 0x7f7f58119c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:44:55.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.549+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 >> 172.21.15.22:0/387415230 conn(0x7f7f580ff360 msgr2=0x7f7f58078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:44:55.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.550+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 shutdown_connections 2024-03-24T06:44:55.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:44:55.550+0000 7f7f60800640 1 -- 172.21.15.22:0/387415230 wait complete. 2024-03-24T06:44:55.586 INFO:tasks.cephfs.filesystem:scrub status for tag:2bfb7f80-b67e-4b84-9742-1162332769f9 - {'path': '/', 'tag': '2bfb7f80-b67e-4b84-9742-1162332769f9', 'options': 'recursive,force'} 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 MiB/s wr, 4.83k op/s 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:44:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:56 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 MiB/s wr, 4.83k op/s 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:44:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:44:56.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:56 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:44:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 MiB/s wr, 4.83k op/s 2024-03-24T06:44:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:44:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:44:56.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:44:56.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:44:56.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:44:56.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:44:56.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:56 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:44:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:44:58 smithi022 ceph-mon[32003]: pgmap v152: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 34 MiB/s wr, 6.01k op/s 2024-03-24T06:44:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:44:58 smithi143 ceph-mon[36520]: pgmap v152: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 34 MiB/s wr, 6.01k op/s 2024-03-24T06:44:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:44:58 smithi155 ceph-mon[36262]: pgmap v152: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 34 MiB/s wr, 6.01k op/s 2024-03-24T06:45:00.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:00 smithi022 ceph-mon[32003]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 MiB/s wr, 4.96k op/s 2024-03-24T06:45:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:00 smithi143 ceph-mon[36520]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 MiB/s wr, 4.96k op/s 2024-03-24T06:45:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:00 smithi155 ceph-mon[36262]: pgmap v153: 129 pgs: 129 active+clean; 17 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 28 MiB/s wr, 4.96k op/s 2024-03-24T06:45:01.587 INFO:tasks.workunit.client.0.smithi022.stdout:Wrote -1 instead of 4096 bytes. 2024-03-24T06:45:01.587 INFO:tasks.workunit.client.0.smithi022.stdout:Probably out of disk space 2024-03-24T06:45:01.587 INFO:tasks.workunit.client.0.smithi022.stderr:write: Resource temporarily unavailable 2024-03-24T06:45:02.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:02 smithi022 ceph-mon[32003]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 30 MiB/s wr, 5.37k op/s 2024-03-24T06:45:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:02 smithi143 ceph-mon[36520]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 30 MiB/s wr, 5.37k op/s 2024-03-24T06:45:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:02 smithi155 ceph-mon[36262]: pgmap v154: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 30 MiB/s wr, 5.37k op/s 2024-03-24T06:45:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T06:45:03.054 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-24T06:45:03.055 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T06:45:03.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:03 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:03.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:03 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:03 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:03.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:03 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:03 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:03 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:04 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:04 smithi022 ceph-mon[32003]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 6.13k op/s 2024-03-24T06:45:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:04 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:04 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:04 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:04 smithi143 ceph-mon[36520]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 6.13k op/s 2024-03-24T06:45:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:04 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:04 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:04 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:04 smithi155 ceph-mon[36262]: pgmap v155: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 6.13k op/s 2024-03-24T06:45:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:04 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:04 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:06.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:06 smithi022 ceph-mon[32003]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 23 MiB/s wr, 4.36k op/s 2024-03-24T06:45:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:06 smithi143 ceph-mon[36520]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 23 MiB/s wr, 4.36k op/s 2024-03-24T06:45:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:06 smithi155 ceph-mon[36262]: pgmap v156: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 23 MiB/s wr, 4.36k op/s 2024-03-24T06:45:08.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:08 smithi022 ceph-mon[32003]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 4.81k op/s 2024-03-24T06:45:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:08 smithi143 ceph-mon[36520]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 4.81k op/s 2024-03-24T06:45:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:08 smithi155 ceph-mon[36262]: pgmap v157: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 25 MiB/s wr, 4.81k op/s 2024-03-24T06:45:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:10 smithi022 ceph-mon[32003]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 3.25k op/s 2024-03-24T06:45:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:10 smithi143 ceph-mon[36520]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 3.25k op/s 2024-03-24T06:45:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:10 smithi155 ceph-mon[36262]: pgmap v158: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 16 MiB/s wr, 3.25k op/s 2024-03-24T06:45:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:12 smithi022 ceph-mon[32003]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 17 MiB/s wr, 3.48k op/s 2024-03-24T06:45:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:12 smithi143 ceph-mon[36520]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 17 MiB/s wr, 3.48k op/s 2024-03-24T06:45:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:12 smithi155 ceph-mon[36262]: pgmap v159: 129 pgs: 129 active+clean; 17 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 17 MiB/s wr, 3.48k op/s 2024-03-24T06:45:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:13.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:13.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:13.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:13.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:13.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:14.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:14 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:14.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:14 smithi155 ceph-mon[36262]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 4.52k op/s 2024-03-24T06:45:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:14 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:14 smithi143 ceph-mon[36520]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 4.52k op/s 2024-03-24T06:45:14.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:14 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:14.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:14 smithi022 ceph-mon[32003]: pgmap v160: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 21 MiB/s wr, 4.52k op/s 2024-03-24T06:45:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:16 smithi143 ceph-mon[36520]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 3.99k op/s 2024-03-24T06:45:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:16 smithi155 ceph-mon[36262]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 3.99k op/s 2024-03-24T06:45:16.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:16 smithi022 ceph-mon[32003]: pgmap v161: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 3.99k op/s 2024-03-24T06:45:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:18 smithi143 ceph-mon[36520]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 MiB/s wr, 4.93k op/s 2024-03-24T06:45:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:18 smithi155 ceph-mon[36262]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 MiB/s wr, 4.93k op/s 2024-03-24T06:45:18.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:18 smithi022 ceph-mon[32003]: pgmap v162: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 MiB/s wr, 4.93k op/s 2024-03-24T06:45:20.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:20 smithi143 ceph-mon[36520]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.29k op/s 2024-03-24T06:45:20.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:20 smithi155 ceph-mon[36262]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.29k op/s 2024-03-24T06:45:20.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:20 smithi022 ceph-mon[32003]: pgmap v163: 129 pgs: 129 active+clean; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.29k op/s 2024-03-24T06:45:21.838 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:21 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:21.838 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:21 smithi155 ceph-mon[36262]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.40k op/s 2024-03-24T06:45:21.838 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:21 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:45:21.898 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:21 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:21 smithi022 ceph-mon[32003]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.40k op/s 2024-03-24T06:45:21.899 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:21 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:45:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:21 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:21 smithi143 ceph-mon[36520]: pgmap v164: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.40k op/s 2024-03-24T06:45:21.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:21 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:45:24.309 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:23 smithi155 ceph-mon[36262]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 MiB/s wr, 5.26k op/s 2024-03-24T06:45:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:23 smithi022 ceph-mon[32003]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 MiB/s wr, 5.26k op/s 2024-03-24T06:45:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:23 smithi143 ceph-mon[36520]: pgmap v165: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 MiB/s wr, 5.26k op/s 2024-03-24T06:45:24.589 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-24T06:45:24.818 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7e59b48e9ef042e19595b23984ec83cd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi022 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-24T06:45:24.821 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-24T06:45:24.830 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-24T06:45:24.830 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T06:45:24.831 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:45:24.831 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-24T06:45:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:24 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:45:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:24 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:45:25.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:24 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:45:25.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:24 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:45:25.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:24 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:45:25.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:24 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:45:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:24 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:45:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:24 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:45:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:24 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:45:25.462 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-24T06:45:25.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:45:25.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:45:25.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.946+0000 7f1ff2600640 1 -- 172.21.15.22:0/1877376883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec109330 msgr2=0x7f1fec10c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.946+0000 7f1ff2600640 1 --2- 172.21.15.22:0/1877376883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec109330 0x7f1fec10c940 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009860 tx=0x7f1fd802bb70 comp rx=0 tx=0).stop 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.947+0000 7f1ff2600640 1 -- 172.21.15.22:0/1877376883 shutdown_connections 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.947+0000 7f1ff2600640 1 --2- 172.21.15.22:0/1877376883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec079a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.947+0000 7f1ff2600640 1 --2- 172.21.15.22:0/1877376883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec109330 0x7f1fec10c940 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.947+0000 7f1ff2600640 1 --2- 172.21.15.22:0/1877376883 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1fec108980 0x7f1fec108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.947+0000 7f1ff2600640 1 -- 172.21.15.22:0/1877376883 >> 172.21.15.22:0/1877376883 conn(0x7f1fec06c7f0 msgr2=0x7f1fec06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:25.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.949+0000 7f1ff2600640 1 -- 172.21.15.22:0/1877376883 shutdown_connections 2024-03-24T06:45:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.949+0000 7f1ff2600640 1 -- 172.21.15.22:0/1877376883 wait complete. 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.950+0000 7f1ff2600640 1 Processor -- start 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.950+0000 7f1ff2600640 1 -- start start 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec108980 0x7f1fec084300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1fec082bf0 0x7f1fec083080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b5870 con 0x7f1fec082bf0 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b59e0 con 0x7f1fec108980 2024-03-24T06:45:25.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1ff2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec1b5ba0 con 0x7f1fec10d050 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1feaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1feaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57654/0 (socket says 172.21.15.22:57654) 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.951+0000 7f1feaa00640 1 -- 172.21.15.22:0/106325262 learned_addr learned my addr 172.21.15.22:0/106325262 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.952+0000 7f1feaa00640 1 -- 172.21.15.22:0/106325262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec108980 msgr2=0x7f1fec084300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.952+0000 7f1feaa00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec108980 0x7f1fec084300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.952+0000 7f1feaa00640 1 -- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1fec082bf0 msgr2=0x7f1fec083080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.952+0000 7f1feaa00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1fec082bf0 0x7f1fec083080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.952+0000 7f1feaa00640 1 -- 172.21.15.22:0/106325262 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe4009840 con 0x7f1fec10d050 2024-03-24T06:45:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.953+0000 7f1feaa00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8037040 tx=0x7f1fd8004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.958+0000 7f1fe3e00640 1 -- 172.21.15.22:0/106325262 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80348d0 con 0x7f1fec10d050 2024-03-24T06:45:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.958+0000 7f1fe3e00640 1 -- 172.21.15.22:0/106325262 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd8034a30 con 0x7f1fec10d050 2024-03-24T06:45:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.958+0000 7f1fe3e00640 1 -- 172.21.15.22:0/106325262 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8032ed0 con 0x7f1fec10d050 2024-03-24T06:45:25.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.958+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd8009490 con 0x7f1fec10d050 2024-03-24T06:45:25.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.959+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec1b6000 con 0x7f1fec10d050 2024-03-24T06:45:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.968+0000 7f1fe3e00640 1 -- 172.21.15.22:0/106325262 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1fd803b040 con 0x7f1fec10d050 2024-03-24T06:45:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.968+0000 7f1ff5233740 1 Processor -- start 2024-03-24T06:45:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 -- start start 2024-03-24T06:45:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a4a60 0x55eb918a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918ad0f0 0x55eb918af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55eb91812790 con 0x55eb918ad0f0 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eb9180fcc0 con 0x55eb918a4a60 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.969+0000 7f1ff5233740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb9180b4a0 con 0x55eb918a3d80 2024-03-24T06:45:25.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.970+0000 7f1febe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a4a60 0x55eb918a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.970+0000 7f1feb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.970+0000 7f1feb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57660/0 (socket says 172.21.15.22:57660) 2024-03-24T06:45:25.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.970+0000 7f1feb400640 1 -- 172.21.15.22:0/34552575 learned_addr learned my addr 172.21.15.22:0/34552575 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:25.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feaa00640 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918ad0f0 0x55eb918af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feb400640 1 -- 172.21.15.22:0/34552575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a4a60 msgr2=0x55eb918a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:25.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feb400640 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a4a60 0x55eb918a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feb400640 1 -- 172.21.15.22:0/34552575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918ad0f0 msgr2=0x55eb918af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:25.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feb400640 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918ad0f0 0x55eb918af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.971+0000 7f1feb400640 1 -- 172.21.15.22:0/34552575 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb917ccc20 con 0x55eb918a3d80 2024-03-24T06:45:25.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.972+0000 7f1feb400640 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400e350 tx=0x7f1fe400e750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd6933da15fbca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:25.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.972+0000 7f1fe0c00640 1 -- 172.21.15.22:0/34552575 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4009b80 con 0x55eb918a3d80 2024-03-24T06:45:25.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.973+0000 7f1fe0c00640 1 -- 172.21.15.22:0/34552575 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe4003b70 con 0x55eb918a3d80 2024-03-24T06:45:25.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.973+0000 7f1fe0c00640 1 -- 172.21.15.22:0/34552575 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4010830 con 0x55eb918a3d80 2024-03-24T06:45:25.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.974+0000 7f1ff5233740 1 -- 172.21.15.22:0/34552575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 msgr2=0x55eb918a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:25.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.974+0000 7f1ff5233740 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400e350 tx=0x7f1fe400e750 comp rx=0 tx=0).stop 2024-03-24T06:45:25.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.975+0000 7f1fe3e00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 0x7f1fd00a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.975+0000 7f1fe3e00640 1 -- 172.21.15.22:0/106325262 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fd80efad0 con 0x7f1fec10d050 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1febe00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 0x7f1fd00a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 -- 172.21.15.22:0/34552575 shutdown_connections 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918ad0f0 0x55eb918af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918a4200 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 --2- 172.21.15.22:0/34552575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a4a60 0x55eb918a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 -- 172.21.15.22:0/34552575 >> 172.21.15.22:0/34552575 conn(0x55eb9186ab30 msgr2=0x55eb9189e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:25.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.976+0000 7f1ff5233740 1 -- 172.21.15.22:0/34552575 shutdown_connections 2024-03-24T06:45:25.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.977+0000 7f1ff5233740 1 -- 172.21.15.22:0/34552575 wait complete. 2024-03-24T06:45:25.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.977+0000 7f1febe00640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 0x7f1fd00a4900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc005f30 tx=0x7f1fdc00e040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:25.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.977+0000 7f1ff5233740 1 -- start start 2024-03-24T06:45:25.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.978+0000 7f1ff5233740 1 Processor -- start 2024-03-24T06:45:25.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.979+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.979+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.979+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 0x55eb918aab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.980+0000 7f1feb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.980+0000 7f1feb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918aa0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57664/0 (socket says 172.21.15.22:57664) 2024-03-24T06:45:25.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.980+0000 7f1feaa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.980+0000 7f1feaa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:57758/0 (socket says 172.21.15.22:57758) 2024-03-24T06:45:25.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1feb400640 1 -- 172.21.15.22:0/2152321283 learned_addr learned my addr 172.21.15.22:0/2152321283 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:25.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55eb917aff20 con 0x55eb918a4a60 2024-03-24T06:45:25.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eb917afbc0 con 0x55eb918ad0f0 2024-03-24T06:45:25.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb915791c0 con 0x55eb918a3d80 2024-03-24T06:45:25.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1febe00640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 0x55eb918aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1ff5233740 10 client.0 _ll_register_callbacks cb 0x55eb9151bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:25.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1febe00640 1 -- 172.21.15.22:0/2152321283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 msgr2=0x55eb918aa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:25.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1febe00640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918aa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1febe00640 1 -- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 msgr2=0x55eb918aa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:25.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.983+0000 7f1febe00640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:25.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1febe00640 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe4009ce0 con 0x55eb918ad0f0 2024-03-24T06:45:25.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1feaa00640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1fd5600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1febe00640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 0x55eb918aab40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc018670 tx=0x7f1fdc00ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3d49c7a67a2170 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc004230 con 0x55eb918ad0f0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.984+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb91803160 con 0x55eb918ad0f0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.985+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fdc004390 con 0x55eb918ad0f0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.985+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1fdc004230 con 0x55eb918ad0f0 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.985+0000 7f1ff5233740 10 client.26528 fetch_fsmap learned FSMap version 35 2024-03-24T06:45:25.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.985+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eb91809b20 con 0x55eb918ad0f0 2024-03-24T06:45:25.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.985+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eb91803160 con 0x55eb918ad0f0 2024-03-24T06:45:25.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1fdc0035f0 con 0x55eb918ad0f0 2024-03-24T06:45:25.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 10 client.26528 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T06:45:25.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:25.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 20 client.26528 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:25.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb918bd690 0x55eb918bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:25.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 4 client.26528 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:25.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.987+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb9150a7e0 con 0x55eb918bd690 2024-03-24T06:45:25.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.991+0000 7f1feb400640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb918bd690 0x55eb918bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:25.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.993+0000 7f1feb400640 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb918bd690 0x55eb918bfad0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cda3e1c2cde96ecb server_cookie=fc1cf87d338be185 in_seq=0 out_seq=0 2024-03-24T06:45:25.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.993+0000 7f1fd5600640 10 client.26528 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:25.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.998+0000 7f1fd5600640 1 -- 172.21.15.22:0/2152321283 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eb9150a7e0 con 0x55eb918bd690 2024-03-24T06:45:25.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.998+0000 7f1fd5600640 10 client.26528 handle_command_reply: tid=0 2024-03-24T06:45:25.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:25.998+0000 7f1ff5233740 1 client.26528 shutdown 2024-03-24T06:45:25.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 msgr2=0x55eb918aab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:25.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 0x55eb918aab40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc018670 tx=0x7f1fdc00ac00 comp rx=0 tx=0).stop 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 shutdown_connections 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb918bd690 0x55eb918bfad0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918ad0f0 0x55eb918aab40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918aa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1fd5600640 0 client.26528 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:26.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2152321283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918a3d80 0x55eb918aa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.000+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 >> 172.21.15.22:0/2152321283 conn(0x55eb9186ab30 msgr2=0x55eb918a0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.001+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 shutdown_connections 2024-03-24T06:45:26.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.001+0000 7f1ff5233740 1 -- 172.21.15.22:0/2152321283 wait complete. 2024-03-24T06:45:26.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.002+0000 7f1ff5233740 20 client.26528 trim_cache size 0 max 16384 2024-03-24T06:45:26.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 Processor -- start 2024-03-24T06:45:26.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 -- start start 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55eb918a3000 con 0x55eb918a4a60 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eb918a2dd0 con 0x55eb918a3d80 2024-03-24T06:45:26.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.041+0000 7f1ff5233740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb918a9eb0 con 0x55eb918ad0f0 2024-03-24T06:45:26.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.042+0000 7f1febe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.042+0000 7f1febe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57674/0 (socket says 172.21.15.22:57674) 2024-03-24T06:45:26.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.042+0000 7f1febe00640 1 -- 172.21.15.22:0/2618222893 learned_addr learned my addr 172.21.15.22:0/2618222893 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:26.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.042+0000 7f1feb400640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1feaa00640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1febe00640 1 -- 172.21.15.22:0/2618222893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 msgr2=0x55eb918ac7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:26.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1febe00640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918ac7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1febe00640 1 -- 172.21.15.22:0/2618222893 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 msgr2=0x55eb918a9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:26.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1febe00640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1febe00640 1 -- 172.21.15.22:0/2618222893 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb917a59a0 con 0x55eb918ad0f0 2024-03-24T06:45:26.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1feb400640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:26.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.043+0000 7f1feaa00640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:26.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.044+0000 7f1febe00640 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc018f00 tx=0x7f1fdc016510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b26ddd3dcc04ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:26.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.045+0000 7f1fd7e00640 1 -- 172.21.15.22:0/2618222893 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc004230 con 0x55eb918ad0f0 2024-03-24T06:45:26.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.045+0000 7f1fd7e00640 1 -- 172.21.15.22:0/2618222893 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eb91803160 con 0x55eb918ad0f0 2024-03-24T06:45:26.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.045+0000 7f1fd7e00640 1 -- 172.21.15.22:0/2618222893 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0202f0 con 0x55eb918ad0f0 2024-03-24T06:45:26.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.046+0000 7f1ff5233740 1 -- 172.21.15.22:0/2618222893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 msgr2=0x55eb9189e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.046+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc018f00 tx=0x7f1fdc016510 comp rx=0 tx=0).stop 2024-03-24T06:45:26.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 -- 172.21.15.22:0/2618222893 shutdown_connections 2024-03-24T06:45:26.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb9189e300 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 --2- 172.21.15.22:0/2618222893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918ac7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 -- 172.21.15.22:0/2618222893 >> 172.21.15.22:0/2618222893 conn(0x55eb9186ab30 msgr2=0x55eb918aa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 -- 172.21.15.22:0/2618222893 shutdown_connections 2024-03-24T06:45:26.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 -- 172.21.15.22:0/2618222893 wait complete. 2024-03-24T06:45:26.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.051+0000 7f1ff5233740 1 -- start start 2024-03-24T06:45:26.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.052+0000 7f1ff5233740 1 Processor -- start 2024-03-24T06:45:26.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.055+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.056+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.056+0000 7f1ff5233740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb918a6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1ff5233740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55eb918b57f0 con 0x55eb918a4a60 2024-03-24T06:45:26.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1ff5233740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eb918a6dd0 con 0x55eb918a3d80 2024-03-24T06:45:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1ff5233740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb918a6f90 con 0x55eb918ad0f0 2024-03-24T06:45:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1ff5233740 10 client.0 _ll_register_callbacks cb 0x7f1fdc00a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1feb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1feb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51850/0 (socket says 172.21.15.22:51850) 2024-03-24T06:45:26.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1feb400640 1 -- 172.21.15.22:0/664854873 learned_addr learned my addr 172.21.15.22:0/664854873 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:26.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.057+0000 7f1febe00640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb918a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feaa00640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feb400640 1 -- 172.21.15.22:0/664854873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 msgr2=0x55eb918a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:26.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feb400640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb918a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feb400640 1 -- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 msgr2=0x55eb918a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feb400640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.058+0000 7f1feb400640 1 -- 172.21.15.22:0/664854873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe401c040 con 0x55eb918a3d80 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.059+0000 7f1feaa00640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.059+0000 7f1feb400640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400ec50 tx=0x7f1fe4009ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2569d87f7a627d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:26.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.059+0000 7f1fd5600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:26.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe402e070 con 0x55eb918a3d80 2024-03-24T06:45:26.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1fe4014070 con 0x55eb918a3d80 2024-03-24T06:45:26.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb9189e960 con 0x55eb918a3d80 2024-03-24T06:45:26.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1fe4003970 con 0x55eb918a3d80 2024-03-24T06:45:26.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1ff5233740 10 client.26534 fetch_fsmap learned FSMap version 35 2024-03-24T06:45:26.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.063+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eb91809b20 con 0x55eb918a3d80 2024-03-24T06:45:26.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.064+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eb9189e960 con 0x55eb918a3d80 2024-03-24T06:45:26.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1fe4031020 con 0x55eb918a3d80 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 10 client.26534 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 10 client.26534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 20 client.26534 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb91924c70 0x55eb91927060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 4 client.26534 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:45:26.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.065+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f1fe4003b70 con 0x55eb91924c70 2024-03-24T06:45:26.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.066+0000 7f1feaa00640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb91924c70 0x55eb91927060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.068+0000 7f1feaa00640 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb91924c70 0x55eb91927060 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=855fa0307c9f73df server_cookie=5fc02812020e75aa in_seq=0 out_seq=0 2024-03-24T06:45:26.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.068+0000 7f1fd5600640 10 client.26534 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:26.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.068+0000 7f1fd5600640 1 -- 172.21.15.22:0/664854873 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f1fe4003b70 con 0x55eb91924c70 2024-03-24T06:45:26.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.068+0000 7f1fd5600640 10 client.26534 handle_command_reply: tid=0 2024-03-24T06:45:26.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 client.26534 shutdown 2024-03-24T06:45:26.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 msgr2=0x55eb918b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400ec50 tx=0x7f1fe4009ab0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 shutdown_connections 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb918ad0f0 0x55eb918a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55eb918a4a60 0x55eb918a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55eb91924c70 0x55eb91927060 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 --2- 172.21.15.22:0/664854873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eb918a3d80 0x55eb918b50a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1fd5600640 0 client.26534 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:26.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.069+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 >> 172.21.15.22:0/664854873 conn(0x55eb9186ab30 msgr2=0x55eb918a9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.073+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 shutdown_connections 2024-03-24T06:45:26.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.073+0000 7f1ff5233740 1 -- 172.21.15.22:0/664854873 wait complete. 2024-03-24T06:45:26.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.075+0000 7f1ff5233740 20 client.26534 trim_cache size 0 max 16384 2024-03-24T06:45:26.074 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:45:26.074 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:45:26.075 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:45:26.075 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:45:26.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 msgr2=0x7f1fd00a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 0x7f1fd00a4900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc005f30 tx=0x7f1fdc00e040 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 msgr2=0x7f1fec0826b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8037040 tx=0x7f1fd8004580 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 shutdown_connections 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1fd00a2440 0x7f1fd00a4900 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1fec082bf0 0x7f1fec083080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1fec10d050 0x7f1fec0826b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 --2- 172.21.15.22:0/106325262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fec108980 0x7f1fec084300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 >> 172.21.15.22:0/106325262 conn(0x7f1fec06c7f0 msgr2=0x7f1fec071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 shutdown_connections 2024-03-24T06:45:26.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.076+0000 7f1ff2600640 1 -- 172.21.15.22:0/106325262 wait complete. 2024-03-24T06:45:26.100 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:25 smithi022 ceph-mon[32003]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-24T06:45:26.101 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:25 smithi022 ceph-mon[32003]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.49k op/s 2024-03-24T06:45:26.101 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:25 smithi022 ceph-mon[32003]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-24T06:45:26.101 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:25 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:45:26.102 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:45:26.102 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:45:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:25 smithi155 ceph-mon[36262]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-24T06:45:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:25 smithi155 ceph-mon[36262]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.49k op/s 2024-03-24T06:45:26.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:25 smithi155 ceph-mon[36262]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-24T06:45:26.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:25 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:45:26.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.421+0000 7fe4e7400640 1 -- 172.21.15.22:0/989733030 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0032ab0 con 0x7fe4e8112540 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 -- 172.21.15.22:0/989733030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8112540 msgr2=0x7fe4e8114930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 --2- 172.21.15.22:0/989733030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8112540 0x7fe4e8114930 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009cd0 tx=0x7fe4e002c0e0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 -- 172.21.15.22:0/989733030 shutdown_connections 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 --2- 172.21.15.22:0/989733030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8112540 0x7fe4e8114930 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 --2- 172.21.15.22:0/989733030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e8112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 --2- 172.21.15.22:0/989733030 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8075c40 0x7fe4e8076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 -- 172.21.15.22:0/989733030 >> 172.21.15.22:0/989733030 conn(0x7fe4e80ff370 msgr2=0x7fe4e8101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 -- 172.21.15.22:0/989733030 shutdown_connections 2024-03-24T06:45:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.422+0000 7fe4ef800640 1 -- 172.21.15.22:0/989733030 wait complete. 2024-03-24T06:45:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.424+0000 7fe4ef800640 1 Processor -- start 2024-03-24T06:45:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.424+0000 7fe4ef800640 1 -- start start 2024-03-24T06:45:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8075c40 0x7fe4e819e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8112540 0x7fe4e81a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e81a3200 con 0x7fe4e8112540 2024-03-24T06:45:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e81a3370 con 0x7fe4e8076560 2024-03-24T06:45:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ef800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e81a3530 con 0x7fe4e8075c40 2024-03-24T06:45:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4e7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4e7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51880/0 (socket says 172.21.15.22:51880) 2024-03-24T06:45:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4e7e00640 1 -- 172.21.15.22:0/3620950898 learned_addr learned my addr 172.21.15.22:0/3620950898 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.425+0000 7fe4ece00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8075c40 0x7fe4e819e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4ed800640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8112540 0x7fe4e81a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 -- 172.21.15.22:0/3620950898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8075c40 msgr2=0x7fe4e819e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8075c40 0x7fe4e819e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 -- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8112540 msgr2=0x7fe4e81a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8112540 0x7fe4e81a2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 -- 172.21.15.22:0/3620950898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e0009900 con 0x7fe4e8076560 2024-03-24T06:45:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.426+0000 7fe4e7e00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400d950 tx=0x7fe4d400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.427+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4009c30 con 0x7fe4e8076560 2024-03-24T06:45:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.427+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4009d90 con 0x7fe4e8076560 2024-03-24T06:45:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.427+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400f7e0 con 0x7fe4e8076560 2024-03-24T06:45:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.427+0000 7fe4ef800640 1 -- 172.21.15.22:0/3620950898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e81a3810 con 0x7fe4e8076560 2024-03-24T06:45:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.427+0000 7fe4ef800640 1 -- 172.21.15.22:0/3620950898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e81aaeb0 con 0x7fe4e8076560 2024-03-24T06:45:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.429+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4005290 con 0x7fe4e8076560 2024-03-24T06:45:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:25 smithi143 ceph-mon[36520]: pgmap v166: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 4.27k op/s 2024-03-24T06:45:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:25 smithi143 ceph-mon[36520]: pgmap v167: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 4.49k op/s 2024-03-24T06:45:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:25 smithi143 ceph-mon[36520]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-24T06:45:26.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:25 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:45:26.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.436+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe4d4010ad0 con 0x7fe4e8076560 2024-03-24T06:45:26.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.439+0000 7fe4e5600640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 0x7fe4d00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:26.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.439+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe4d40cdfb0 con 0x7fe4e8076560 2024-03-24T06:45:26.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.439+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe4d40ce490 con 0x7fe4e8076560 2024-03-24T06:45:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.440+0000 7fe4ece00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 0x7fe4d00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:26.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.442+0000 7fe4ece00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 0x7fe4d00a4770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009b60 tx=0x7fe4dc005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:26.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.830+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4c40058f0 con 0x7fe4e8076560 2024-03-24T06:45:26.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.831+0000 7fe4e5600640 1 -- 172.21.15.22:0/3620950898 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11941 (secure 0 0 0) 0x7fe4c40058f0 con 0x7fe4e8076560 2024-03-24T06:45:26.831 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:45:26.831 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:45:20.185423+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24544,24565,24532,24601,24584]},"id":1}]} 2024-03-24T06:45:26.831 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 35 2024-03-24T06:45:26.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 msgr2=0x7fe4d00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 0x7fe4d00a4770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009b60 tx=0x7fe4dc005f90 comp rx=0 tx=0).stop 2024-03-24T06:45:26.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 msgr2=0x7fe4e819e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:26.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400d950 tx=0x7fe4d400de10 comp rx=0 tx=0).stop 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 shutdown_connections 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4d00a22b0 0x7fe4d00a4770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4e8112540 0x7fe4e81a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4e8076560 0x7fe4e819e880 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 --2- 172.21.15.22:0/3620950898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4e8075c40 0x7fe4e819e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 >> 172.21.15.22:0/3620950898 conn(0x7fe4e80ff370 msgr2=0x7fe4e8114080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:26.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 shutdown_connections 2024-03-24T06:45:26.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:26.834+0000 7fe4daa00640 1 -- 172.21.15.22:0/3620950898 wait complete. 2024-03-24T06:45:26.849 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:45:27.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:26 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3620950898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:45:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:26 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3620950898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:45:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:26 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3620950898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:45:28.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 -- 172.21.15.22:0/559456023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100320 msgr2=0x7f66a0100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 --2- 172.21.15.22:0/559456023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100320 0x7f66a0100700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6690009900 tx=0x7f669002bc30 comp rx=0 tx=0).stop 2024-03-24T06:45:28.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 -- 172.21.15.22:0/559456023 shutdown_connections 2024-03-24T06:45:28.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 --2- 172.21.15.22:0/559456023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a01102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 --2- 172.21.15.22:0/559456023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100cd0 0x7f66a010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 --2- 172.21.15.22:0/559456023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100320 0x7f66a0100700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.092+0000 7f66a5a00640 1 -- 172.21.15.22:0/559456023 >> 172.21.15.22:0/559456023 conn(0x7f66a00fbf60 msgr2=0x7f66a00fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.093+0000 7f66a5a00640 1 -- 172.21.15.22:0/559456023 shutdown_connections 2024-03-24T06:45:28.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.093+0000 7f66a5a00640 1 -- 172.21.15.22:0/559456023 wait complete. 2024-03-24T06:45:28.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.094+0000 7f66a5a00640 1 Processor -- start 2024-03-24T06:45:28.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.095+0000 7f66a5a00640 1 -- start start 2024-03-24T06:45:28.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.095+0000 7f66a5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 0x7f66a0102600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f66a5a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100cd0 0x7f66a0102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f66a5a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a0103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f66a5a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01aa400 con 0x7f66a0100cd0 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f66a5a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f66a0199ba0 con 0x7f66a010ded0 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f66a5a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f66a0199d60 con 0x7f66a0100320 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.096+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 0x7f66a0102600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a0103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 0x7f66a0102600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57718/0 (socket says 172.21.15.22:57718) 2024-03-24T06:45:28.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f669fe00640 1 -- 172.21.15.22:0/3725541196 learned_addr learned my addr 172.21.15.22:0/3725541196 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f669f400640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100cd0 0x7f66a0102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 -- 172.21.15.22:0/3725541196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 msgr2=0x7f66a0102600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 0x7f66a0102600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 -- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100cd0 msgr2=0x7f66a0102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100cd0 0x7f66a0102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.097+0000 7f66a5000640 1 -- 172.21.15.22:0/3725541196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6698009840 con 0x7f66a010ded0 2024-03-24T06:45:28.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f66a5000640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a0103080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f669800ec50 tx=0x7f669800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f669cc00640 1 -- 172.21.15.22:0/3725541196 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698003b70 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f669cc00640 1 -- 172.21.15.22:0/3725541196 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f669800ce70 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f669cc00640 1 -- 172.21.15.22:0/3725541196 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698010810 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6690009560 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.098+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a019a3c0 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.103+0000 7f66a9d6c740 1 Processor -- start 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.104+0000 7f669cc00640 1 -- 172.21.15.22:0/3725541196 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f6698010970 con 0x7f66a010ded0 2024-03-24T06:45:28.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.104+0000 7f66a9d6c740 1 -- start start 2024-03-24T06:45:28.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8a4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8afd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560aaf8126e0 con 0x560aaf8ad950 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560aaf80fc10 con 0x560aaf8a4450 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560aaf80b3f0 con 0x560aaf8a4d70 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57730/0 (socket says 172.21.15.22:57730) 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.105+0000 7f669fe00640 1 -- 172.21.15.22:0/516877038 learned_addr learned my addr 172.21.15.22:0/516877038 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.106 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.106+0000 7f669fe00640 1 -- 172.21.15.22:0/516877038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 msgr2=0x560aaf8a4830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:45:28.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.106+0000 7f669f400640 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669cc00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 0x7f66840a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669fe00640 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669fe00640 1 -- 172.21.15.22:0/516877038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 msgr2=0x560aaf8afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669fe00640 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669fe00640 1 -- 172.21.15.22:0/516877038 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aaf7cce40 con 0x560aaf8a4d70 2024-03-24T06:45:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.107+0000 7f669cc00640 1 -- 172.21.15.22:0/3725541196 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6698014070 con 0x7f66a010ded0 2024-03-24T06:45:28.108 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.108+0000 7f669f400640 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:28.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.109+0000 7f669fe00640 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f669002c770 tx=0x7f66900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4492cf16f294804 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.110+0000 7f66a5000640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 0x7f66840a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.110+0000 7f6695600640 1 -- 172.21.15.22:0/516877038 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66900047e0 con 0x560aaf8a4d70 2024-03-24T06:45:28.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.110+0000 7f6695600640 1 -- 172.21.15.22:0/516877038 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6690004940 con 0x560aaf8a4d70 2024-03-24T06:45:28.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.110+0000 7f6695600640 1 -- 172.21.15.22:0/516877038 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690005120 con 0x560aaf8a4d70 2024-03-24T06:45:28.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 -- 172.21.15.22:0/516877038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 msgr2=0x560aaf8ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f669002c770 tx=0x7f66900046c0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a5000640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 0x7f66840a47b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6698097b40 tx=0x7f669800ef50 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 -- 172.21.15.22:0/516877038 shutdown_connections 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8ad410 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/516877038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 -- 172.21.15.22:0/516877038 >> 172.21.15.22:0/516877038 conn(0x560aaf89eb80 msgr2=0x560aaf89ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.114+0000 7f66a9d6c740 1 -- 172.21.15.22:0/516877038 shutdown_connections 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.115+0000 7f66a9d6c740 1 -- 172.21.15.22:0/516877038 wait complete. 2024-03-24T06:45:28.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.115+0000 7f66a9d6c740 1 -- start start 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 Processor -- start 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8ab480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560aaf7b0060 con 0x560aaf8ad950 2024-03-24T06:45:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560aaf7afd00 con 0x560aaf8a4450 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.116+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560aaf5791c0 con 0x560aaf8a4d70 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f66a9d6c740 10 client.0 _ll_register_callbacks cb 0x560aaf51bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57740/0 (socket says 172.21.15.22:57740) 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 -- 172.21.15.22:0/118493472 learned_addr learned my addr 172.21.15.22:0/118493472 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669f400640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669f400640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 -- 172.21.15.22:0/118493472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 msgr2=0x560aaf8aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8aaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 -- 172.21.15.22:0/118493472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 msgr2=0x560aaf8ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669fe00640 1 -- 172.21.15.22:0/118493472 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a0110ea0 con 0x560aaf8a4d70 2024-03-24T06:45:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.117+0000 7f669f400640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.118+0000 7f669fe00640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6690036040 tx=0x7f6690005090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b351df02bacbc4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.118+0000 7f668a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.119+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66900047e0 con 0x560aaf8a4d70 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.119+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6690005590 con 0x560aaf8a4d70 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.119+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560aaf50a7e0 con 0x560aaf8a4d70 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.120+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f669004a450 con 0x560aaf8a4d70 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.120+0000 7f66a9d6c740 10 client.26578 fetch_fsmap learned FSMap version 35 2024-03-24T06:45:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.120+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560aaf809a70 con 0x560aaf8a4d70 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560aaf50a7e0 con 0x560aaf8a4d70 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f6690052030 con 0x560aaf8a4d70 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 10 client.26578 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 10 client.26578 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 20 client.26578 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf8bda00 0x560aaf8bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 4 client.26578 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:28.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.121+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560aaf547720 con 0x560aaf8bda00 2024-03-24T06:45:28.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.122+0000 7f669f400640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf8bda00 0x560aaf8bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.124+0000 7f669f400640 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf8bda00 0x560aaf8bfe20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=704a71710d935917 server_cookie=fe5fa25777bb8ada in_seq=0 out_seq=0 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.124+0000 7f668a000640 10 client.26578 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f668a000640 1 -- 172.21.15.22:0/118493472 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560aaf547720 con 0x560aaf8bda00 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f668a000640 10 client.26578 handle_command_reply: tid=0 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 client.26578 shutdown 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 msgr2=0x560aaf8aaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6690036040 tx=0x7f6690005090 comp rx=0 tx=0).stop 2024-03-24T06:45:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 shutdown_connections 2024-03-24T06:45:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf8bda00 0x560aaf8bfe20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8ad950 0x560aaf8ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8aaf40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/118493472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4450 0x560aaf8aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 >> 172.21.15.22:0/118493472 conn(0x560aaf89eb80 msgr2=0x560aaf8a08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.125+0000 7f668a000640 0 client.26578 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.127+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 shutdown_connections 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.127+0000 7f66a9d6c740 1 -- 172.21.15.22:0/118493472 wait complete. 2024-03-24T06:45:28.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.127+0000 7f66a9d6c740 20 client.26578 trim_cache size 0 max 16384 2024-03-24T06:45:28.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.158+0000 7f66a9d6c740 1 Processor -- start 2024-03-24T06:45:28.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.158+0000 7f66a9d6c740 1 -- start start 2024-03-24T06:45:28.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 0x560aaf8ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560aaf8b96c0 con 0x560aaf8a4450 2024-03-24T06:45:28.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560aaf89a090 con 0x560aaf8a4d70 2024-03-24T06:45:28.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.159+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560aaf8c0480 con 0x560aaf8ad950 2024-03-24T06:45:28.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.160+0000 7f669fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.160+0000 7f669fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51926/0 (socket says 172.21.15.22:51926) 2024-03-24T06:45:28.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.160+0000 7f669fe00640 1 -- 172.21.15.22:0/3387349455 learned_addr learned my addr 172.21.15.22:0/3387349455 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.160+0000 7f669f400640 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 0x560aaf8ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 -- 172.21.15.22:0/3387349455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 msgr2=0x560aaf8ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 0x560aaf8ab2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 -- 172.21.15.22:0/3387349455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 msgr2=0x560aaf8b9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 -- 172.21.15.22:0/3387349455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aaf7a5b00 con 0x560aaf8a4d70 2024-03-24T06:45:28.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669f400640 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 0x560aaf8ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.161+0000 7f669fe00640 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6690006010 tx=0x7f6690004df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354fb696337a84fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.162+0000 7f6694c00640 1 -- 172.21.15.22:0/3387349455 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669002caf0 con 0x560aaf8a4d70 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.162+0000 7f6694c00640 1 -- 172.21.15.22:0/3387349455 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f669004e070 con 0x560aaf8a4d70 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.162+0000 7f6694c00640 1 -- 172.21.15.22:0/3387349455 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669003fb40 con 0x560aaf8a4d70 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.163+0000 7f66a9d6c740 1 -- 172.21.15.22:0/3387349455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 msgr2=0x560aaf899b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.163+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6690006010 tx=0x7f6690004df0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 -- 172.21.15.22:0/3387349455 shutdown_connections 2024-03-24T06:45:28.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8ad950 0x560aaf8ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8a4d70 0x560aaf899b50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/3387349455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 -- 172.21.15.22:0/3387349455 >> 172.21.15.22:0/3387349455 conn(0x560aaf89eb80 msgr2=0x560aaf89dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 -- 172.21.15.22:0/3387349455 shutdown_connections 2024-03-24T06:45:28.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 -- 172.21.15.22:0/3387349455 wait complete. 2024-03-24T06:45:28.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.165+0000 7f66a9d6c740 1 -- start start 2024-03-24T06:45:28.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.166+0000 7f66a9d6c740 1 Processor -- start 2024-03-24T06:45:28.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8ad950 0x560aaf8b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560aaf922d20 con 0x560aaf8a4450 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560aaf922e60 con 0x560aaf8ad950 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560aaf922fa0 con 0x560aaf8a4d70 2024-03-24T06:45:28.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.167+0000 7f66a9d6c740 10 client.0 _ll_register_callbacks cb 0x7f6690009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:28.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.168+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.168+0000 7f669f400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.168+0000 7f669fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57766/0 (socket says 172.21.15.22:57766) 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.168+0000 7f669fe00640 1 -- 172.21.15.22:0/1340840644 learned_addr learned my addr 172.21.15.22:0/1340840644 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.168+0000 7f669f400640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8ad950 0x560aaf8b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669fe00640 1 -- 172.21.15.22:0/1340840644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8ad950 msgr2=0x560aaf8b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669fe00640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8ad950 0x560aaf8b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669fe00640 1 -- 172.21.15.22:0/1340840644 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 msgr2=0x560aaf8b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669fe00640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669fe00640 1 -- 172.21.15.22:0/1340840644 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aaf809a70 con 0x560aaf8a4d70 2024-03-24T06:45:28.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.169+0000 7f669f400640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.170+0000 7f669fe00640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f669002bc70 tx=0x7f66900045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acfcf423720f7b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.170+0000 7f668a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.170+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669003fcf0 con 0x560aaf8a4d70 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.171+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f669004a450 con 0x560aaf8a4d70 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.171+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6690004940 con 0x560aaf8a4d70 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.171+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f669003fe50 con 0x560aaf8a4d70 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.171+0000 7f66a9d6c740 10 client.26584 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:28.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.171+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560aaf7cb6a0 con 0x560aaf8a4d70 2024-03-24T06:45:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6690004940 con 0x560aaf8a4d70 2024-03-24T06:45:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f6690072070 con 0x560aaf8a4d70 2024-03-24T06:45:28.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 10 client.26584 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 10 client.26584 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 20 client.26584 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf924320 0x560aaf926710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 4 client.26584 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560aaf596890 con 0x560aaf924320 2024-03-24T06:45:28.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.172+0000 7f669f400640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf924320 0x560aaf926710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f668a000640 10 client.26584 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:45:28.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f669f400640 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf924320 0x560aaf926710 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82104974ee4e2a21 server_cookie=50d6dad35016d528 in_seq=0 out_seq=0 2024-03-24T06:45:28.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f668a000640 1 -- 172.21.15.22:0/1340840644 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560aaf596890 con 0x560aaf924320 2024-03-24T06:45:28.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f668a000640 10 client.26584 handle_command_reply: tid=0 2024-03-24T06:45:28.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f66a9d6c740 1 client.26584 shutdown 2024-03-24T06:45:28.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 msgr2=0x560aaf8b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.174+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f669002bc70 tx=0x7f66900045e0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 shutdown_connections 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x560aaf924320 0x560aaf926710 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560aaf8ad950 0x560aaf8b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560aaf8a4d70 0x560aaf8b1600 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 --2- 172.21.15.22:0/1340840644 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560aaf8a4450 0x560aaf8b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 >> 172.21.15.22:0/1340840644 conn(0x560aaf89eb80 msgr2=0x560aaf89fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f668a000640 0 client.26584 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:45:28.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 shutdown_connections 2024-03-24T06:45:28.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 1 -- 172.21.15.22:0/1340840644 wait complete. 2024-03-24T06:45:28.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.176+0000 7f66a9d6c740 20 client.26584 trim_cache size 0 max 16384 2024-03-24T06:45:28.177 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:45:28.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 msgr2=0x7f66840a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 0x7f66840a47b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6698097b40 tx=0x7f669800ef50 comp rx=0 tx=0).stop 2024-03-24T06:45:28.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 msgr2=0x7f66a0103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a0103080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f669800ec50 tx=0x7f669800c6a0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 shutdown_connections 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f66840a22f0 0x7f66840a47b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a010ded0 0x7f66a0103080 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66a0100cd0 0x7f66a0102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 --2- 172.21.15.22:0/3725541196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a0100320 0x7f66a0102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 >> 172.21.15.22:0/3725541196 conn(0x7f66a00fbf60 msgr2=0x7f66a00fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.177+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 shutdown_connections 2024-03-24T06:45:28.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.178+0000 7f66a5a00640 1 -- 172.21.15.22:0/3725541196 wait complete. 2024-03-24T06:45:28.201 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:45:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:28 smithi022 ceph-mon[32003]: pgmap v168: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.42k op/s 2024-03-24T06:45:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:28 smithi143 ceph-mon[36520]: pgmap v168: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.42k op/s 2024-03-24T06:45:28.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 -- 172.21.15.22:0/679951659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228011aef0 msgr2=0x7f228011d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 --2- 172.21.15.22:0/679951659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228011aef0 0x7f228011d2e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f226c009900 tx=0x7f226c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:45:28.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 -- 172.21.15.22:0/679951659 shutdown_connections 2024-03-24T06:45:28.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 --2- 172.21.15.22:0/679951659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228011aef0 0x7f228011d2e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 --2- 172.21.15.22:0/679951659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228010df60 0x7f228010e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 --2- 172.21.15.22:0/679951659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 0x7f228010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.503+0000 7f2285a00640 1 -- 172.21.15.22:0/679951659 >> 172.21.15.22:0/679951659 conn(0x7f228006c7f0 msgr2=0x7f228006cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.504+0000 7f2285a00640 1 -- 172.21.15.22:0/679951659 shutdown_connections 2024-03-24T06:45:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.504+0000 7f2285a00640 1 -- 172.21.15.22:0/679951659 wait complete. 2024-03-24T06:45:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.505+0000 7f2285a00640 1 Processor -- start 2024-03-24T06:45:28.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.505+0000 7f2285a00640 1 -- start start 2024-03-24T06:45:28.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228010df60 0x7f22801ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 0x7f22801acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f22801a7000 con 0x7f228010ff30 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f22801a7170 con 0x7f228010df60 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f2285a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f22801a7330 con 0x7f228011aef0 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f227f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f227f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57772/0 (socket says 172.21.15.22:57772) 2024-03-24T06:45:28.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.506+0000 7f227f400640 1 -- 172.21.15.22:0/3479217814 learned_addr learned my addr 172.21.15.22:0/3479217814 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.507+0000 7f227f400640 1 -- 172.21.15.22:0/3479217814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228010df60 msgr2=0x7f22801ac800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T06:45:28.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.507+0000 7f227f400640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228010df60 0x7f22801ac800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.507+0000 7f227f400640 1 -- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 msgr2=0x7f22801acd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T06:45:28.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f227e000640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 0x7f22801acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f227f400640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 0x7f22801acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f227f400640 1 -- 172.21.15.22:0/3479217814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c009560 con 0x7f228011aef0 2024-03-24T06:45:28.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f227f400640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f226c02c1c0 tx=0x7f226c02cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f2277400640 1 -- 172.21.15.22:0/3479217814 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0042d0 con 0x7f228011aef0 2024-03-24T06:45:28.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.508+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22801a75b0 con 0x7f228011aef0 2024-03-24T06:45:28.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.509+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801b38a0 con 0x7f228011aef0 2024-03-24T06:45:28.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.509+0000 7f2277400640 1 -- 172.21.15.22:0/3479217814 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f226c004430 con 0x7f228011aef0 2024-03-24T06:45:28.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.509+0000 7f2277400640 1 -- 172.21.15.22:0/3479217814 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c003c50 con 0x7f228011aef0 2024-03-24T06:45:28.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.512+0000 7f2277400640 1 -- 172.21.15.22:0/3479217814 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f226c004df0 con 0x7f228011aef0 2024-03-24T06:45:28.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.514+0000 7f2277400640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 0x7f22640a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.514+0000 7f2277400640 1 -- 172.21.15.22:0/3479217814 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f226c0f0c50 con 0x7f228011aef0 2024-03-24T06:45:28.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.515+0000 7f227ea00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 0x7f22640a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.522 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 Processor -- start 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 -- start start 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 0x5644834c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.521+0000 7f22885fa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644834366e0 con 0x5644834c8890 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f22885fa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564483433c10 con 0x5644834c7f70 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f22885fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56448342f3f0 con 0x5644834e2790 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 0x5644834c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57784/0 (socket says 172.21.15.22:57784) 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 0x5644834c8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51962/0 (socket says 172.21.15.22:51962) 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227f400640 1 -- 172.21.15.22:0/2267301684 learned_addr learned my addr 172.21.15.22:0/2267301684 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.522+0000 7f227e000640 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 -- 172.21.15.22:0/2267301684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 msgr2=0x5644834c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 0x5644834c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 -- 172.21.15.22:0/2267301684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 msgr2=0x5644834e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 -- 172.21.15.22:0/2267301684 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644833f0e40 con 0x5644834e2790 2024-03-24T06:45:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227ea00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 0x7f22640a47b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f22801a7af0 tx=0x7f2278009250 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f227f400640 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f226c0b9970 tx=0x7f226c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6664d163304be0e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f226be00640 1 -- 172.21.15.22:0/2267301684 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0d5070 con 0x5644834e2790 2024-03-24T06:45:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f226be00640 1 -- 172.21.15.22:0/2267301684 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f226c0c1660 con 0x5644834e2790 2024-03-24T06:45:28.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.523+0000 7f226be00640 1 -- 172.21.15.22:0/2267301684 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0054f0 con 0x5644834e2790 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.524+0000 7f22885fa740 1 -- 172.21.15.22:0/2267301684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 msgr2=0x5644834e4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.524+0000 7f22885fa740 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f226c0b9970 tx=0x7f226c009260 comp rx=0 tx=0).stop 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.525+0000 7f22885fa740 1 -- 172.21.15.22:0/2267301684 shutdown_connections 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.525+0000 7f22885fa740 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834e2790 0x5644834e4b80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.525+0000 7f22885fa740 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.525+0000 7f22885fa740 1 --2- 172.21.15.22:0/2267301684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c7f70 0x5644834c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.525+0000 7f22885fa740 1 -- 172.21.15.22:0/2267301684 >> 172.21.15.22:0/2267301684 conn(0x5644834c29b0 msgr2=0x5644834c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.526+0000 7f22885fa740 1 -- 172.21.15.22:0/2267301684 shutdown_connections 2024-03-24T06:45:28.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.526+0000 7f22885fa740 1 -- 172.21.15.22:0/2267301684 wait complete. 2024-03-24T06:45:28.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.526+0000 7f22885fa740 1 -- start start 2024-03-24T06:45:28.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.527+0000 7f22885fa740 1 Processor -- start 2024-03-24T06:45:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:28 smithi155 ceph-mon[36262]: pgmap v168: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.42k op/s 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c8890 0x5644834dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f227ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 0x5644834e0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644833d4060 con 0x5644834e2790 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644833d3d00 con 0x5644834c8890 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f22885fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56448319d1c0 con 0x5644834c7f70 2024-03-24T06:45:28.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f227ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57796/0 (socket says 172.21.15.22:57796) 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.528+0000 7f227ea00640 1 -- 172.21.15.22:0/2020676389 learned_addr learned my addr 172.21.15.22:0/2020676389 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.529+0000 7f22885fa740 10 client.0 _ll_register_callbacks cb 0x56448313fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.529+0000 7f227f400640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 0x5644834e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.529+0000 7f227e000640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c8890 0x5644834dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.529+0000 7f227f400640 1 -- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 msgr2=0x5644834df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.529+0000 7f227f400640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f227f400640 1 -- 172.21.15.22:0/2020676389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c8890 msgr2=0x5644834dfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f227f400640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c8890 0x5644834dfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f227f400640 1 -- 172.21.15.22:0/2020676389 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801b38a0 con 0x5644834e2790 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f227f400640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 0x5644834e0280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5644834e5250 tx=0x7f226c0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e08fd454850dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f2268c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f227ea00640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.530+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0d5070 con 0x5644834e2790 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.531+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56448312e7e0 con 0x5644834e2790 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.531+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f226c0c5cb0 con 0x5644834e2790 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.531+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f226c0052b0 con 0x5644834e2790 2024-03-24T06:45:28.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.531+0000 7f22885fa740 10 client.16212 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.531+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56448342da70 con 0x5644834e2790 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56448312e7e0 con 0x5644834e2790 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f226c03c070 con 0x5644834e2790 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 10 client.16212 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 10 client.16212 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 20 client.16212 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5644834f2440 0x5644834f4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 4 client.16212 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.532+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56448316b720 con 0x5644834f2440 2024-03-24T06:45:28.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.533+0000 7f227ea00640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5644834f2440 0x5644834f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.537+0000 7f227ea00640 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5644834f2440 0x5644834f4860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ac0ab773371cf44 server_cookie=6a4fd27d93cac62 in_seq=0 out_seq=0 2024-03-24T06:45:28.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.537+0000 7f2268c00640 10 client.16212 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:45:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.538+0000 7f2268c00640 1 -- 172.21.15.22:0/2020676389 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56448316b720 con 0x5644834f2440 2024-03-24T06:45:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.538+0000 7f2268c00640 10 client.16212 handle_command_reply: tid=0 2024-03-24T06:45:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.539+0000 7f22885fa740 1 client.16212 shutdown 2024-03-24T06:45:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.539+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 msgr2=0x5644834e0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.539+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 0x5644834e0280 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x5644834e5250 tx=0x7f226c0057a0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.539+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 shutdown_connections 2024-03-24T06:45:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.539+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5644834f2440 0x5644834f4860 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.540+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834e2790 0x5644834e0280 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.540+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834c8890 0x5644834dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.540+0000 7f22885fa740 1 --2- 172.21.15.22:0/2020676389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.540+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 >> 172.21.15.22:0/2020676389 conn(0x5644834c29b0 msgr2=0x5644834c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.542+0000 7f2268c00640 0 client.16212 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.542+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 shutdown_connections 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.542+0000 7f22885fa740 1 -- 172.21.15.22:0/2020676389 wait complete. 2024-03-24T06:45:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.542+0000 7f22885fa740 20 client.16212 trim_cache size 0 max 16384 2024-03-24T06:45:28.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.570+0000 7f22885fa740 1 Processor -- start 2024-03-24T06:45:28.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.570+0000 7f22885fa740 1 -- start start 2024-03-24T06:45:28.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 0x5644834e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c8890 0x5644834e00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644834c8d60 con 0x5644834c7f70 2024-03-24T06:45:28.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644834f4ec0 con 0x5644834e2790 2024-03-24T06:45:28.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.571+0000 7f22885fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5644834ef4f0 con 0x5644834c8890 2024-03-24T06:45:28.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:51980/0 (socket says 172.21.15.22:51980) 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 -- 172.21.15.22:0/257890126 learned_addr learned my addr 172.21.15.22:0/257890126 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227ea00640 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 0x5644834e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 -- 172.21.15.22:0/257890126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c8890 msgr2=0x5644834e00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c8890 0x5644834e00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 -- 172.21.15.22:0/257890126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 msgr2=0x5644834e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 0x5644834e1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227f400640 1 -- 172.21.15.22:0/257890126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644833c9b00 con 0x5644834e2790 2024-03-24T06:45:28.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.573+0000 7f227ea00640 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 0x5644834e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.574+0000 7f227f400640 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f226c0bb7c0 tx=0x7f226c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147eeb9b5f410420 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.574+0000 7f226b400640 1 -- 172.21.15.22:0/257890126 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0d5070 con 0x5644834e2790 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.574+0000 7f226b400640 1 -- 172.21.15.22:0/257890126 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f226c0bd2e0 con 0x5644834e2790 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.574+0000 7f226b400640 1 -- 172.21.15.22:0/257890126 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c0bcc40 con 0x5644834e2790 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 -- 172.21.15.22:0/257890126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 msgr2=0x5644834c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f226c0bb7c0 tx=0x7f226c009260 comp rx=0 tx=0).stop 2024-03-24T06:45:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 -- 172.21.15.22:0/257890126 shutdown_connections 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834c18f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c8890 0x5644834e00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 --2- 172.21.15.22:0/257890126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c7f70 0x5644834e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 -- 172.21.15.22:0/257890126 >> 172.21.15.22:0/257890126 conn(0x5644834c29b0 msgr2=0x5644834c11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.575+0000 7f22885fa740 1 -- 172.21.15.22:0/257890126 shutdown_connections 2024-03-24T06:45:28.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.576+0000 7f22885fa740 1 -- 172.21.15.22:0/257890126 wait complete. 2024-03-24T06:45:28.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.576+0000 7f22885fa740 1 -- start start 2024-03-24T06:45:28.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.576+0000 7f22885fa740 1 Processor -- start 2024-03-24T06:45:28.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.577+0000 7f22885fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834ea690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.577+0000 7f22885fa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.577+0000 7f22885fa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f22885fa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644834e6630 con 0x5644834c8890 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f227e000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f22885fa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644834e67a0 con 0x5644834e2790 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f22885fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564483557740 con 0x5644834c7f70 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f227e000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:57920/0 (socket says 172.21.15.22:57920) 2024-03-24T06:45:28.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f22885fa740 10 client.0 _ll_register_callbacks cb 0x7f226c0bd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:28.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.578+0000 7f227e000640 1 -- 172.21.15.22:0/1984873037 learned_addr learned my addr 172.21.15.22:0/1984873037 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 -- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 msgr2=0x5644834ea690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834ea690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 -- 172.21.15.22:0/1984873037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 msgr2=0x5644834e5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227f400640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 -- 172.21.15.22:0/1984873037 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56448342da70 con 0x5644834c8890 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f227e000640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22801a81a0 tx=0x7f22700137d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5e7030dab95838 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.579+0000 7f2268c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:45:28.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.581+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2270004520 con 0x5644834c8890 2024-03-24T06:45:28.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.581+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f2270005020 con 0x5644834c8890 2024-03-24T06:45:28.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.582+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f226c0c1660 con 0x5644834c8890 2024-03-24T06:45:28.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.583+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f22700051e0 con 0x5644834c8890 2024-03-24T06:45:28.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.583+0000 7f22885fa740 10 client.16218 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:28.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.583+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5644833ef6a0 con 0x5644834c8890 2024-03-24T06:45:28.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.585+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f226c0c1660 con 0x5644834c8890 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.585+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f2270003840 con 0x5644834c8890 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.585+0000 7f22885fa740 10 client.16218 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.585+0000 7f22885fa740 10 client.16218 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.585+0000 7f22885fa740 20 client.16218 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.586+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564483558ee0 0x56448355b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.586+0000 7f22885fa740 4 client.16218 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:45:28.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.586+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644831ba890 con 0x564483558ee0 2024-03-24T06:45:28.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.586+0000 7f227ea00640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564483558ee0 0x56448355b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.587+0000 7f227ea00640 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564483558ee0 0x56448355b300 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7df0c4429c08cdcc server_cookie=388ebd9f644c2beb in_seq=0 out_seq=0 2024-03-24T06:45:28.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.587+0000 7f2268c00640 10 client.16218 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:45:28.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.587+0000 7f2268c00640 1 -- 172.21.15.22:0/1984873037 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644831ba890 con 0x564483558ee0 2024-03-24T06:45:28.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.587+0000 7f2268c00640 10 client.16218 handle_command_reply: tid=0 2024-03-24T06:45:28.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.587+0000 7f22885fa740 1 client.16218 shutdown 2024-03-24T06:45:28.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.588+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 msgr2=0x5644834e5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.588+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22801a81a0 tx=0x7f22700137d0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 shutdown_connections 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564483558ee0 0x56448355b300 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644834e2790 0x5644834e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644834c8890 0x5644834e5a50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f2268c00640 0 client.16218 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 --2- 172.21.15.22:0/1984873037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644834c7f70 0x5644834ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 >> 172.21.15.22:0/1984873037 conn(0x5644834c29b0 msgr2=0x5644834c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.590+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 shutdown_connections 2024-03-24T06:45:28.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.591+0000 7f22885fa740 1 -- 172.21.15.22:0/1984873037 wait complete. 2024-03-24T06:45:28.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.591+0000 7f22885fa740 20 client.16218 trim_cache size 0 max 16384 2024-03-24T06:45:28.591 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:45:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.592+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 msgr2=0x7f22640a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.592+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 0x7f22640a47b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f22801a7af0 tx=0x7f2278009250 comp rx=0 tx=0).stop 2024-03-24T06:45:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 msgr2=0x7f22801a69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f226c02c1c0 tx=0x7f226c02cd10 comp rx=0 tx=0).stop 2024-03-24T06:45:28.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 shutdown_connections 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f22640a22f0 0x7f22640a47b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f228011aef0 0x7f22801a69b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f228010ff30 0x7f22801acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 --2- 172.21.15.22:0/3479217814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f228010df60 0x7f22801ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.593+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 >> 172.21.15.22:0/3479217814 conn(0x7f228006c7f0 msgr2=0x7f22800721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.594+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 shutdown_connections 2024-03-24T06:45:28.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.594+0000 7f2285a00640 1 -- 172.21.15.22:0/3479217814 wait complete. 2024-03-24T06:45:28.622 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:45:28.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 -- 172.21.15.22:0/3816902535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b4105900 msgr2=0x7fb4b4105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3816902535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b4105900 0x7fb4b4105ce0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009930 tx=0x7fb4ac02bc60 comp rx=0 tx=0).stop 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 -- 172.21.15.22:0/3816902535 shutdown_connections 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3816902535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b41002b0 0x7fb4b410c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3816902535 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 0x7fb4b40ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3816902535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b4105900 0x7fb4b4105ce0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 -- 172.21.15.22:0/3816902535 >> 172.21.15.22:0/3816902535 conn(0x7fb4b4076760 msgr2=0x7fb4b40fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.934+0000 7fb4bae00640 1 -- 172.21.15.22:0/3816902535 shutdown_connections 2024-03-24T06:45:28.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.935+0000 7fb4bae00640 1 -- 172.21.15.22:0/3816902535 wait complete. 2024-03-24T06:45:28.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.937+0000 7fb4bae00640 1 Processor -- start 2024-03-24T06:45:28.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.937+0000 7fb4bae00640 1 -- start start 2024-03-24T06:45:28.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 0x7fb4b41079a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b4105900 0x7fb4b4108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41aa790 con 0x7fb4b40ff8f0 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41aa8d0 con 0x7fb4b4105900 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.938+0000 7fb4bae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b41aaa10 con 0x7fb4b41002b0 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.939+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.939+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57832/0 (socket says 172.21.15.22:57832) 2024-03-24T06:45:28.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.939+0000 7fb4b3400640 1 -- 172.21.15.22:0/3175909548 learned_addr learned my addr 172.21.15.22:0/3175909548 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.939+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 0x7fb4b41079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.939+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b4105900 0x7fb4b4108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.940+0000 7fb4b3400640 1 -- 172.21.15.22:0/3175909548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b4105900 msgr2=0x7fb4b4108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.940+0000 7fb4b3400640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b4105900 0x7fb4b4108420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.940+0000 7fb4b3400640 1 -- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 msgr2=0x7fb4b41079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.940+0000 7fb4b3400640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 0x7fb4b41079a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.940+0000 7fb4b3400640 1 -- 172.21.15.22:0/3175909548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4ac009560 con 0x7fb4b41002b0 2024-03-24T06:45:28.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.941+0000 7fb4b3400640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b4d0 tx=0x7fb49c00b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.943+0000 7fb4b0c00640 1 -- 172.21.15.22:0/3175909548 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c010870 con 0x7fb4b41002b0 2024-03-24T06:45:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.943+0000 7fb4b0c00640 1 -- 172.21.15.22:0/3175909548 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb49c00f3c0 con 0x7fb4b41002b0 2024-03-24T06:45:28.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.943+0000 7fb4b0c00640 1 -- 172.21.15.22:0/3175909548 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0175c0 con 0x7fb4b41002b0 2024-03-24T06:45:28.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.943+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b41aacc0 con 0x7fb4b41002b0 2024-03-24T06:45:28.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.943+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b41ab180 con 0x7fb4b41002b0 2024-03-24T06:45:28.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.951+0000 7fb4bdbb8740 1 Processor -- start 2024-03-24T06:45:28.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.953+0000 7fb4b0c00640 1 -- 172.21.15.22:0/3175909548 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb49c00f530 con 0x7fb4b41002b0 2024-03-24T06:45:28.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.954+0000 7fb4bdbb8740 1 -- start start 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bc7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 0x560506bd0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506bd2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560506b356e0 con 0x560506bd0780 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560506b32c10 con 0x560506bc7ba0 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.956+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560506b2e3f0 con 0x560506bc7280 2024-03-24T06:45:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.957+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 0x560506bd0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.957+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 0x560506bd0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52018/0 (socket says 172.21.15.22:52018) 2024-03-24T06:45:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.957+0000 7fb4b3e00640 1 -- 172.21.15.22:0/1220878472 learned_addr learned my addr 172.21.15.22:0/1220878472 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.957+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bc7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.959+0000 7fb4b3400640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506bd2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.959+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1220878472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 msgr2=0x560506bd0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.959+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 0x560506bd0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.959+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1220878472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 msgr2=0x560506bd2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.960+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506bd2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.960+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1220878472 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560506aefe40 con 0x560506bc7280 2024-03-24T06:45:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.960+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bc7660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009f90 tx=0x7fb4a4009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d23be2df6a6b5b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.961+0000 7fb4b3400640 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506bd2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.961+0000 7fb4a9600640 1 -- 172.21.15.22:0/1220878472 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4010040 con 0x560506bc7280 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.961+0000 7fb4a9600640 1 -- 172.21.15.22:0/1220878472 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4a400ad90 con 0x560506bc7280 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.962+0000 7fb4b0c00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 0x7fb4980a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.963+0000 7fb4b0c00640 1 -- 172.21.15.22:0/3175909548 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb49c0cdb00 con 0x7fb4b41002b0 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.964+0000 7fb4a9600640 1 -- 172.21.15.22:0/1220878472 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4013380 con 0x560506bc7280 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.965+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1220878472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 msgr2=0x560506bc7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.965+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 0x7fb4980a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.965+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bc7660 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009f90 tx=0x7fb4a4009630 comp rx=0 tx=0).stop 2024-03-24T06:45:28.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.966+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 0x7fb4980a4880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac006010 tx=0x7fb4ac005850 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1220878472 shutdown_connections 2024-03-24T06:45:28.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506bd2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7ba0 0x560506bd0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1220878472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bc7660 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1220878472 >> 172.21.15.22:0/1220878472 conn(0x560506bc19b0 msgr2=0x560506bc1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1220878472 shutdown_connections 2024-03-24T06:45:28.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.967+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1220878472 wait complete. 2024-03-24T06:45:28.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.968+0000 7fb4bdbb8740 1 -- start start 2024-03-24T06:45:28.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.969+0000 7fb4bdbb8740 1 Processor -- start 2024-03-24T06:45:28.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.970+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bcdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57850/0 (socket says 172.21.15.22:57850) 2024-03-24T06:45:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4b3e00640 1 -- 172.21.15.22:0/1948417496 learned_addr learned my addr 172.21.15.22:0/1948417496 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560506ad3060 con 0x560506bc7ba0 2024-03-24T06:45:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560506ad2d00 con 0x560506bd0780 2024-03-24T06:45:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.971+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56050689c1c0 con 0x560506bc7280 2024-03-24T06:45:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.972+0000 7fb4bdbb8740 10 client.0 _ll_register_callbacks cb 0x56050683ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.972+0000 7fb4b3400640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bcdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.972+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 msgr2=0x560506bcd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.972+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.972+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1948417496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 msgr2=0x560506bcdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.973+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bcdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.973+0000 7fb4b8e00640 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a4009bc0 con 0x560506bd0780 2024-03-24T06:45:28.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.973+0000 7fb4b3400640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bcdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:28.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.973+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bce200 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400bd70 tx=0x7fb4a4015f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb633345edec71a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:28.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.973+0000 7fb4a2000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:28.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.974+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4010040 con 0x560506bd0780 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.974+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb4a40146f0 con 0x560506bd0780 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.974+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.974+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56050682d7e0 con 0x560506bd0780 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.975+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb4a400a8b0 con 0x560506bd0780 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.975+0000 7fb4bdbb8740 10 client.26573 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:28.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.975+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560506b2ca70 con 0x560506bd0780 2024-03-24T06:45:28.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.975+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56050682d7e0 con 0x560506bd0780 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fb4a401e400 con 0x560506bd0780 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 10 client.26573 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 10 client.26573 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 20 client.26573 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506be0c80 0x560506be3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 4 client.26573 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.977+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56050686a720 con 0x560506be0c80 2024-03-24T06:45:28.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.978+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506be0c80 0x560506be3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:28.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.979+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506be0c80 0x560506be3070 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d7eaef03a25be0e server_cookie=97d2908e22d91bf8 in_seq=0 out_seq=0 2024-03-24T06:45:28.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.979+0000 7fb4a2000640 10 client.26573 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:45:28.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.980+0000 7fb4a2000640 1 -- 172.21.15.22:0/1948417496 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56050686a720 con 0x560506be0c80 2024-03-24T06:45:28.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.980+0000 7fb4a2000640 10 client.26573 handle_command_reply: tid=0 2024-03-24T06:45:28.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.980+0000 7fb4bdbb8740 1 client.26573 shutdown 2024-03-24T06:45:28.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 msgr2=0x560506bce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:28.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bce200 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400bd70 tx=0x7fb4a4015f30 comp rx=0 tx=0).stop 2024-03-24T06:45:28.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 shutdown_connections 2024-03-24T06:45:28.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506be0c80 0x560506be3070 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bce200 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bcdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/1948417496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bcd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:28.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 >> 172.21.15.22:0/1948417496 conn(0x560506bc19b0 msgr2=0x560506bc3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:28.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.981+0000 7fb4a2000640 0 client.26573 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:45:28.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.982+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 shutdown_connections 2024-03-24T06:45:28.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.982+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/1948417496 wait complete. 2024-03-24T06:45:28.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:28.983+0000 7fb4bdbb8740 20 client.26573 trim_cache size 0 max 16384 2024-03-24T06:45:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.018+0000 7fb4bdbb8740 1 Processor -- start 2024-03-24T06:45:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 -- start start 2024-03-24T06:45:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 0x560506bcfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 0x560506bdc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506be8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560506bdc640 con 0x560506bd0780 2024-03-24T06:45:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560506bbd420 con 0x560506bc7280 2024-03-24T06:45:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.019+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560506bd2f90 con 0x560506bc7ba0 2024-03-24T06:45:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.020+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 0x560506bdc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.020+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 0x560506bdc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57852/0 (socket says 172.21.15.22:57852) 2024-03-24T06:45:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.020+0000 7fb4b3400640 1 -- 172.21.15.22:0/648299396 learned_addr learned my addr 172.21.15.22:0/648299396 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.020+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 0x560506bcfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.020+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506be8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.021+0000 7fb4b3e00640 1 -- 172.21.15.22:0/648299396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 msgr2=0x560506bdc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.021+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 0x560506bdc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.021+0000 7fb4b3e00640 1 -- 172.21.15.22:0/648299396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 msgr2=0x560506be8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.021+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506be8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.021+0000 7fb4b3e00640 1 -- 172.21.15.22:0/648299396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560506ac8b00 con 0x560506bc7280 2024-03-24T06:45:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.022+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 0x560506bcfeb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009900 tx=0x7fb4ac042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a45346021f39e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.022+0000 7fb4a8c00640 1 -- 172.21.15.22:0/648299396 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac055af0 con 0x560506bc7280 2024-03-24T06:45:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.022+0000 7fb4a8c00640 1 -- 172.21.15.22:0/648299396 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4ac052040 con 0x560506bc7280 2024-03-24T06:45:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.022+0000 7fb4a8c00640 1 -- 172.21.15.22:0/648299396 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac055af0 con 0x560506bc7280 2024-03-24T06:45:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/648299396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 msgr2=0x560506bcfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 0x560506bcfeb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009900 tx=0x7fb4ac042000 comp rx=0 tx=0).stop 2024-03-24T06:45:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/648299396 shutdown_connections 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bd0780 0x560506be8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7ba0 0x560506bdc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/648299396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bc7280 0x560506bcfeb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.023+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/648299396 >> 172.21.15.22:0/648299396 conn(0x560506bc19b0 msgr2=0x560506bc1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.024+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/648299396 shutdown_connections 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.024+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/648299396 wait complete. 2024-03-24T06:45:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.024+0000 7fb4bdbb8740 1 -- start start 2024-03-24T06:45:29.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 Processor -- start 2024-03-24T06:45:29.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bd4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bd4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560506c46010 con 0x560506bc7ba0 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560506c46180 con 0x560506bd0780 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.025+0000 7fb4bdbb8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560506c57350 con 0x560506bc7280 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.026+0000 7fb4bdbb8740 10 client.0 _ll_register_callbacks cb 0x7fb4ac005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.026+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.026+0000 7fb4b3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57866/0 (socket says 172.21.15.22:57866) 2024-03-24T06:45:29.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.026+0000 7fb4b3e00640 1 -- 172.21.15.22:0/510714818 learned_addr learned my addr 172.21.15.22:0/510714818 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b8e00640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bd4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3400640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bd4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3e00640 1 -- 172.21.15.22:0/510714818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 msgr2=0x560506bd4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bd4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3e00640 1 -- 172.21.15.22:0/510714818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 msgr2=0x560506bd4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bd4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3e00640 1 -- 172.21.15.22:0/510714818 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b410d0c0 con 0x560506bc7280 2024-03-24T06:45:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.027+0000 7fb4b3400640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bd4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.028+0000 7fb4a2000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.028+0000 7fb4b3e00640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009850 tx=0x7fb4ac055ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a81d076d734ed4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac055720 con 0x560506bc7280 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fb4ac004df0 con 0x560506bc7280 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4a400ad90 con 0x560506bc7280 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb4ac055720 con 0x560506bc7280 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4bdbb8740 10 client.26611 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.029+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560506aee6a0 con 0x560506bc7280 2024-03-24T06:45:29.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.030+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4ac044070 con 0x560506bc7280 2024-03-24T06:45:29.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fb4ac055070 con 0x560506bc7280 2024-03-24T06:45:29.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 10 client.26611 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:29.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 10 client.26611 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:45:29.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 20 client.26611 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:29.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506c58990 0x560506c5ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 4 client.26611 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:45:29.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.031+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb4a400ad90 con 0x560506c58990 2024-03-24T06:45:29.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.032+0000 7fb4b3400640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506c58990 0x560506c5ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.033+0000 7fb4b3400640 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506c58990 0x560506c5ad80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d9f5bf1341d3a633 server_cookie=f5b46c37d98bbe5b in_seq=0 out_seq=0 2024-03-24T06:45:29.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.033+0000 7fb4a2000640 10 client.26611 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:45:29.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.034+0000 7fb4a2000640 1 -- 172.21.15.22:0/510714818 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb4a400ad90 con 0x560506c58990 2024-03-24T06:45:29.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.034+0000 7fb4a2000640 10 client.26611 handle_command_reply: tid=0 2024-03-24T06:45:29.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.034+0000 7fb4bdbb8740 1 client.26611 shutdown 2024-03-24T06:45:29.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.034+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 msgr2=0x560506bd4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.034+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009850 tx=0x7fb4ac055ee0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 shutdown_connections 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x560506c58990 0x560506c5ad80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560506bd0780 0x560506bd4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560506bc7ba0 0x560506bd4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 --2- 172.21.15.22:0/510714818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560506bc7280 0x560506bd4200 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4a2000640 0 client.26611 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:45:29.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 >> 172.21.15.22:0/510714818 conn(0x560506bc19b0 msgr2=0x560506bc01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 shutdown_connections 2024-03-24T06:45:29.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 1 -- 172.21.15.22:0/510714818 wait complete. 2024-03-24T06:45:29.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.035+0000 7fb4bdbb8740 20 client.26611 trim_cache size 0 max 16384 2024-03-24T06:45:29.036 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.036+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 msgr2=0x7fb4980a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.036+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 0x7fb4980a4880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac006010 tx=0x7fb4ac005850 comp rx=0 tx=0).stop 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.036+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 msgr2=0x7fb4b4107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00b4d0 tx=0x7fb49c00b990 comp rx=0 tx=0).stop 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 shutdown_connections 2024-03-24T06:45:29.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb4980a23c0 0x7fb4980a4880 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b4105900 0x7fb4b4108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4b41002b0 0x7fb4b4107ee0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 --2- 172.21.15.22:0/3175909548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb4b40ff8f0 0x7fb4b41079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 >> 172.21.15.22:0/3175909548 conn(0x7fb4b4076760 msgr2=0x7fb4b4102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 shutdown_connections 2024-03-24T06:45:29.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.037+0000 7fb4bae00640 1 -- 172.21.15.22:0/3175909548 wait complete. 2024-03-24T06:45:29.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:45:29.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 -- 172.21.15.22:0/643796612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 msgr2=0x7f885c068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 --2- 172.21.15.22:0/643796612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c068910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f884c009930 tx=0x7f884c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:45:29.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 -- 172.21.15.22:0/643796612 shutdown_connections 2024-03-24T06:45:29.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 --2- 172.21.15.22:0/643796612 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 --2- 172.21.15.22:0/643796612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c0692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 --2- 172.21.15.22:0/643796612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c068910 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 -- 172.21.15.22:0/643796612 >> 172.21.15.22:0/643796612 conn(0x7f885c077710 msgr2=0x7f885c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 -- 172.21.15.22:0/643796612 shutdown_connections 2024-03-24T06:45:29.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.377+0000 7f8862600640 1 -- 172.21.15.22:0/643796612 wait complete. 2024-03-24T06:45:29.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.379+0000 7f8862600640 1 Processor -- start 2024-03-24T06:45:29.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.379+0000 7f8862600640 1 -- start start 2024-03-24T06:45:29.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c1a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1a31d0 con 0x7f885c1041d0 2024-03-24T06:45:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1a3310 con 0x7f885c106660 2024-03-24T06:45:29.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.380+0000 7f8862600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1a34d0 con 0x7f885c068e50 2024-03-24T06:45:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:58006/0 (socket says 172.21.15.22:58006) 2024-03-24T06:45:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885aa00640 1 -- 172.21.15.22:0/1942826017 learned_addr learned my addr 172.21.15.22:0/1942826017 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c1a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.381+0000 7f885be00640 1 -- 172.21.15.22:0/1942826017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 msgr2=0x7f885c19e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.384+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c19e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.384+0000 7f885be00640 1 -- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 msgr2=0x7f885c19e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.384+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.384+0000 7f885be00640 1 -- 172.21.15.22:0/1942826017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c009560 con 0x7f885c106660 2024-03-24T06:45:29.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f885b400640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:29.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c1a2bd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f885400be50 tx=0x7f885400bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f885aa00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:29.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f8853e00640 1 -- 172.21.15.22:0/1942826017 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885400ce90 con 0x7f885c106660 2024-03-24T06:45:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f8853e00640 1 -- 172.21.15.22:0/1942826017 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854014920 con 0x7f885c106660 2024-03-24T06:45:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c1a37b0 con 0x7f885c106660 2024-03-24T06:45:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.385+0000 7f8853e00640 1 -- 172.21.15.22:0/1942826017 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854012a70 con 0x7f885c106660 2024-03-24T06:45:29.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.386+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c106d80 con 0x7f885c106660 2024-03-24T06:45:29.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.392+0000 7f8853e00640 1 -- 172.21.15.22:0/1942826017 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8854012c50 con 0x7f885c106660 2024-03-24T06:45:29.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.392+0000 7f886527d740 1 Processor -- start 2024-03-24T06:45:29.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 -- start start 2024-03-24T06:45:29.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb875c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb875eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cdb86c16e0 con 0x55cdb8753ba0 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cdb86bec10 con 0x55cdb8753280 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.393+0000 7f886527d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cdb86ba3f0 con 0x55cdb875c780 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.394+0000 7f885aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.394+0000 7f885aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52068/0 (socket says 172.21.15.22:52068) 2024-03-24T06:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.394+0000 7f885aa00640 1 -- 172.21.15.22:0/2128563014 learned_addr learned my addr 172.21.15.22:0/2128563014 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.394+0000 7f885aa00640 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb875eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.394+0000 7f8853e00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 0x7f88400a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.395+0000 7f8853e00640 1 -- 172.21.15.22:0/1942826017 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f88540cee50 con 0x7f885c106660 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.395+0000 7f885b400640 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb875c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.395+0000 7f885aa00640 1 -- 172.21.15.22:0/2128563014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 msgr2=0x55cdb875eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.395+0000 7f885aa00640 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb875eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.395+0000 7f885aa00640 1 -- 172.21.15.22:0/2128563014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 msgr2=0x55cdb875c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.396+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 0x7f88400a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.396+0000 7f885aa00640 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb875c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.396+0000 7f885aa00640 1 -- 172.21.15.22:0/2128563014 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb867be40 con 0x55cdb8753280 2024-03-24T06:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.397+0000 7f885be00640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 0x7f88400a47f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88540125a0 tx=0x7f8854009f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.397+0000 7f885aa00640 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8848013cb0 tx=0x7f88480095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf355749911f125 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.397+0000 7f8850c00640 1 -- 172.21.15.22:0/2128563014 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848013da0 con 0x55cdb8753280 2024-03-24T06:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.397+0000 7f8850c00640 1 -- 172.21.15.22:0/2128563014 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8848013170 con 0x55cdb8753280 2024-03-24T06:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.397+0000 7f8850c00640 1 -- 172.21.15.22:0/2128563014 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848010dd0 con 0x55cdb8753280 2024-03-24T06:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.398+0000 7f886527d740 1 -- 172.21.15.22:0/2128563014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 msgr2=0x55cdb8753660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.398+0000 7f886527d740 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8848013cb0 tx=0x7f88480095b0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:29 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 -- 172.21.15.22:0/2128563014 shutdown_connections 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb875eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb875c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 --2- 172.21.15.22:0/2128563014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb8753660 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 -- 172.21.15.22:0/2128563014 >> 172.21.15.22:0/2128563014 conn(0x55cdb874d9b0 msgr2=0x55cdb874dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 -- 172.21.15.22:0/2128563014 shutdown_connections 2024-03-24T06:45:29.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 -- 172.21.15.22:0/2128563014 wait complete. 2024-03-24T06:45:29.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.399+0000 7f886527d740 1 -- start start 2024-03-24T06:45:29.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.400+0000 7f886527d740 1 Processor -- start 2024-03-24T06:45:29.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb8759c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb875c780 0x55cdb875a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cdb865f060 con 0x55cdb8753ba0 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cdb865ed00 con 0x55cdb875c780 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cdb84281c0 con 0x55cdb8753280 2024-03-24T06:45:29.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.403+0000 7f886527d740 10 client.0 _ll_register_callbacks cb 0x55cdb83cabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:29.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.404+0000 7f885aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.404+0000 7f885aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57886/0 (socket says 172.21.15.22:57886) 2024-03-24T06:45:29.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.404+0000 7f885aa00640 1 -- 172.21.15.22:0/3590208676 learned_addr learned my addr 172.21.15.22:0/3590208676 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.404+0000 7f885aa00640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb875c780 0x55cdb875a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 -- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb875c780 msgr2=0x55cdb875a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb875c780 0x55cdb875a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 -- 172.21.15.22:0/3590208676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 msgr2=0x55cdb8759c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb8759c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 -- 172.21.15.22:0/3590208676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884801b040 con 0x55cdb8753280 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f885aa00640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f884800fff0 tx=0x7f88480109b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6efe54dd3af358fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.405+0000 7f8845600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:29.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848010e20 con 0x55cdb8753280 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8848010430 con 0x55cdb8753280 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cdb83b97e0 con 0x55cdb8753280 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f88480105f0 con 0x55cdb8753280 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f886527d740 10 client.26620 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.406+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cdb86b8a70 con 0x55cdb8753280 2024-03-24T06:45:29.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.407+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8848004510 con 0x55cdb8753280 2024-03-24T06:45:29.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f88480046c0 con 0x55cdb8753280 2024-03-24T06:45:29.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 10 client.26620 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:29.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 10 client.26620 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:45:29.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 20 client.26620 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:29.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb876caf0 0x55cdb876ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 4 client.26620 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:29.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.409+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdb83b97e0 con 0x55cdb876caf0 2024-03-24T06:45:29.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.410+0000 7f885b400640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb876caf0 0x55cdb876ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.410+0000 7f8845600640 10 client.26620 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:45:29.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.411+0000 7f885b400640 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb876caf0 0x55cdb876ef70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3698249da2f5335 server_cookie=25a232a4e0cd7af3 in_seq=0 out_seq=0 2024-03-24T06:45:29.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.411+0000 7f8845600640 1 -- 172.21.15.22:0/3590208676 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cdb83b97e0 con 0x55cdb876caf0 2024-03-24T06:45:29.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.411+0000 7f8845600640 10 client.26620 handle_command_reply: tid=0 2024-03-24T06:45:29.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.412+0000 7f886527d740 1 client.26620 shutdown 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.412+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 msgr2=0x55cdb8759740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.412+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f884800fff0 tx=0x7f88480109b0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 shutdown_connections 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb876caf0 0x55cdb876ef70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb875c780 0x55cdb875a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753ba0 0x55cdb8759c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 --2- 172.21.15.22:0/3590208676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753280 0x55cdb8759740 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 >> 172.21.15.22:0/3590208676 conn(0x55cdb874d9b0 msgr2=0x55cdb874f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.413+0000 7f8845600640 0 client.26620 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:45:29.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.414+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 shutdown_connections 2024-03-24T06:45:29.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.414+0000 7f886527d740 1 -- 172.21.15.22:0/3590208676 wait complete. 2024-03-24T06:45:29.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.414+0000 7f886527d740 20 client.26620 trim_cache size 0 max 16384 2024-03-24T06:45:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:29 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:29.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.442+0000 7f886527d740 1 Processor -- start 2024-03-24T06:45:29.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.442+0000 7f886527d740 1 -- start start 2024-03-24T06:45:29.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.442+0000 7f886527d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f886527d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 0x55cdb8759fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f886527d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb875c780 0x55cdb8774830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f886527d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cdb875eba0 con 0x55cdb875c780 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f886527d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cdb8768610 con 0x55cdb8753280 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f886527d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cdb8752ee0 con 0x55cdb8753ba0 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f885aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f885b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 0x55cdb8759fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f885aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52082/0 (socket says 172.21.15.22:52082) 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f885aa00640 1 -- 172.21.15.22:0/2811313352 learned_addr learned my addr 172.21.15.22:0/2811313352 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.443+0000 7f885b400640 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 0x55cdb8759fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57902/0 (socket says 172.21.15.22:57902) 2024-03-24T06:45:29.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb875c780 0x55cdb8774830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 -- 172.21.15.22:0/2811313352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 msgr2=0x55cdb8759fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 0x55cdb8759fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 -- 172.21.15.22:0/2811313352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb875c780 msgr2=0x55cdb8774830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb875c780 0x55cdb8774830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.444+0000 7f885aa00640 1 -- 172.21.15.22:0/2811313352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdb8654b00 con 0x55cdb8753280 2024-03-24T06:45:29.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.445+0000 7f885aa00640 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8848005440 tx=0x7f8848005330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=128335cefe76cf7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.445+0000 7f8847e00640 1 -- 172.21.15.22:0/2811313352 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88480105f0 con 0x55cdb8753280 2024-03-24T06:45:29.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.446+0000 7f8847e00640 1 -- 172.21.15.22:0/2811313352 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8848004510 con 0x55cdb8753280 2024-03-24T06:45:29.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.446+0000 7f8847e00640 1 -- 172.21.15.22:0/2811313352 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88480105f0 con 0x55cdb8753280 2024-03-24T06:45:29.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 -- 172.21.15.22:0/2811313352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 msgr2=0x55cdb87680a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8848005440 tx=0x7f8848005330 comp rx=0 tx=0).stop 2024-03-24T06:45:29.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 -- 172.21.15.22:0/2811313352 shutdown_connections 2024-03-24T06:45:29.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb875c780 0x55cdb8774830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb8753ba0 0x55cdb8759fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 --2- 172.21.15.22:0/2811313352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753280 0x55cdb87680a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.449+0000 7f886527d740 1 -- 172.21.15.22:0/2811313352 >> 172.21.15.22:0/2811313352 conn(0x55cdb874d9b0 msgr2=0x55cdb874f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.450+0000 7f886527d740 1 -- 172.21.15.22:0/2811313352 shutdown_connections 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.450+0000 7f886527d740 1 -- 172.21.15.22:0/2811313352 wait complete. 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.451+0000 7f886527d740 1 -- start start 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.451+0000 7f886527d740 1 Processor -- start 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.452+0000 7f886527d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753280 0x55cdb8756b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb87d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cdb87d23a0 con 0x55cdb8753280 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cdb87d24e0 con 0x55cdb8753ba0 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cdb87d2670 con 0x55cdb875c780 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f886527d740 10 client.0 _ll_register_callbacks cb 0x7f8848005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.453+0000 7f885b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.454+0000 7f885b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52086/0 (socket says 172.21.15.22:52086) 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.454+0000 7f885aa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753280 0x55cdb8756b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.454+0000 7f885b400640 1 -- 172.21.15.22:0/1755568572 learned_addr learned my addr 172.21.15.22:0/1755568572 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885aa00640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb87d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 -- 172.21.15.22:0/1755568572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 msgr2=0x55cdb87d1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb87d1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 -- 172.21.15.22:0/1755568572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753280 msgr2=0x55cdb8756b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753280 0x55cdb8756b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 -- 172.21.15.22:0/1755568572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdb86b8a70 con 0x55cdb8753ba0 2024-03-24T06:45:29.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885aa00640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb87d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:45:29.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f885b400640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f884c02e010 tx=0x7f884c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788e9e985463a57d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.455+0000 7f8845600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:29.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.460+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c02cea0 con 0x55cdb8753ba0 2024-03-24T06:45:29.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.460+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f884c044040 con 0x55cdb8753ba0 2024-03-24T06:45:29.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.460+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8848013170 con 0x55cdb8753ba0 2024-03-24T06:45:29.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.460+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f884c02c7e0 con 0x55cdb8753ba0 2024-03-24T06:45:29.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.461+0000 7f886527d740 10 client.26597 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:29.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.461+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cdb867a6a0 con 0x55cdb8753ba0 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8848013170 con 0x55cdb8753ba0 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f884c03c040 con 0x55cdb8753ba0 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 10 client.26597 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 10 client.26597 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 20 client.26597 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:29.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb87d3f00 0x55cdb87d6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 4 client.26597 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:45:29.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdb83f6720 con 0x55cdb87d3f00 2024-03-24T06:45:29.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.462+0000 7f885aa00640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb87d3f00 0x55cdb87d6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f885aa00640 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb87d3f00 0x55cdb87d6320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aeb17fd053b96427 server_cookie=814246f7c77b7a2c in_seq=0 out_seq=0 2024-03-24T06:45:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f8845600640 10 client.26597 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:45:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f8845600640 1 -- 172.21.15.22:0/1755568572 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdb83f6720 con 0x55cdb87d3f00 2024-03-24T06:45:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f8845600640 10 client.26597 handle_command_reply: tid=0 2024-03-24T06:45:29.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f886527d740 1 client.26597 shutdown 2024-03-24T06:45:29.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 msgr2=0x55cdb8757050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.464+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f884c02e010 tx=0x7f884c005e80 comp rx=0 tx=0).stop 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 shutdown_connections 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cdb875c780 0x55cdb87d1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55cdb87d3f00 0x55cdb87d6320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cdb8753ba0 0x55cdb8757050 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 --2- 172.21.15.22:0/1755568572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cdb8753280 0x55cdb8756b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 >> 172.21.15.22:0/1755568572 conn(0x55cdb874d9b0 msgr2=0x55cdb874c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f8845600640 0 client.26597 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:45:29.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 shutdown_connections 2024-03-24T06:45:29.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 1 -- 172.21.15.22:0/1755568572 wait complete. 2024-03-24T06:45:29.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.465+0000 7f886527d740 20 client.26597 trim_cache size 0 max 16384 2024-03-24T06:45:29.466 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:45:29.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 msgr2=0x7f88400a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 0x7f88400a47f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88540125a0 tx=0x7f8854009f90 comp rx=0 tx=0).stop 2024-03-24T06:45:29.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 msgr2=0x7f885c1a2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c1a2bd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f885400be50 tx=0x7f885400bf40 comp rx=0 tx=0).stop 2024-03-24T06:45:29.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 shutdown_connections 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f88400a2330 0x7f88400a47f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f885c106660 0x7f885c1a2bd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f885c1041d0 0x7f885c19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 --2- 172.21.15.22:0/1942826017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f885c068e50 0x7f885c19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 >> 172.21.15.22:0/1942826017 conn(0x7f885c077710 msgr2=0x7f885c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.467+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 shutdown_connections 2024-03-24T06:45:29.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.468+0000 7f8862600640 1 -- 172.21.15.22:0/1942826017 wait complete. 2024-03-24T06:45:29.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:45:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:29 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:45:29.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 -- 172.21.15.22:0/4201467595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c0ff7e0 msgr2=0x7fc43c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 --2- 172.21.15.22:0/4201467595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c0ff7e0 0x7fc43c0ffbc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009ca0 tx=0x7fc42c02c120 comp rx=0 tx=0).stop 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 -- 172.21.15.22:0/4201467595 shutdown_connections 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 --2- 172.21.15.22:0/4201467595 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c112500 0x7fc43c1148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 --2- 172.21.15.22:0/4201467595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c100100 0x7fc43c111fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 --2- 172.21.15.22:0/4201467595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c0ff7e0 0x7fc43c0ffbc0 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009ca0 tx=0x7fc42c02c120 comp rx=0 tx=0).stop 2024-03-24T06:45:29.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 -- 172.21.15.22:0/4201467595 >> 172.21.15.22:0/4201467595 conn(0x7fc43c076760 msgr2=0x7fc43c0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.799+0000 7fc442800640 1 -- 172.21.15.22:0/4201467595 shutdown_connections 2024-03-24T06:45:29.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.800+0000 7fc442800640 1 -- 172.21.15.22:0/4201467595 wait complete. 2024-03-24T06:45:29.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.801+0000 7fc442800640 1 Processor -- start 2024-03-24T06:45:29.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 -- start start 2024-03-24T06:45:29.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c100100 0x7fc43c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 0x7fc43c1a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c1a38a0 con 0x7fc43c1a2eb0 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c1a3a10 con 0x7fc43c100100 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc442800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c1a3b80 con 0x7fc43c112500 2024-03-24T06:45:29.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.802+0000 7fc432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57930/0 (socket says 172.21.15.22:57930) 2024-03-24T06:45:29.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 -- 172.21.15.22:0/3843018810 learned_addr learned my addr 172.21.15.22:0/3843018810 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc43b400640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c100100 0x7fc43c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc43be00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 0x7fc43c1a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 -- 172.21.15.22:0/3843018810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c100100 msgr2=0x7fc43c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c100100 0x7fc43c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 -- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 msgr2=0x7fc43c1a3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 0x7fc43c1a3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.803+0000 7fc432a00640 1 -- 172.21.15.22:0/3843018810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c009900 con 0x7fc43c112500 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc43be00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 0x7fc43c1a3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc432a00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b450 tx=0x7fc42400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc438c00640 1 -- 172.21.15.22:0/3843018810 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424011890 con 0x7fc43c112500 2024-03-24T06:45:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc438c00640 1 -- 172.21.15.22:0/3843018810 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4240103e0 con 0x7fc43c112500 2024-03-24T06:45:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43c1ab160 con 0x7fc43c112500 2024-03-24T06:45:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.804+0000 7fc438c00640 1 -- 172.21.15.22:0/3843018810 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400f5a0 con 0x7fc43c112500 2024-03-24T06:45:29.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.805+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43c1ab660 con 0x7fc43c112500 2024-03-24T06:45:29.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.810+0000 7fc445411740 1 Processor -- start 2024-03-24T06:45:29.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.811+0000 7fc445411740 1 -- start start 2024-03-24T06:45:29.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.811+0000 7fc445411740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa2107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.811+0000 7fc445411740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa2110240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc445411740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa2112b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.811 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc445411740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556aa20756e0 con 0x556aa2110780 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc445411740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556aa2072c10 con 0x556aa2107ba0 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc445411740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556aa206e3f0 con 0x556aa2107280 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa2107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa2110240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa2110240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52104/0 (socket says 172.21.15.22:52104) 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43b400640 1 -- 172.21.15.22:0/1394653256 learned_addr learned my addr 172.21.15.22:0/1394653256 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43be00640 1 -- 172.21.15.22:0/1394653256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 msgr2=0x556aa2110240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43be00640 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa2110240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43be00640 1 -- 172.21.15.22:0/1394653256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 msgr2=0x556aa2112b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:45:29.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.812+0000 7fc43be00640 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa2112b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.813+0000 7fc43be00640 1 -- 172.21.15.22:0/1394653256 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aa202fe40 con 0x556aa2107280 2024-03-24T06:45:29.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.813+0000 7fc43be00640 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa2107660 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc43c073a60 tx=0x7fc43400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=471d20a776227676 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.813+0000 7fc430c00640 1 -- 172.21.15.22:0/1394653256 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434004020 con 0x556aa2107280 2024-03-24T06:45:29.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.813+0000 7fc430c00640 1 -- 172.21.15.22:0/1394653256 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc434004180 con 0x556aa2107280 2024-03-24T06:45:29.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.813+0000 7fc430c00640 1 -- 172.21.15.22:0/1394653256 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340116a0 con 0x556aa2107280 2024-03-24T06:45:29.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc438c00640 1 -- 172.21.15.22:0/3843018810 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc424010550 con 0x7fc43c112500 2024-03-24T06:45:29.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 -- 172.21.15.22:0/1394653256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 msgr2=0x556aa2107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa2107660 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc43c073a60 tx=0x7fc43400f730 comp rx=0 tx=0).stop 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 -- 172.21.15.22:0/1394653256 shutdown_connections 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa2112b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa2110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 --2- 172.21.15.22:0/1394653256 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa2107660 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.814+0000 7fc445411740 1 -- 172.21.15.22:0/1394653256 >> 172.21.15.22:0/1394653256 conn(0x556aa21019b0 msgr2=0x556aa2101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.817+0000 7fc445411740 1 -- 172.21.15.22:0/1394653256 shutdown_connections 2024-03-24T06:45:29.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.817+0000 7fc445411740 1 -- 172.21.15.22:0/1394653256 wait complete. 2024-03-24T06:45:29.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.818+0000 7fc445411740 1 -- start start 2024-03-24T06:45:29.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.818+0000 7fc445411740 1 Processor -- start 2024-03-24T06:45:29.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.820+0000 7fc438c00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 0x7fc4200a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.822+0000 7fc43be00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 0x7fc4200a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.822+0000 7fc438c00640 1 -- 172.21.15.22:0/3843018810 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc4240ce0a0 con 0x7fc43c112500 2024-03-24T06:45:29.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.823+0000 7fc43be00640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 0x7fc4200a4700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc43400fc30 tx=0x7fc434003db0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.823+0000 7fc445411740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.823+0000 7fc445411740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc445411740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc445411740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556aa2013060 con 0x556aa2110780 2024-03-24T06:45:29.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc445411740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556aa2012d00 con 0x556aa2107ba0 2024-03-24T06:45:29.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc445411740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556aa1ddc1c0 con 0x556aa2107280 2024-03-24T06:45:29.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc445411740 10 client.0 _ll_register_callbacks cb 0x556aa1d7ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:29.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57942/0 (socket says 172.21.15.22:57942) 2024-03-24T06:45:29.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 -- 172.21.15.22:0/1252876736 learned_addr learned my addr 172.21.15.22:0/1252876736 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43be00640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 -- 172.21.15.22:0/1252876736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 msgr2=0x556aa210ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 -- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 msgr2=0x556aa210e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.824+0000 7fc43b400640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.825+0000 7fc43b400640 1 -- 172.21.15.22:0/1252876736 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43400b8e0 con 0x556aa2107280 2024-03-24T06:45:29.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.826+0000 7fc43b400640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc42c037040 tx=0x7fc42c032bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e135fdec3a8ee35d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.826+0000 7fc429600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:29.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.828+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c034850 con 0x556aa2107280 2024-03-24T06:45:29.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.828+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc42c046070 con 0x556aa2107280 2024-03-24T06:45:29.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.829+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556aa1d6d7e0 con 0x556aa2107280 2024-03-24T06:45:29.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.829+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc42c032eb0 con 0x556aa2107280 2024-03-24T06:45:29.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.829+0000 7fc445411740 10 client.26632 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:29.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.829+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556aa206ca70 con 0x556aa2107280 2024-03-24T06:45:29.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.830+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556aa1d6d7e0 con 0x556aa2107280 2024-03-24T06:45:29.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.831+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fc42c04b480 con 0x556aa2107280 2024-03-24T06:45:29.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 10 client.26632 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:29.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 10 client.26632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:29.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 20 client.26632 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:29.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2120a70 0x556aa2122e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 4 client.26632 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:29.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.832+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556aa1daa720 con 0x556aa2120a70 2024-03-24T06:45:29.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.833+0000 7fc432a00640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2120a70 0x556aa2122e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.838+0000 7fc432a00640 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2120a70 0x556aa2122e90 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e94e239a6e7ee0 server_cookie=23d0c3752e5e06db in_seq=0 out_seq=0 2024-03-24T06:45:29.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.838+0000 7fc429600640 10 client.26632 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:29.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.839+0000 7fc429600640 1 -- 172.21.15.22:0/1252876736 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556aa1daa720 con 0x556aa2120a70 2024-03-24T06:45:29.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.839+0000 7fc429600640 10 client.26632 handle_command_reply: tid=0 2024-03-24T06:45:29.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.840+0000 7fc445411740 1 client.26632 shutdown 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 msgr2=0x556aa210d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc42c037040 tx=0x7fc42c032bf0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 shutdown_connections 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2120a70 0x556aa2122e90 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 --2- 172.21.15.22:0/1252876736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107280 0x556aa210d8b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 >> 172.21.15.22:0/1252876736 conn(0x556aa21019b0 msgr2=0x556aa2103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc429600640 0 client.26632 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:29.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 shutdown_connections 2024-03-24T06:45:29.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.841+0000 7fc445411740 1 -- 172.21.15.22:0/1252876736 wait complete. 2024-03-24T06:45:29.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.842+0000 7fc445411740 20 client.26632 trim_cache size 0 max 16384 2024-03-24T06:45:29.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.866+0000 7fc445411740 1 Processor -- start 2024-03-24T06:45:29.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.866+0000 7fc445411740 1 -- start start 2024-03-24T06:45:29.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2107280 0x556aa210ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556aa2107a00 con 0x556aa2107280 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556aa2106ee0 con 0x556aa2107ba0 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc445411740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556aa2102740 con 0x556aa2110780 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc43be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc432a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc43be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57948/0 (socket says 172.21.15.22:57948) 2024-03-24T06:45:29.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc432a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52132/0 (socket says 172.21.15.22:52132) 2024-03-24T06:45:29.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc43be00640 1 -- 172.21.15.22:0/3192673243 learned_addr learned my addr 172.21.15.22:0/3192673243 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.867+0000 7fc43b400640 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2107280 0x556aa210ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 -- 172.21.15.22:0/3192673243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 msgr2=0x556aa210e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 -- 172.21.15.22:0/3192673243 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2107280 msgr2=0x556aa210ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2107280 0x556aa210ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 -- 172.21.15.22:0/3192673243 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aa2008b00 con 0x556aa2110780 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.868+0000 7fc43be00640 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc434018000 tx=0x7fc434024000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2df2af4777d8f399 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.869+0000 7fc42be00640 1 -- 172.21.15.22:0/3192673243 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434003e20 con 0x556aa2110780 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.869+0000 7fc42be00640 1 -- 172.21.15.22:0/3192673243 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc43402b040 con 0x556aa2110780 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.869+0000 7fc42be00640 1 -- 172.21.15.22:0/3192673243 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434004680 con 0x556aa2110780 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 -- 172.21.15.22:0/3192673243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 msgr2=0x556aa2128780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc434018000 tx=0x7fc434024000 comp rx=0 tx=0).stop 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 -- 172.21.15.22:0/3192673243 shutdown_connections 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2110780 0x556aa2128780 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107ba0 0x556aa210e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 --2- 172.21.15.22:0/3192673243 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2107280 0x556aa210ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.870+0000 7fc445411740 1 -- 172.21.15.22:0/3192673243 >> 172.21.15.22:0/3192673243 conn(0x556aa21019b0 msgr2=0x556aa2102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.871+0000 7fc445411740 1 -- 172.21.15.22:0/3192673243 shutdown_connections 2024-03-24T06:45:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.871+0000 7fc445411740 1 -- 172.21.15.22:0/3192673243 wait complete. 2024-03-24T06:45:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.871+0000 7fc445411740 1 -- start start 2024-03-24T06:45:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.871+0000 7fc445411740 1 Processor -- start 2024-03-24T06:45:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.872+0000 7fc445411740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 0x556aa2118640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.872+0000 7fc445411740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 0x556aa210aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.872+0000 7fc445411740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc445411740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556aa2118b80 con 0x556aa2110780 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc445411740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556aa2118cf0 con 0x556aa2107280 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc445411740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556aa2118eb0 con 0x556aa2107ba0 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc445411740 10 client.0 _ll_register_callbacks cb 0x7fc42c032e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 0x556aa210aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 0x556aa2118640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc432a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 0x556aa210aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:57962/0 (socket says 172.21.15.22:57962) 2024-03-24T06:45:29.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc432a00640 1 -- 172.21.15.22:0/324784055 learned_addr learned my addr 172.21.15.22:0/324784055 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43be00640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 -- 172.21.15.22:0/324784055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 msgr2=0x556aa210aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 0x556aa210aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 -- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 msgr2=0x556aa210afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.873+0000 7fc43b400640 1 -- 172.21.15.22:0/324784055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aa202fe40 con 0x556aa2107280 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.874+0000 7fc43b400640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 0x556aa2118640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc42c04b960 tx=0x7fc42c0327f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6a35e3d04a0e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.874+0000 7fc429600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:29.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.874+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c032860 con 0x556aa2107280 2024-03-24T06:45:29.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.874+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fc42c046040 con 0x556aa2107280 2024-03-24T06:45:29.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.874+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc434004180 con 0x556aa2107280 2024-03-24T06:45:29.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.875+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc42c032eb0 con 0x556aa2107280 2024-03-24T06:45:29.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.875+0000 7fc445411740 10 client.26612 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:29.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.875+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556aa202e6a0 con 0x556aa2107280 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc434004180 con 0x556aa2107280 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fc42c004410 con 0x556aa2107280 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 10 client.26612 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 10 client.26612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 20 client.26612 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2198b90 0x556aa219af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 4 client.26612 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:45:29.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.877+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556aa1df9890 con 0x556aa2198b90 2024-03-24T06:45:29.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.882+0000 7fc432a00640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2198b90 0x556aa219af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.883+0000 7fc432a00640 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2198b90 0x556aa219af80 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8897fc514247e9b server_cookie=73b2ee7001bdfb4c in_seq=0 out_seq=0 2024-03-24T06:45:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.883+0000 7fc429600640 10 client.26612 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.884+0000 7fc429600640 1 -- 172.21.15.22:0/324784055 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556aa1df9890 con 0x556aa2198b90 2024-03-24T06:45:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.884+0000 7fc429600640 10 client.26612 handle_command_reply: tid=0 2024-03-24T06:45:29.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.884+0000 7fc445411740 1 client.26612 shutdown 2024-03-24T06:45:29.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.884+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 msgr2=0x556aa2118640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.884+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 0x556aa2118640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc42c04b960 tx=0x7fc42c0327f0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 shutdown_connections 2024-03-24T06:45:29.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556aa2110780 0x556aa210afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556aa2107ba0 0x556aa210aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556aa2198b90 0x556aa219af80 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 --2- 172.21.15.22:0/324784055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556aa2107280 0x556aa2118640 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc429600640 0 client.26612 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:29.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.885+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 >> 172.21.15.22:0/324784055 conn(0x556aa21019b0 msgr2=0x556aa21033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.888+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 shutdown_connections 2024-03-24T06:45:29.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.888+0000 7fc445411740 1 -- 172.21.15.22:0/324784055 wait complete. 2024-03-24T06:45:29.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.888+0000 7fc445411740 20 client.26612 trim_cache size 0 max 16384 2024-03-24T06:45:29.888 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:45:29.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.893+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 msgr2=0x7fc4200a4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.893+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 0x7fc4200a4700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc43400fc30 tx=0x7fc434003db0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.893+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 msgr2=0x7fc43c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:29.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.893+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b450 tx=0x7fc42400b910 comp rx=0 tx=0).stop 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 shutdown_connections 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc4200a2240 0x7fc4200a4700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc43c1a2eb0 0x7fc43c1a3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc43c112500 0x7fc43c19eb80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 --2- 172.21.15.22:0/3843018810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc43c100100 0x7fc43c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.895+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 >> 172.21.15.22:0/3843018810 conn(0x7fc43c076760 msgr2=0x7fc43c110f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:29.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.896+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 shutdown_connections 2024-03-24T06:45:29.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:29.896+0000 7fc442800640 1 -- 172.21.15.22:0/3843018810 wait complete. 2024-03-24T06:45:30.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:30 smithi022 ceph-mon[32003]: pgmap v169: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.03k op/s 2024-03-24T06:45:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:30 smithi143 ceph-mon[36520]: pgmap v169: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.03k op/s 2024-03-24T06:45:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:30 smithi155 ceph-mon[36262]: pgmap v169: 129 pgs: 129 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.03k op/s 2024-03-24T06:45:30.919 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:45:30.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:45:30.920 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:45:31.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 -- 172.21.15.22:0/3147440515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae4100df0 msgr2=0x7feae410db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 --2- 172.21.15.22:0/3147440515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae4100df0 0x7feae410db40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7feacc009860 tx=0x7feacc02bb70 comp rx=0 tx=0).stop 2024-03-24T06:45:31.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 -- 172.21.15.22:0/3147440515 shutdown_connections 2024-03-24T06:45:31.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 --2- 172.21.15.22:0/3147440515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae410e080 0x7feae4110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 --2- 172.21.15.22:0/3147440515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae4100df0 0x7feae410db40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 --2- 172.21.15.22:0/3147440515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae41008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 -- 172.21.15.22:0/3147440515 >> 172.21.15.22:0/3147440515 conn(0x7feae4076990 msgr2=0x7feae4076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 -- 172.21.15.22:0/3147440515 shutdown_connections 2024-03-24T06:45:31.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.172+0000 7feaea800640 1 -- 172.21.15.22:0/3147440515 wait complete. 2024-03-24T06:45:31.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.173+0000 7feaea800640 1 Processor -- start 2024-03-24T06:45:31.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.173+0000 7feaea800640 1 -- start start 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae410e080 0x7feae419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7feae419ee80 con 0x7feae41004d0 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feae419eff0 con 0x7feae4100df0 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feaea800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feae419f1b0 con 0x7feae410e080 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feadaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feae3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feadaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52150/0 (socket says 172.21.15.22:52150) 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feadaa00640 1 -- 172.21.15.22:0/1965088570 learned_addr learned my addr 172.21.15.22:0/1965088570 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.174+0000 7feae3e00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae410e080 0x7feae419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 -- 172.21.15.22:0/1965088570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae410e080 msgr2=0x7feae419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae410e080 0x7feae419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 -- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 msgr2=0x7feae4199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae4199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 -- 172.21.15.22:0/1965088570 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc009490 con 0x7feae4100df0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feae3400640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feadaa00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feacc037040 tx=0x7feacc0045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feae0c00640 1 -- 172.21.15.22:0/1965088570 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0349d0 con 0x7feae4100df0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feae0c00640 1 -- 172.21.15.22:0/1965088570 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feacc02c9f0 con 0x7feae4100df0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae419f430 con 0x7feae4100df0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.175+0000 7feae0c00640 1 -- 172.21.15.22:0/1965088570 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc0035f0 con 0x7feae4100df0 2024-03-24T06:45:31.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.176+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae419f700 con 0x7feae4100df0 2024-03-24T06:45:31.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.178+0000 7feae0c00640 1 -- 172.21.15.22:0/1965088570 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7feacc03b040 con 0x7feae4100df0 2024-03-24T06:45:31.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.179+0000 7feaed569740 1 Processor -- start 2024-03-24T06:45:31.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.179+0000 7feaed569740 1 -- start start 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b61210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b6c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5633a4acf2d0 con 0x5633a4b60e30 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633a4acc800 con 0x5633a4b61750 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feaed569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5633a4ac7fe0 con 0x5633a4b6a330 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feadaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b6c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52166/0 (socket says 172.21.15.22:52166) 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b61210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3400640 1 -- 172.21.15.22:0/3105440145 learned_addr learned my addr 172.21.15.22:0/3105440145 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae0c00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 0x7feac40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3400640 1 -- 172.21.15.22:0/3105440145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 msgr2=0x5633a4b6c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.180+0000 7feae3400640 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b6c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae3400640 1 -- 172.21.15.22:0/3105440145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 msgr2=0x5633a4b61210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae3400640 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b61210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae3400640 1 -- 172.21.15.22:0/3105440145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633a4a89e90 con 0x5633a4b61750 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae0c00640 1 -- 172.21.15.22:0/1965088570 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feacc0eff00 con 0x7feae4100df0 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae3e00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 0x7feac40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7feae3400640 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fead4011510 tx=0x7fead40119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c8a5241883dfa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7fead8c00640 1 -- 172.21.15.22:0/3105440145 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4004020 con 0x5633a4b61750 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7fead8c00640 1 -- 172.21.15.22:0/3105440145 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead4004180 con 0x5633a4b61750 2024-03-24T06:45:31.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.181+0000 7fead8c00640 1 -- 172.21.15.22:0/3105440145 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead400f930 con 0x5633a4b61750 2024-03-24T06:45:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 -- 172.21.15.22:0/3105440145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 msgr2=0x5633a4b69df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feae3e00640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 0x7feac40a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feadc005c40 tx=0x7feadc004a40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fead4011510 tx=0x7fead40119d0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 -- 172.21.15.22:0/3105440145 shutdown_connections 2024-03-24T06:45:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b6c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b61750 0x5633a4b69df0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 --2- 172.21.15.22:0/3105440145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b61210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 -- 172.21.15.22:0/3105440145 >> 172.21.15.22:0/3105440145 conn(0x5633a4b5b560 msgr2=0x5633a4b5b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 -- 172.21.15.22:0/3105440145 shutdown_connections 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.182+0000 7feaed569740 1 -- 172.21.15.22:0/3105440145 wait complete. 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.183+0000 7feaed569740 1 -- start start 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.183+0000 7feaed569740 1 Processor -- start 2024-03-24T06:45:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.183+0000 7feaed569740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.183+0000 7feaed569740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b678c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feaed569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b67e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feaed569740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5633a4a6cfc0 con 0x5633a4b61750 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feaed569740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633a4a6cc60 con 0x5633a4b60e30 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feaed569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5633a48100c0 con 0x5633a4b6a330 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feaed569740 10 client.0 _ll_register_callbacks cb 0x5633a4a84e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52172/0 (socket says 172.21.15.22:52172) 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b67e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3400640 1 -- 172.21.15.22:0/2642329492 learned_addr learned my addr 172.21.15.22:0/2642329492 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feadaa00640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b678c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 -- 172.21.15.22:0/2642329492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 msgr2=0x5633a4b67380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 -- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 msgr2=0x5633a4b678c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b678c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3e00640 1 -- 172.21.15.22:0/2642329492 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead4009d00 con 0x5633a4b6a330 2024-03-24T06:45:31.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feadaa00640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.184+0000 7feae3400640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7feae3e00640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b67e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abde0796cc12f27c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7fead1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc022070 con 0x5633a4b6a330 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feadc003970 con 0x5633a4b6a330 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633a4aece90 con 0x5633a4b6a330 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.185+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feadc003b30 con 0x5633a4b6a330 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 10 client.26650 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:31.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5633a4ac6660 con 0x5633a4b6a330 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5633a4aece90 con 0x5633a4b6a330 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7feadc01d580 con 0x5633a4b6a330 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 10 client.26650 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 10 client.26650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 20 client.26650 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4b7a8f0 0x5633a4b7cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 4 client.26650 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:31.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633a4abe210 con 0x5633a4b7a8f0 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.186+0000 7feae3400640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4b7a8f0 0x5633a4b7cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.187+0000 7fead1600640 10 client.26650 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.187+0000 7feae3400640 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4b7a8f0 0x5633a4b7cd10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4a1dc1f2a70d825 server_cookie=626c5f21e8055ea in_seq=0 out_seq=0 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7fead1600640 1 -- 172.21.15.22:0/2642329492 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633a4abe210 con 0x5633a4b7a8f0 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7fead1600640 10 client.26650 handle_command_reply: tid=0 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 client.26650 shutdown 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 msgr2=0x5633a4b67e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b67e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc004ab0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 shutdown_connections 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b67e00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b678c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4b7a8f0 0x5633a4b7cd10 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 --2- 172.21.15.22:0/2642329492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b67380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 >> 172.21.15.22:0/2642329492 conn(0x5633a4b5b560 msgr2=0x5633a4b5d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7fead1600640 0 client.26650 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.188+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 shutdown_connections 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.189+0000 7feaed569740 1 -- 172.21.15.22:0/2642329492 wait complete. 2024-03-24T06:45:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.189+0000 7feaed569740 20 client.26650 trim_cache size 0 max 16384 2024-03-24T06:45:31.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 Processor -- start 2024-03-24T06:45:31.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 -- start start 2024-03-24T06:45:31.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b67c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b5d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5633a4b62190 con 0x5633a4b61750 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633a4b567d0 con 0x5633a4b60e30 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.210+0000 7feaed569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5633a4b5c560 con 0x5633a4b6a330 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b5d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52178/0 (socket says 172.21.15.22:52178) 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 -- 172.21.15.22:0/1796241082 learned_addr learned my addr 172.21.15.22:0/1796241082 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feadaa00640 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b67c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 -- 172.21.15.22:0/1796241082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 msgr2=0x5633a4b5d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b5d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 -- 172.21.15.22:0/1796241082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 msgr2=0x5633a4b67c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b67c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 -- 172.21.15.22:0/1796241082 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633a4a62ae0 con 0x5633a4b60e30 2024-03-24T06:45:31.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3e00640 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b5d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.211+0000 7feae3400640 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fead4005c50 tx=0x7fead4003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d093edde015d21 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.212+0000 7fead3e00640 1 -- 172.21.15.22:0/1796241082 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead402a070 con 0x5633a4b60e30 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.212+0000 7fead3e00640 1 -- 172.21.15.22:0/1796241082 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead4009b90 con 0x5633a4b60e30 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.212+0000 7fead3e00640 1 -- 172.21.15.22:0/1796241082 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead402bd30 con 0x5633a4b60e30 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- 172.21.15.22:0/1796241082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 msgr2=0x5633a4b75bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fead4005c50 tx=0x7fead4003e30 comp rx=0 tx=0).stop 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- 172.21.15.22:0/1796241082 shutdown_connections 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b6a330 0x5633a4b5d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b61750 0x5633a4b67c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 --2- 172.21.15.22:0/1796241082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b60e30 0x5633a4b75bf0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- 172.21.15.22:0/1796241082 >> 172.21.15.22:0/1796241082 conn(0x5633a4b5b560 msgr2=0x5633a4b5c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- 172.21.15.22:0/1796241082 shutdown_connections 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- 172.21.15.22:0/1796241082 wait complete. 2024-03-24T06:45:31.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 -- start start 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.213+0000 7feaed569740 1 Processor -- start 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b6de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 0x5633a4bdfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5633a4b6ead0 con 0x5633a4b60e30 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633a4b6ec40 con 0x5633a4b6a330 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5633a4be0190 con 0x5633a4b61750 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feaed569740 10 client.0 _ll_register_callbacks cb 0x7feadc009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:31.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feadaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feae3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 0x5633a4bdfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feadaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58018/0 (socket says 172.21.15.22:58018) 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.214+0000 7feadaa00640 1 -- 172.21.15.22:0/4277523635 learned_addr learned my addr 172.21.15.22:0/4277523635 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 -- 172.21.15.22:0/4277523635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 msgr2=0x5633a4b6e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 -- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 msgr2=0x5633a4b6de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b6de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 -- 172.21.15.22:0/4277523635 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633a4ac6660 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feadaa00640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feae3e00640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 0x5633a4bdfc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feadc01da00 tx=0x7feadc01d3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557cf7e928fd42a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7fead1600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc022040 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead4004180 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7feadc003b70 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feadc003d70 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feaed569740 10 client.26630 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.215+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5633a4a89e90 con 0x5633a4b6a330 2024-03-24T06:45:31.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feadc03f080 con 0x5633a4b6a330 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7feadc0355c0 con 0x5633a4b6a330 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 10 client.26630 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 10 client.26630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 20 client.26630 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4be1a40 0x5633a4be3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 4 client.26630 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5633a47c7860 con 0x5633a4be1a40 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.216+0000 7feae3400640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4be1a40 0x5633a4be3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.217+0000 7feae3400640 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4be1a40 0x5633a4be3e60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41858d3547de3a20 server_cookie=1ada90d69b4c4059 in_seq=0 out_seq=0 2024-03-24T06:45:31.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.217+0000 7fead1600640 10 client.26630 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.217+0000 7fead1600640 1 -- 172.21.15.22:0/4277523635 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5633a47c7860 con 0x5633a4be1a40 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.217+0000 7fead1600640 10 client.26630 handle_command_reply: tid=0 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.217+0000 7feaed569740 1 client.26630 shutdown 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 msgr2=0x5633a4bdfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 0x5633a4bdfc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feadc01da00 tx=0x7feadc01d3a0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 shutdown_connections 2024-03-24T06:45:31.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5633a4be1a40 0x5633a4be3e60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633a4b6a330 0x5633a4bdfc50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5633a4b61750 0x5633a4b6e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 --2- 172.21.15.22:0/4277523635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5633a4b60e30 0x5633a4b6de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 >> 172.21.15.22:0/4277523635 conn(0x5633a4b5b560 msgr2=0x5633a4b5a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7fead1600640 0 client.26630 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 shutdown_connections 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 1 -- 172.21.15.22:0/4277523635 wait complete. 2024-03-24T06:45:31.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.218+0000 7feaed569740 20 client.26630 trim_cache size 0 max 16384 2024-03-24T06:45:31.225 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:45:31.225 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:45:31.225 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "8083c046-bd02-4ba5-974d-770d48c92e66", 2024-03-24T06:45:31.225 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:45:31.225 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.219+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 msgr2=0x7feac40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.219+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 0x7feac40a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feadc005c40 tx=0x7feadc004a40 comp rx=0 tx=0).stop 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.219+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 msgr2=0x7feae419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.219+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feacc037040 tx=0x7feacc0045b0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 shutdown_connections 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7feac40a2230 0x7feac40a46f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae410e080 0x7feae419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4100df0 0x7feae419a500 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 --2- 172.21.15.22:0/1965088570 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feae41004d0 0x7feae4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 >> 172.21.15.22:0/1965088570 conn(0x7feae4076990 msgr2=0x7feae410f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 shutdown_connections 2024-03-24T06:45:31.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.220+0000 7feaea800640 1 -- 172.21.15.22:0/1965088570 wait complete. 2024-03-24T06:45:31.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:45:31.242 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:45:31.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 -- 172.21.15.22:0/2630202349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 msgr2=0x7fd7a4073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 --2- 172.21.15.22:0/2630202349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4073ea0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd794009900 tx=0x7fd79402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 -- 172.21.15.22:0/2630202349 shutdown_connections 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 --2- 172.21.15.22:0/2630202349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a410df60 0x7fd7a4110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 --2- 172.21.15.22:0/2630202349 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a40743e0 0x7fd7a410da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 --2- 172.21.15.22:0/2630202349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4073ea0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 -- 172.21.15.22:0/2630202349 >> 172.21.15.22:0/2630202349 conn(0x7fd7a40fd1b0 msgr2=0x7fd7a40ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 -- 172.21.15.22:0/2630202349 shutdown_connections 2024-03-24T06:45:31.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.442+0000 7fd7aac00640 1 -- 172.21.15.22:0/2630202349 wait complete. 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.443+0000 7fd7aac00640 1 Processor -- start 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.443+0000 7fd7aac00640 1 -- start start 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 0x7fd7a419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 0x7fd7a419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a419ee20 con 0x7fd7a410df60 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a419ef90 con 0x7fd7a40743e0 2024-03-24T06:45:31.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7aac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a419f150 con 0x7fd7a4073ac0 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58038/0 (socket says 172.21.15.22:58038) 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 -- 172.21.15.22:0/139674432 learned_addr learned my addr 172.21.15.22:0/139674432 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3400640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 0x7fd7a419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 0x7fd7a419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 -- 172.21.15.22:0/139674432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 msgr2=0x7fd7a419a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 0x7fd7a419a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 -- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 msgr2=0x7fd7a419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 0x7fd7a419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3e00640 1 -- 172.21.15.22:0/139674432 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd794009560 con 0x7fd7a4073ac0 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a3400640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 0x7fd7a419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.444+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 0x7fd7a419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c00 tx=0x7fd79402c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7a0c00640 1 -- 172.21.15.22:0/139674432 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79402dc60 con 0x7fd7a4073ac0 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7a0c00640 1 -- 172.21.15.22:0/139674432 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79402ddc0 con 0x7fd7a4073ac0 2024-03-24T06:45:31.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7a0c00640 1 -- 172.21.15.22:0/139674432 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794003bf0 con 0x7fd7a4073ac0 2024-03-24T06:45:31.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a419f3d0 con 0x7fd7a4073ac0 2024-03-24T06:45:31.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.445+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a6cb0 con 0x7fd7a4073ac0 2024-03-24T06:45:31.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.448+0000 7fd7a0c00640 1 -- 172.21.15.22:0/139674432 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd794003d50 con 0x7fd7a4073ac0 2024-03-24T06:45:31.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 Processor -- start 2024-03-24T06:45:31.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 -- start start 2024-03-24T06:45:31.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc1d80 0x556f01bc2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f01b30790 con 0x556f01bc2a60 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f01b2dcc0 con 0x556f01bcb0f0 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7ad982740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f01b294a0 con 0x556f01bc1d80 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7a8c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc1d80 0x556f01bc2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.449+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52220/0 (socket says 172.21.15.22:52220) 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 -- 172.21.15.22:0/781356801 learned_addr learned my addr 172.21.15.22:0/781356801 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a0c00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 0x7fd7880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 -- 172.21.15.22:0/781356801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc1d80 msgr2=0x556f01bc2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc1d80 0x556f01bc2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 -- 172.21.15.22:0/781356801 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 msgr2=0x556f01bc17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 -- 172.21.15.22:0/781356801 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f01aeac20 con 0x556f01bcb0f0 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a0c00640 1 -- 172.21.15.22:0/139674432 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd7940f09e0 con 0x7fd7a4073ac0 2024-03-24T06:45:31.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 0x7fd7880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.450+0000 7fd7a3400640 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd79000bd30 tx=0x7fd79000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8634b1cbc2395fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd799200640 1 -- 172.21.15.22:0/781356801 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79001a660 con 0x556f01bcb0f0 2024-03-24T06:45:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd799200640 1 -- 172.21.15.22:0/781356801 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79000be20 con 0x556f01bcb0f0 2024-03-24T06:45:31.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd799200640 1 -- 172.21.15.22:0/781356801 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd790011b70 con 0x556f01bcb0f0 2024-03-24T06:45:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 -- 172.21.15.22:0/781356801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 msgr2=0x556f01bcd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 0x7fd7880a4770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005fd0 tx=0x7fd79c004a40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd79000bd30 tx=0x7fd79000f730 comp rx=0 tx=0).stop 2024-03-24T06:45:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 -- 172.21.15.22:0/781356801 shutdown_connections 2024-03-24T06:45:31.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01bcd4e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc1d80 0x556f01bc2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 --2- 172.21.15.22:0/781356801 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.451+0000 7fd7ad982740 1 -- 172.21.15.22:0/781356801 >> 172.21.15.22:0/781356801 conn(0x556f01b88b30 msgr2=0x556f01bbc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.452+0000 7fd7ad982740 1 -- 172.21.15.22:0/781356801 shutdown_connections 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.452+0000 7fd7ad982740 1 -- 172.21.15.22:0/781356801 wait complete. 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.452+0000 7fd7ad982740 1 -- start start 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.452+0000 7fd7ad982740 1 Processor -- start 2024-03-24T06:45:31.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 0x556f01bc8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f01acdf20 con 0x556f01bcb0f0 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f01acdbc0 con 0x556f01bc1d80 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f018971c0 con 0x556f01bc2a60 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7ad982740 10 client.0 _ll_register_callbacks cb 0x556f01839be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:52234/0 (socket says 172.21.15.22:52234) 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7a3e00640 1 -- 172.21.15.22:0/1841842657 learned_addr learned my addr 172.21.15.22:0/1841842657 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.453+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 0x556f01bc8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 -- 172.21.15.22:0/1841842657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 msgr2=0x556f01bc85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 -- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 msgr2=0x556f01bc8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 0x556f01bc8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 -- 172.21.15.22:0/1841842657 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79000b960 con 0x556f01bc1d80 2024-03-24T06:45:31.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 0x556f01bc8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd794005ec0 tx=0x7fd7940b93c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=688654f5744fe2ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd78d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7940ee1f0 con 0x556f01bc1d80 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd7940bd820 con 0x556f01bc1d80 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f01b21160 con 0x556f01bc1d80 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd7940bca80 con 0x556f01bc1d80 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7ad982740 10 client.26642 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.454+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556f01b27b20 con 0x556f01bc1d80 2024-03-24T06:45:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556f01b21160 con 0x556f01bc1d80 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fd79403c070 con 0x556f01bc1d80 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 10 client.26642 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 10 client.26642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 20 client.26642 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01bdb470 0x556f01bdd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 4 client.26642 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f018287e0 con 0x556f01bdb470 2024-03-24T06:45:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.455+0000 7fd7a3400640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01bdb470 0x556f01bdd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.456+0000 7fd7a3400640 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01bdb470 0x556f01bdd860 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd705b1d411d3e89 server_cookie=8aec860730f6ad11 in_seq=0 out_seq=0 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.456+0000 7fd78d600640 10 client.26642 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd78d600640 1 -- 172.21.15.22:0/1841842657 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f018287e0 con 0x556f01bdb470 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd78d600640 10 client.26642 handle_command_reply: tid=0 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 client.26642 shutdown 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 msgr2=0x556f01bc80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd794005ec0 tx=0x7fd7940b93c0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 shutdown_connections 2024-03-24T06:45:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bcb0f0 0x556f01bc8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01bdb470 0x556f01bdd860 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 --2- 172.21.15.22:0/1841842657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bc80a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd78d600640 0 client.26642 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 >> 172.21.15.22:0/1841842657 conn(0x556f01b88b30 msgr2=0x556f01bbe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 shutdown_connections 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.457+0000 7fd7ad982740 1 -- 172.21.15.22:0/1841842657 wait complete. 2024-03-24T06:45:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.458+0000 7fd7ad982740 20 client.26642 trim_cache size 0 max 16384 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 Processor -- start 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 -- start start 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f01bcdb00 con 0x556f01bc2a60 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f01bcde40 con 0x556f01bc1d80 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7ad982740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f018287e0 con 0x556f01bcb0f0 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7a8c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7a8c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58054/0 (socket says 172.21.15.22:58054) 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7a8c00640 1 -- 172.21.15.22:0/2020698875 learned_addr learned my addr 172.21.15.22:0/2020698875 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.477+0000 7fd7a3400640 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 -- 172.21.15.22:0/2020698875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 msgr2=0x556f01bca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 -- 172.21.15.22:0/2020698875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 msgr2=0x556f01bc7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 -- 172.21.15.22:0/2020698875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f01ac39a0 con 0x556f01bcb0f0 2024-03-24T06:45:31.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd79c010700 tx=0x7fd79c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83da47b84fce35fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd78fe00640 1 -- 172.21.15.22:0/2020698875 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c011940 con 0x556f01bcb0f0 2024-03-24T06:45:31.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd78fe00640 1 -- 172.21.15.22:0/2020698875 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c01f420 con 0x556f01bcb0f0 2024-03-24T06:45:31.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.478+0000 7fd78fe00640 1 -- 172.21.15.22:0/2020698875 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c01e680 con 0x556f01bcb0f0 2024-03-24T06:45:31.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 -- 172.21.15.22:0/2020698875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 msgr2=0x556f01bbc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd79c010700 tx=0x7fd79c016000 comp rx=0 tx=0).stop 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 -- 172.21.15.22:0/2020698875 shutdown_connections 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bcb0f0 0x556f01bbc300 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc2a60 0x556f01bc7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 --2- 172.21.15.22:0/2020698875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bc1d80 0x556f01bca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 -- 172.21.15.22:0/2020698875 >> 172.21.15.22:0/2020698875 conn(0x556f01b88b30 msgr2=0x556f01bbe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 -- 172.21.15.22:0/2020698875 shutdown_connections 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.479+0000 7fd7ad982740 1 -- 172.21.15.22:0/2020698875 wait complete. 2024-03-24T06:45:31.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.480+0000 7fd7ad982740 1 -- start start 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.480+0000 7fd7ad982740 1 Processor -- start 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc1d80 0x556f01bc54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01c42f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f01c434a0 con 0x556f01bc1d80 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f01c435e0 con 0x556f01bcb0f0 2024-03-24T06:45:31.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f01c437a0 con 0x556f01bc2a60 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7ad982740 10 client.0 _ll_register_callbacks cb 0x7fd7940035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7a3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc1d80 0x556f01bc54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:58066/0 (socket says 172.21.15.22:58066) 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7a3400640 1 -- 172.21.15.22:0/508309824 learned_addr learned my addr 172.21.15.22:0/508309824 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.481+0000 7fd7a8c00640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01c42f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 -- 172.21.15.22:0/508309824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 msgr2=0x556f01c42f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01c42f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 -- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc1d80 msgr2=0x556f01bc54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc1d80 0x556f01bc54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 -- 172.21.15.22:0/508309824 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c013170 con 0x556f01bc2a60 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7a3400640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd79000bd30 tx=0x7fd79000fd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23a57229f73f2601 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd78d600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79000b410 con 0x556f01bc2a60 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f01bbc960 con 0x556f01bc2a60 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.482+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fd790004230 con 0x556f01bc2a60 2024-03-24T06:45:31.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd79000b7e0 con 0x556f01bc2a60 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 10 client.26674 fetch_fsmap learned FSMap version 36 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556f01ae9480 con 0x556f01bc2a60 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556f01bbc960 con 0x556f01bc2a60 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fd7900054d0 con 0x556f01bc2a60 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 10 client.26674 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 10 client.26674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 20 client.26674 populate_metadata read hostname 'smithi022' 2024-03-24T06:45:31.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01c4b010 0x556f01c4d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 4 client.26674 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fd79000be20 con 0x556f01c4b010 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.483+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01c4b010 0x556f01c4d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd7a3e00640 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01c4b010 0x556f01c4d430 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9b9373ff6fcd43 server_cookie=9706eb2b3259a389 in_seq=0 out_seq=0 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd78d600640 10 client.26674 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd78d600640 1 -- 172.21.15.22:0/508309824 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fd79000be20 con 0x556f01c4b010 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd78d600640 10 client.26674 handle_command_reply: tid=0 2024-03-24T06:45:31.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd7ad982740 1 client.26674 shutdown 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 msgr2=0x556f01bc5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.484+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd79000bd30 tx=0x7fd79000fd70 comp rx=0 tx=0).stop 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 shutdown_connections 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f01c4b010 0x556f01c4d430 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f01bcb0f0 0x556f01c42f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f01bc2a60 0x556f01bc5a00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 --2- 172.21.15.22:0/508309824 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f01bc1d80 0x556f01bc54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 >> 172.21.15.22:0/508309824 conn(0x556f01b88b30 msgr2=0x556f01bbdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd78d600640 0 client.26674 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 shutdown_connections 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 1 -- 172.21.15.22:0/508309824 wait complete. 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7ad982740 20 client.26674 trim_cache size 0 max 16384 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:45:31.492 INFO:teuthology.orchestra.run.smithi022.stdout: "8083c046-bd02-4ba5-974d-770d48c92e66": { 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "8083c046-bd02-4ba5-974d-770d48c92e66", 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:45:31.493 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.485+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 msgr2=0x7fd7880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 0x7fd7880a4770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd79c005fd0 tx=0x7fd79c004a40 comp rx=0 tx=0).stop 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 msgr2=0x7fd7a4199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c00 tx=0x7fd79402c740 comp rx=0 tx=0).stop 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 shutdown_connections 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd7880a22b0 0x7fd7880a4770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd7a410df60 0x7fd7a419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7a40743e0 0x7fd7a419a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 --2- 172.21.15.22:0/139674432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7a4073ac0 0x7fd7a4199fb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:45:31.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 >> 172.21.15.22:0/139674432 conn(0x7fd7a40fd1b0 msgr2=0x7fd7a40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:45:31.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 shutdown_connections 2024-03-24T06:45:31.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:45:31.486+0000 7fd7aac00640 1 -- 172.21.15.22:0/139674432 wait complete. 2024-03-24T06:45:31.503 INFO:tasks.cephfs.filesystem:scrub status for tag:8083c046-bd02-4ba5-974d-770d48c92e66 - {'path': '/', 'tag': '8083c046-bd02-4ba5-974d-770d48c92e66', 'options': 'recursive,force'} 2024-03-24T06:45:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: pgmap v170: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-03-24T06:45:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:45:32.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:32 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: pgmap v170: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-03-24T06:45:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:45:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:45:32.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:32 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: pgmap v170: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 MiB/s wr, 4.30k op/s 2024-03-24T06:45:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:45:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:32.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:32 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:33.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:33 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:33 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:33 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:45:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:34 smithi143 ceph-mon[36520]: pgmap v171: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 3.08k op/s 2024-03-24T06:45:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:34 smithi155 ceph-mon[36262]: pgmap v171: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 3.08k op/s 2024-03-24T06:45:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:34 smithi022 ceph-mon[32003]: pgmap v171: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 3.08k op/s 2024-03-24T06:45:36.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:36 smithi022 ceph-mon[32003]: pgmap v172: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 MiB/s wr, 2.38k op/s 2024-03-24T06:45:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:36 smithi155 ceph-mon[36262]: pgmap v172: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 MiB/s wr, 2.38k op/s 2024-03-24T06:45:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:36 smithi143 ceph-mon[36520]: pgmap v172: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 MiB/s wr, 2.38k op/s 2024-03-24T06:45:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:38 smithi155 ceph-mon[36262]: pgmap v173: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 1.99k op/s 2024-03-24T06:45:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:38 smithi022 ceph-mon[32003]: pgmap v173: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 1.99k op/s 2024-03-24T06:45:38.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:38 smithi143 ceph-mon[36520]: pgmap v173: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.4 MiB/s wr, 1.99k op/s 2024-03-24T06:45:40.452 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:40 smithi022 ceph-mon[32003]: pgmap v174: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 812 op/s 2024-03-24T06:45:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:40 smithi155 ceph-mon[36262]: pgmap v174: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 812 op/s 2024-03-24T06:45:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:40 smithi143 ceph-mon[36520]: pgmap v174: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 812 op/s 2024-03-24T06:45:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:42 smithi155 ceph-mon[36262]: pgmap v175: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.4 MiB/s wr, 814 op/s 2024-03-24T06:45:42.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:42 smithi022 ceph-mon[32003]: pgmap v175: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.4 MiB/s wr, 814 op/s 2024-03-24T06:45:42.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:42 smithi143 ceph-mon[36520]: pgmap v175: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.4 MiB/s wr, 814 op/s 2024-03-24T06:45:43.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:43.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:43.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:43.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:44 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:45:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:44 smithi155 ceph-mon[36262]: pgmap v176: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.5 MiB/s wr, 371 op/s 2024-03-24T06:45:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:44 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:45:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:44 smithi022 ceph-mon[32003]: pgmap v176: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.5 MiB/s wr, 371 op/s 2024-03-24T06:45:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:44 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:45:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:44 smithi143 ceph-mon[36520]: pgmap v176: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 1.5 MiB/s wr, 371 op/s 2024-03-24T06:45:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:46 smithi155 ceph-mon[36262]: pgmap v177: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 308 KiB/s wr, 80 op/s 2024-03-24T06:45:46.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:46 smithi022 ceph-mon[32003]: pgmap v177: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 308 KiB/s wr, 80 op/s 2024-03-24T06:45:46.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:46 smithi143 ceph-mon[36520]: pgmap v177: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 308 KiB/s wr, 80 op/s 2024-03-24T06:45:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:48 smithi022 ceph-mon[32003]: pgmap v178: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-24T06:45:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:48 smithi155 ceph-mon[36262]: pgmap v178: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-24T06:45:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:48 smithi143 ceph-mon[36520]: pgmap v178: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-03-24T06:45:50.479 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:50 smithi143 ceph-mon[36520]: pgmap v179: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T06:45:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:50 smithi155 ceph-mon[36262]: pgmap v179: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T06:45:50.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:50 smithi022 ceph-mon[32003]: pgmap v179: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-03-24T06:45:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:52 smithi155 ceph-mon[36262]: pgmap v180: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-24T06:45:52.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:52 smithi022 ceph-mon[32003]: pgmap v180: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-24T06:45:52.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:52 smithi143 ceph-mon[36520]: pgmap v180: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 14 op/s 2024-03-24T06:45:54.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:54 smithi155 ceph-mon[36262]: pgmap v181: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T06:45:54.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:54 smithi022 ceph-mon[32003]: pgmap v181: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T06:45:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:54 smithi143 ceph-mon[36520]: pgmap v181: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 16 op/s 2024-03-24T06:45:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:56 smithi155 ceph-mon[36262]: pgmap v182: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-24T06:45:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:56 smithi022 ceph-mon[32003]: pgmap v182: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-24T06:45:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:56 smithi143 ceph-mon[36520]: pgmap v182: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-24T06:45:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:45:58 smithi155 ceph-mon[36262]: pgmap v183: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 13 op/s 2024-03-24T06:45:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:45:58 smithi022 ceph-mon[32003]: pgmap v183: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 13 op/s 2024-03-24T06:45:58.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:45:58 smithi143 ceph-mon[36520]: pgmap v183: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 13 op/s 2024-03-24T06:46:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:00 smithi155 ceph-mon[36262]: pgmap v184: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-24T06:46:00.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:00 smithi022 ceph-mon[32003]: pgmap v184: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-24T06:46:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:00 smithi143 ceph-mon[36520]: pgmap v184: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-03-24T06:46:01.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:46:01.505 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:46:01.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 -- 172.21.15.22:0/1434107504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101570 msgr2=0x7f6f9c1019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1434107504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101570 0x7f6f9c1019f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009930 tx=0x7f6f8802bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:01.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 -- 172.21.15.22:0/1434107504 shutdown_connections 2024-03-24T06:46:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1434107504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101f30 0x7f6f9c10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1434107504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101570 0x7f6f9c1019f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1434107504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 -- 172.21.15.22:0/1434107504 >> 172.21.15.22:0/1434107504 conn(0x7f6f9c0fd080 msgr2=0x7f6f9c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 -- 172.21.15.22:0/1434107504 shutdown_connections 2024-03-24T06:46:01.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.741+0000 7f6fa1600640 1 -- 172.21.15.22:0/1434107504 wait complete. 2024-03-24T06:46:01.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.743+0000 7f6fa1600640 1 Processor -- start 2024-03-24T06:46:01.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.743+0000 7f6fa1600640 1 -- start start 2024-03-24T06:46:01.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 0x7f6f9c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c198010 con 0x7f6f9c101570 2024-03-24T06:46:01.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c198180 con 0x7f6f9c107630 2024-03-24T06:46:01.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa1600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c198340 con 0x7f6f9c101f30 2024-03-24T06:46:01.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa0c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 0x7f6f9c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38672/0 (socket says 172.21.15.22:38672) 2024-03-24T06:46:01.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6f9be00640 1 -- 172.21.15.22:0/1764774164 learned_addr learned my addr 172.21.15.22:0/1764774164 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:01.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa0c00640 1 -- 172.21.15.22:0/1764774164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 msgr2=0x7f6f9c19dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.744+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 0x7f6f9c19dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa0c00640 1 -- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 msgr2=0x7f6f9c19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa0c00640 1 -- 172.21.15.22:0/1764774164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c009490 con 0x7f6f9c107630 2024-03-24T06:46:01.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6f9be00640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:01.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6f9b400640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 0x7f6f9c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c1979c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009fd0 tx=0x7f6f9400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6f98c00640 1 -- 172.21.15.22:0/1764774164 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400ce20 con 0x7f6f9c107630 2024-03-24T06:46:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6f98c00640 1 -- 172.21.15.22:0/1764774164 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f94005020 con 0x7f6f9c107630 2024-03-24T06:46:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6f98c00640 1 -- 172.21.15.22:0/1764774164 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94010690 con 0x7f6f9c107630 2024-03-24T06:46:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f88009560 con 0x7f6f9c107630 2024-03-24T06:46:01.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.745+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c1a46d0 con 0x7f6f9c107630 2024-03-24T06:46:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.748+0000 7f6f98c00640 1 -- 172.21.15.22:0/1764774164 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f6f94005500 con 0x7f6f9c107630 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 Processor -- start 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 -- start start 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5347fc40 0x562d5347e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 0x562d5348ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562d533ed790 con 0x562d5348c6e0 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d533eacc0 con 0x562d5347fc40 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.749+0000 7f6fa598a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562d533e64a0 con 0x562d5347ef60 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5347fc40 0x562d5347e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38198/0 (socket says 172.21.15.22:38198) 2024-03-24T06:46:01.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 -- 172.21.15.22:0/2066968048 learned_addr learned my addr 172.21.15.22:0/2066968048 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:01.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9b400640 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 0x562d5348ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 -- 172.21.15.22:0/2066968048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5347fc40 msgr2=0x562d5347e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5347fc40 0x562d5347e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 -- 172.21.15.22:0/2066968048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 msgr2=0x562d5348ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 0x562d5348ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 -- 172.21.15.22:0/2066968048 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d533a7c20 con 0x562d5347ef60 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9b400640 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 0x562d5348ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.750+0000 7f6f9be00640 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c011760 tx=0x7f6f8c011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a16e89123c729a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f91200640 1 -- 172.21.15.22:0/2066968048 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00fba0 con 0x562d5347ef60 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f98c00640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 0x7f6f800a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f91200640 1 -- 172.21.15.22:0/2066968048 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f8c005500 con 0x562d5347ef60 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f91200640 1 -- 172.21.15.22:0/2066968048 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c019430 con 0x562d5347ef60 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f98c00640 1 -- 172.21.15.22:0/1764774164 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6f94014070 con 0x7f6f9c107630 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.751+0000 7f6f9b400640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 0x7f6f800a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- 172.21.15.22:0/2066968048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 msgr2=0x562d5347f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c011760 tx=0x7f6f8c011c20 comp rx=0 tx=0).stop 2024-03-24T06:46:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- 172.21.15.22:0/2066968048 shutdown_connections 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5348c6e0 0x562d5348ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5347f3e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 --2- 172.21.15.22:0/2066968048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5347fc40 0x562d5347e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- 172.21.15.22:0/2066968048 >> 172.21.15.22:0/2066968048 conn(0x562d53445b30 msgr2=0x562d534799b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- 172.21.15.22:0/2066968048 shutdown_connections 2024-03-24T06:46:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6f9b400640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 0x7f6f800a4770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009900 tx=0x7f6f88005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- 172.21.15.22:0/2066968048 wait complete. 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.752+0000 7f6fa598a740 1 -- start start 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 Processor -- start 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53489b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d5348a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562d5338af20 con 0x562d5347fc40 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d5338abc0 con 0x562d5348c6e0 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562d531541c0 con 0x562d5347ef60 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa598a740 10 client.0 _ll_register_callbacks cb 0x562d530f6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53489b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38202/0 (socket says 172.21.15.22:38202) 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6f9be00640 1 -- 172.21.15.22:0/734969790 learned_addr learned my addr 172.21.15.22:0/734969790 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.753+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d5348a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 -- 172.21.15.22:0/734969790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 msgr2=0x562d5348a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d5348a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 -- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 msgr2=0x562d53489b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53489b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 -- 172.21.15.22:0/734969790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c011440 con 0x562d5347ef60 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d5348a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f9be00640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c01b040 tx=0x7f6f8c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec1848882c26539f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.754+0000 7f6f85600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c004840 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d533de160 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6f8c004490 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6f8c005670 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6fa598a740 10 client.26686 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562d533e4b20 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.755+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562d533de160 con 0x562d5347ef60 2024-03-24T06:46:01.770 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f6f8c003bf0 con 0x562d5347ef60 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 10 client.26686 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 10 client.26686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 20 client.26686 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d5349cae0 0x562d5349ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 4 client.26686 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d530e57e0 con 0x562d5349cae0 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.756+0000 7f6f9b400640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d5349cae0 0x562d5349ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.757+0000 7f6f9b400640 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d5349cae0 0x562d5349ef10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd30dcf8b138aac server_cookie=9b97d70bacf1b89d in_seq=0 out_seq=0 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.757+0000 7f6f85600640 10 client.26686 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.757+0000 7f6f85600640 1 -- 172.21.15.22:0/734969790 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d530e57e0 con 0x562d5349cae0 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.757+0000 7f6f85600640 10 client.26686 handle_command_reply: tid=0 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.757+0000 7f6fa598a740 1 client.26686 shutdown 2024-03-24T06:46:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 msgr2=0x562d53489630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c01b040 tx=0x7f6f8c004270 comp rx=0 tx=0).stop 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 shutdown_connections 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d5349cae0 0x562d5349ef10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d5348a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53489b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6f85600640 0 client.26686 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 --2- 172.21.15.22:0/734969790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d53489630 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 >> 172.21.15.22:0/734969790 conn(0x562d53445b30 msgr2=0x562d5347b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 shutdown_connections 2024-03-24T06:46:01.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 1 -- 172.21.15.22:0/734969790 wait complete. 2024-03-24T06:46:01.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.758+0000 7f6fa598a740 20 client.26686 trim_cache size 0 max 16384 2024-03-24T06:46:01.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.777+0000 7f6fa598a740 1 Processor -- start 2024-03-24T06:46:01.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.777+0000 7f6fa598a740 1 -- start start 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53498060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d534a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562d5347e1e0 con 0x562d5347fc40 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d5347a9b0 con 0x562d5348c6e0 2024-03-24T06:46:01.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa598a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562d534755f0 con 0x562d5347ef60 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6f9b400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53498060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6fa0c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d534a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38216/0 (socket says 172.21.15.22:38216) 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.778+0000 7f6f9be00640 1 -- 172.21.15.22:0/1379728357 learned_addr learned my addr 172.21.15.22:0/1379728357 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 -- 172.21.15.22:0/1379728357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 msgr2=0x562d534a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d534a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 -- 172.21.15.22:0/1379728357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 msgr2=0x562d53498060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53498060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 -- 172.21.15.22:0/1379728357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5348f070 con 0x562d5347ef60 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f9be00640 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c032040 tx=0x7f6f8c001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243231002599699 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f87e00640 1 -- 172.21.15.22:0/1379728357 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00fdb0 con 0x562d5347ef60 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f87e00640 1 -- 172.21.15.22:0/1379728357 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f8c005bf0 con 0x562d5347ef60 2024-03-24T06:46:01.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.779+0000 7f6f87e00640 1 -- 172.21.15.22:0/1379728357 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0042e0 con 0x562d5347ef60 2024-03-24T06:46:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 -- 172.21.15.22:0/1379728357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 msgr2=0x562d5348bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c032040 tx=0x7f6f8c001e50 comp rx=0 tx=0).stop 2024-03-24T06:46:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 -- 172.21.15.22:0/1379728357 shutdown_connections 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d534a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53498060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 --2- 172.21.15.22:0/1379728357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d5348bd60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 -- 172.21.15.22:0/1379728357 >> 172.21.15.22:0/1379728357 conn(0x562d53445b30 msgr2=0x562d534794e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 -- 172.21.15.22:0/1379728357 shutdown_connections 2024-03-24T06:46:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.780+0000 7f6fa598a740 1 -- 172.21.15.22:0/1379728357 wait complete. 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.781+0000 7f6fa598a740 1 -- start start 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.781+0000 7f6fa598a740 1 Processor -- start 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53486700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d53486c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562d53494eb0 con 0x562d5347fc40 2024-03-24T06:46:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d53501e20 con 0x562d5348c6e0 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562d53501f90 con 0x562d5347ef60 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa598a740 10 client.0 _ll_register_callbacks cb 0x7f6f8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38226/0 (socket says 172.21.15.22:38226) 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6f9be00640 1 -- 172.21.15.22:0/4101134650 learned_addr learned my addr 172.21.15.22:0/4101134650 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:01.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6f9b400640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53486700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.782+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d53486c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 -- 172.21.15.22:0/4101134650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 msgr2=0x562d53486c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d53486c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 -- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 msgr2=0x562d53486700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53486700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 -- 172.21.15.22:0/4101134650 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d533e4b20 con 0x562d5347ef60 2024-03-24T06:46:01.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9b400640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53486700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6fa0c00640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d53486c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f9be00640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c005da0 tx=0x7f6f8c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eed1ea7473af44b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f85600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c024820 con 0x562d5347ef60 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f6f8c00fdb0 con 0x562d5347ef60 2024-03-24T06:46:01.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.783+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f8c005500 con 0x562d5347ef60 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6f8c024a40 con 0x562d5347ef60 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 10 client.26692 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x562d533809a0 con 0x562d5347ef60 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f8c005500 con 0x562d5347ef60 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f6f8c004250 con 0x562d5347ef60 2024-03-24T06:46:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 10 client.26692 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 10 client.26692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 20 client.26692 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d53503310 0x562d53505730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 4 client.26692 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562d53122720 con 0x562d53503310 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.784+0000 7f6f9b400640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d53503310 0x562d53505730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6f9b400640 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d53503310 0x562d53505730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=600f6838a6cab49c server_cookie=bd7fe34b384a25e5 in_seq=0 out_seq=0 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6f85600640 10 client.26692 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6f85600640 1 -- 172.21.15.22:0/4101134650 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562d53122720 con 0x562d53503310 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6f85600640 10 client.26692 handle_command_reply: tid=0 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6fa598a740 1 client.26692 shutdown 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 msgr2=0x562d534861c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.785+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c005da0 tx=0x7f6f8c009f60 comp rx=0 tx=0).stop 2024-03-24T06:46:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 shutdown_connections 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562d53503310 0x562d53505730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d5348c6e0 0x562d53486c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562d5347fc40 0x562d53486700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 --2- 172.21.15.22:0/4101134650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562d5347ef60 0x562d534861c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 >> 172.21.15.22:0/4101134650 conn(0x562d53445b30 msgr2=0x562d53478340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6f85600640 0 client.26692 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 shutdown_connections 2024-03-24T06:46:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 1 -- 172.21.15.22:0/4101134650 wait complete. 2024-03-24T06:46:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.786+0000 7f6fa598a740 20 client.26692 trim_cache size 0 max 16384 2024-03-24T06:46:01.798 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:01.799 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:46:01.799 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:46:01.799 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 msgr2=0x7f6f800a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 0x7f6f800a4770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009900 tx=0x7f6f88005a20 comp rx=0 tx=0).stop 2024-03-24T06:46:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 msgr2=0x7f6f9c1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c1979c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009fd0 tx=0x7f6f9400ef10 comp rx=0 tx=0).stop 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 shutdown_connections 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f800a22b0 0x7f6f800a4770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f9c107630 0x7f6f9c1979c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f9c101f30 0x7f6f9c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 --2- 172.21.15.22:0/1764774164 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6f9c101570 0x7f6f9c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 >> 172.21.15.22:0/1764774164 conn(0x7f6f9c0fd080 msgr2=0x7f6f9c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:01.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 shutdown_connections 2024-03-24T06:46:01.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:01.787+0000 7f6fa1600640 1 -- 172.21.15.22:0/1764774164 wait complete. 2024-03-24T06:46:01.812 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:46:01.812 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:46:02.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.046+0000 7f9858e00640 1 -- 172.21.15.22:0/2087617726 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 msgr2=0x7f985410e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.046+0000 7f9858e00640 1 --2- 172.21.15.22:0/2087617726 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985410e380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f984c009ca0 tx=0x7f984c02c100 comp rx=0 tx=0).stop 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 -- 172.21.15.22:0/2087617726 shutdown_connections 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 --2- 172.21.15.22:0/2087617726 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985410e380 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 --2- 172.21.15.22:0/2087617726 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854068870 0x7f9854068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 --2- 172.21.15.22:0/2087617726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854107600 0x7f98541079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 -- 172.21.15.22:0/2087617726 >> 172.21.15.22:0/2087617726 conn(0x7f9854076020 msgr2=0x7f9854076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 -- 172.21.15.22:0/2087617726 shutdown_connections 2024-03-24T06:46:02.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.047+0000 7f9858e00640 1 -- 172.21.15.22:0/2087617726 wait complete. 2024-03-24T06:46:02.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.048+0000 7f9858e00640 1 Processor -- start 2024-03-24T06:46:02.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.048+0000 7f9858e00640 1 -- start start 2024-03-24T06:46:02.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985419fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f985419a0f0 con 0x7f9854107600 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f985419a260 con 0x7f9854068870 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9858e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f985419a420 con 0x7f9854069230 2024-03-24T06:46:02.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9853e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9853400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f984aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985419fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9853e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38742/0 (socket says 172.21.15.22:38742) 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9853400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48432/0 (socket says 172.21.15.22:48432) 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f984aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985419fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38256/0 (socket says 172.21.15.22:38256) 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.049+0000 7f9853e00640 1 -- 172.21.15.22:0/4132454700 learned_addr learned my addr 172.21.15.22:0/4132454700 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 -- 172.21.15.22:0/4132454700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 msgr2=0x7f985419fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985419fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 -- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 msgr2=0x7f9854199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 -- 172.21.15.22:0/4132454700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9844009490 con 0x7f9854068870 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9853400640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f984400e9f0 tx=0x7f984400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984400cd80 con 0x7f9854068870 2024-03-24T06:46:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98440035f0 con 0x7f9854068870 2024-03-24T06:46:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984c009900 con 0x7f9854068870 2024-03-24T06:46:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844010690 con 0x7f9854068870 2024-03-24T06:46:02.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.050+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985419a8f0 con 0x7f9854068870 2024-03-24T06:46:02.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.051+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9854068420 con 0x7f9854068870 2024-03-24T06:46:02.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.053+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f9844003af0 con 0x7f9854068870 2024-03-24T06:46:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.055+0000 7f9850c00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 0x7f98380a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.055+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f9844014070 con 0x7f9854068870 2024-03-24T06:46:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.055+0000 7f984aa00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 0x7f98380a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.056+0000 7f984aa00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 0x7f98380a4770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98541026d0 tx=0x7f983c005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.059+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f98440970a0 con 0x7f9854068870 2024-03-24T06:46:02.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.303+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f985419afd0 con 0x7f9854068870 2024-03-24T06:46:02.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.304+0000 7f9850c00640 1 -- 172.21.15.22:0/4132454700 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11940 (secure 0 0 0) 0x7f984409a760 con 0x7f9854068870 2024-03-24T06:46:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:46:02.305 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:45:28.085844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24584,24601,24532,24544,24565]},"id":1}]} 2024-03-24T06:46:02.305 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 36 2024-03-24T06:46:02.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 msgr2=0x7f98380a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 0x7f98380a4770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f98541026d0 tx=0x7f983c005f90 comp rx=0 tx=0).stop 2024-03-24T06:46:02.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 msgr2=0x7f985419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f984400e9f0 tx=0x7f984400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 shutdown_connections 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f98380a22b0 0x7f98380a4770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9854107600 0x7f9854199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9854069230 0x7f985419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 --2- 172.21.15.22:0/4132454700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9854068870 0x7f985419f8f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 >> 172.21.15.22:0/4132454700 conn(0x7f9854076020 msgr2=0x7f985410c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 shutdown_connections 2024-03-24T06:46:02.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.308+0000 7f9858e00640 1 -- 172.21.15.22:0/4132454700 wait complete. 2024-03-24T06:46:02.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:46:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:02 smithi155 ceph-mon[36262]: pgmap v185: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-03-24T06:46:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.588+0000 7f132b200640 1 -- 172.21.15.22:0/300354067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c1015a0 msgr2=0x7f132c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.588+0000 7f132b200640 1 --2- 172.21.15.22:0/300354067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c1015a0 0x7f132c101a20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1318009930 tx=0x7f131802bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 -- 172.21.15.22:0/300354067 shutdown_connections 2024-03-24T06:46:02.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 --2- 172.21.15.22:0/300354067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c101f60 0x7f132c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 --2- 172.21.15.22:0/300354067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c1015a0 0x7f132c101a20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 --2- 172.21.15.22:0/300354067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 0x7f132c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 -- 172.21.15.22:0/300354067 >> 172.21.15.22:0/300354067 conn(0x7f132c0fd0b0 msgr2=0x7f132c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 -- 172.21.15.22:0/300354067 shutdown_connections 2024-03-24T06:46:02.590 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.589+0000 7f132b200640 1 -- 172.21.15.22:0/300354067 wait complete. 2024-03-24T06:46:02.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.590+0000 7f132b200640 1 Processor -- start 2024-03-24T06:46:02.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 -- start start 2024-03-24T06:46:02.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 0x7f132c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f132c19ec40 con 0x7f132c107660 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f132c19edb0 con 0x7f132c1015a0 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f132c19ef70 con 0x7f132c101f60 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f1329400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f1329e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f1329400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38274/0 (socket says 172.21.15.22:38274) 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f1329400640 1 -- 172.21.15.22:0/2125255240 learned_addr learned my addr 172.21.15.22:0/2125255240 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f1329e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48456/0 (socket says 172.21.15.22:48456) 2024-03-24T06:46:02.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.591+0000 7f132a800640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 0x7f132c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 -- 172.21.15.22:0/2125255240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 msgr2=0x7f132c199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 -- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 msgr2=0x7f132c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 0x7f132c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 -- 172.21.15.22:0/2125255240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1324009840 con 0x7f132c101f60 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329e00640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.592+0000 7f1329400640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1318000c00 tx=0x7f131802c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.593+0000 7f1322a00640 1 -- 172.21.15.22:0/2125255240 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180042d0 con 0x7f132c101f60 2024-03-24T06:46:02.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.593+0000 7f1322a00640 1 -- 172.21.15.22:0/2125255240 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1318004430 con 0x7f132c101f60 2024-03-24T06:46:02.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.593+0000 7f1322a00640 1 -- 172.21.15.22:0/2125255240 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318003b50 con 0x7f132c101f60 2024-03-24T06:46:02.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.593+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1318009560 con 0x7f132c101f60 2024-03-24T06:46:02.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.593+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1a68f0 con 0x7f132c101f60 2024-03-24T06:46:02.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.596+0000 7f1333776740 1 Processor -- start 2024-03-24T06:46:02.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.596+0000 7f1333776740 1 -- start start 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.596+0000 7f1322a00640 1 -- 172.21.15.22:0/2125255240 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1318005460 con 0x7f132c101f60 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 0x558decef5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decef7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558dece5a6e0 con 0x558deceecd70 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558dece57c10 con 0x558deceec450 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1333776740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558dece533f0 con 0x558decef5950 2024-03-24T06:46:02.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48460/0 (socket says 172.21.15.22:48460) 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 -- 172.21.15.22:0/2376633272 learned_addr learned my addr 172.21.15.22:0/2376633272 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1329400640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decef7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1329e00640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 0x558decef5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 -- 172.21.15.22:0/2376633272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 msgr2=0x558decef7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decef7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 -- 172.21.15.22:0/2376633272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 msgr2=0x558decef5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 0x558decef5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f132a800640 1 -- 172.21.15.22:0/2376633272 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dece14e40 con 0x558deceec450 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1329400640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decef7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.597+0000 7f1329e00640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 0x558decef5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f132a800640 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13240115b0 tx=0x7f1324011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f93454ac5078911f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1317000640 1 -- 172.21.15.22:0/2376633272 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324004d20 con 0x558deceec450 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1322a00640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 0x7f13100a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1317000640 1 -- 172.21.15.22:0/2376633272 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1324004e80 con 0x558deceec450 2024-03-24T06:46:02.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1317000640 1 -- 172.21.15.22:0/2376633272 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132400f860 con 0x558deceec450 2024-03-24T06:46:02.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1322a00640 1 -- 172.21.15.22:0/2125255240 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13180f0980 con 0x7f132c101f60 2024-03-24T06:46:02.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.598+0000 7f1329e00640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 0x7f13100a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 -- 172.21.15.22:0/2376633272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 msgr2=0x558deceec830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13240115b0 tx=0x7f1324011a70 comp rx=0 tx=0).stop 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 -- 172.21.15.22:0/2376633272 shutdown_connections 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decef7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceecd70 0x558decef5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 --2- 172.21.15.22:0/2376633272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558deceec830 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 -- 172.21.15.22:0/2376633272 >> 172.21.15.22:0/2376633272 conn(0x558decee6b80 msgr2=0x558decee6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 -- 172.21.15.22:0/2376633272 shutdown_connections 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1333776740 1 -- 172.21.15.22:0/2376633272 wait complete. 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.599+0000 7f1329e00640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 0x7f13100a4770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x558deceebf20 tx=0x7f131c005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 -- start start 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 Processor -- start 2024-03-24T06:46:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 0x558decef3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558decdf8060 con 0x558decef5950 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558decdf7d00 con 0x558deceec450 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558decbc11c0 con 0x558deceecd70 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.600+0000 7f1333776740 10 client.0 _ll_register_callbacks cb 0x558decb63be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f132a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 0x558decef3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48466/0 (socket says 172.21.15.22:48466) 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 -- 172.21.15.22:0/3131351504 learned_addr learned my addr 172.21.15.22:0/3131351504 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329400640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 -- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 msgr2=0x558decef2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 -- 172.21.15.22:0/3131351504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 msgr2=0x558decef3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 0x558decef3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f1329e00640 1 -- 172.21.15.22:0/3131351504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1324009e20 con 0x558deceec450 2024-03-24T06:46:02.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.601+0000 7f132a800640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 0x558decef3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f1329e00640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f131c00fad0 tx=0x7f131c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3654cafe18636273 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f130b400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c023070 con 0x558deceec450 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f131c01e5a0 con 0x558deceec450 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558decb8f720 con 0x558deceec450 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f131c017070 con 0x558deceec450 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f1333776740 10 client.26684 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:02.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.602+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558dece51a70 con 0x558deceec450 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558decb8f720 con 0x558deceec450 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f131c0299c0 con 0x558deceec450 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 10 client.26684 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 10 client.26684 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 20 client.26684 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf05e30 0x558decf08250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 4 client.26684 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558decbde890 con 0x558decf05e30 2024-03-24T06:46:02.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.603+0000 7f1329400640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf05e30 0x558decf08250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.604+0000 7f1329400640 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf05e30 0x558decf08250 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=28f1bb7188cbf09b server_cookie=5cda956db1a7e1a4 in_seq=0 out_seq=0 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.604+0000 7f130b400640 10 client.26684 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f130b400640 1 -- 172.21.15.22:0/3131351504 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558decbde890 con 0x558decf05e30 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f130b400640 10 client.26684 handle_command_reply: tid=0 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 client.26684 shutdown 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 msgr2=0x558decef29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f131c00fad0 tx=0x7f131c004730 comp rx=0 tx=0).stop 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 shutdown_connections 2024-03-24T06:46:02.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf05e30 0x558decf08250 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558decef5950 0x558decef3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 --2- 172.21.15.22:0/3131351504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceec450 0x558decef29a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 >> 172.21.15.22:0/3131351504 conn(0x558decee6b80 msgr2=0x558decee88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f130b400640 0 client.26684 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 shutdown_connections 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.605+0000 7f1333776740 1 -- 172.21.15.22:0/3131351504 wait complete. 2024-03-24T06:46:02.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.606+0000 7f1333776740 20 client.26684 trim_cache size 0 max 16384 2024-03-24T06:46:02.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 Processor -- start 2024-03-24T06:46:02.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 -- start start 2024-03-24T06:46:02.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decf011e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceecd70 0x558decef3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558decee7b80 con 0x558deceec450 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558decee63d0 con 0x558deceecd70 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1333776740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558decee8e90 con 0x558decef5950 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f1329e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decf011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f132a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f132a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38300/0 (socket says 172.21.15.22:38300) 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.628+0000 7f132a800640 1 -- 172.21.15.22:0/186802487 learned_addr learned my addr 172.21.15.22:0/186802487 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f1329400640 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceecd70 0x558decef3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 -- 172.21.15.22:0/186802487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceecd70 msgr2=0x558decef3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceecd70 0x558decef3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 -- 172.21.15.22:0/186802487 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 msgr2=0x558decf011e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decf011e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 -- 172.21.15.22:0/186802487 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558decdedb00 con 0x558decef5950 2024-03-24T06:46:02.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.629+0000 7f132a800640 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1324009fd0 tx=0x7f1324004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c23a002b52dd6525 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1316600640 1 -- 172.21.15.22:0/186802487 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324003b70 con 0x558decef5950 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1316600640 1 -- 172.21.15.22:0/186802487 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1324003cd0 con 0x558decef5950 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1316600640 1 -- 172.21.15.22:0/186802487 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132400fc00 con 0x558decef5950 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 -- 172.21.15.22:0/186802487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 msgr2=0x558decf0db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1324009fd0 tx=0x7f1324004410 comp rx=0 tx=0).stop 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 -- 172.21.15.22:0/186802487 shutdown_connections 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558decef5950 0x558decf0db60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558deceecd70 0x558decef3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 --2- 172.21.15.22:0/186802487 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decf011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 -- 172.21.15.22:0/186802487 >> 172.21.15.22:0/186802487 conn(0x558decee6b80 msgr2=0x558decee5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 -- 172.21.15.22:0/186802487 shutdown_connections 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.630+0000 7f1333776740 1 -- 172.21.15.22:0/186802487 wait complete. 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.631+0000 7f1333776740 1 -- start start 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.631+0000 7f1333776740 1 Processor -- start 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decef9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 0x558decef9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558decf6b190 con 0x558deceec450 2024-03-24T06:46:02.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558decf6b2d0 con 0x558decef5950 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558decf6b410 con 0x558deceecd70 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1333776740 10 client.0 _ll_register_callbacks cb 0x7f131c009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f132a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 0x558decef9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decef9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38308/0 (socket says 172.21.15.22:38308) 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 -- 172.21.15.22:0/1761828409 learned_addr learned my addr 172.21.15.22:0/1761828409 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f132a800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 0x558decef9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48472/0 (socket says 172.21.15.22:48472) 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 -- 172.21.15.22:0/1761828409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 msgr2=0x558decef9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 0x558decef9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 -- 172.21.15.22:0/1761828409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 msgr2=0x558decef9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decef9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.632+0000 7f1329400640 1 -- 172.21.15.22:0/1761828409 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dece14e40 con 0x558deceecd70 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f1329400640 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f13180b9a90 tx=0x7f13180035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ff046584f98bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:02.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180bd7c0 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f13180bd0c0 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558decb527e0 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f13180c1210 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f1333776740 10 client.26716 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558dece136a0 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558decb527e0 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.633+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f13180d9070 con 0x558deceecd70 2024-03-24T06:46:02.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 10 client.26716 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 10 client.26716 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 20 client.26716 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf6c810 0x558decf6ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 4 client.26716 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558decafe700 con 0x558decf6c810 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1329e00640 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf6c810 0x558decf6ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f1329e00640 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf6c810 0x558decf6ec30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86b232941f66f23 server_cookie=eae919e22fa3e46c in_seq=0 out_seq=0 2024-03-24T06:46:02.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.634+0000 7f130b400640 10 client.26716 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f130b400640 1 -- 172.21.15.22:0/1761828409 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558decafe700 con 0x558decf6c810 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f130b400640 10 client.26716 handle_command_reply: tid=0 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 client.26716 shutdown 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 msgr2=0x558decef96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f13180b9a90 tx=0x7f13180035f0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 shutdown_connections 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558decf6c810 0x558decf6ec30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558decef5950 0x558decef9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558deceecd70 0x558decef96a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f130b400640 0 client.26716 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 --2- 172.21.15.22:0/1761828409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558deceec450 0x558decef9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 >> 172.21.15.22:0/1761828409 conn(0x558decee6b80 msgr2=0x558decee7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 shutdown_connections 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 1 -- 172.21.15.22:0/1761828409 wait complete. 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.635+0000 7f1333776740 20 client.26716 trim_cache size 0 max 16384 2024-03-24T06:46:02.643 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:02.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 msgr2=0x7f13100a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 0x7f13100a4770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x558deceebf20 tx=0x7f131c005f90 comp rx=0 tx=0).stop 2024-03-24T06:46:02.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 msgr2=0x7f132c19a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1318000c00 tx=0x7f131802c5f0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 shutdown_connections 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f13100a22b0 0x7f13100a4770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f132c107660 0x7f132c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f132c101f60 0x7f132c19a2c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 --2- 172.21.15.22:0/2125255240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f132c1015a0 0x7f132c199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 >> 172.21.15.22:0/2125255240 conn(0x7f132c0fd0b0 msgr2=0x7f132c074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 shutdown_connections 2024-03-24T06:46:02.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.636+0000 7f132b200640 1 -- 172.21.15.22:0/2125255240 wait complete. 2024-03-24T06:46:02.648 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:02 smithi022 ceph-mon[32003]: pgmap v185: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-03-24T06:46:02.654 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:46:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:02 smithi143 ceph-mon[36520]: pgmap v185: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-03-24T06:46:02.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 -- 172.21.15.22:0/865383270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4102740 msgr2=0x7f68a4102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 --2- 172.21.15.22:0/865383270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4102740 0x7f68a4102b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6894009ca0 tx=0x7f689402c190 comp rx=0 tx=0).stop 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 -- 172.21.15.22:0/865383270 shutdown_connections 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 --2- 172.21.15.22:0/865383270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a410bb60 0x7f68a410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 --2- 172.21.15.22:0/865383270 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 0x7f68a410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 --2- 172.21.15.22:0/865383270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4102740 0x7f68a4102b20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 -- 172.21.15.22:0/865383270 >> 172.21.15.22:0/865383270 conn(0x7f68a40fbf60 msgr2=0x7f68a40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 -- 172.21.15.22:0/865383270 shutdown_connections 2024-03-24T06:46:02.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.883+0000 7f68a9c00640 1 -- 172.21.15.22:0/865383270 wait complete. 2024-03-24T06:46:02.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.884+0000 7f68a9c00640 1 Processor -- start 2024-03-24T06:46:02.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 -- start start 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 0x7f68a419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a410bb60 0x7f68a4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4198050 con 0x7f68a4103060 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f68a41981c0 con 0x7f68a410bb60 2024-03-24T06:46:02.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.885+0000 7f68a9c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f68a4198380 con 0x7f68a4102740 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38324/0 (socket says 172.21.15.22:38324) 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 -- 172.21.15.22:0/609295807 learned_addr learned my addr 172.21.15.22:0/609295807 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3400640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 0x7f68a419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a9200640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a410bb60 0x7f68a4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 -- 172.21.15.22:0/609295807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a410bb60 msgr2=0x7f68a4197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a410bb60 0x7f68a4197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 -- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 msgr2=0x7f68a419dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 0x7f68a419dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 -- 172.21.15.22:0/609295807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6894009900 con 0x7f68a4102740 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.886+0000 7f68a3e00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6894005e30 tx=0x7f68940346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.887+0000 7f68a0c00640 1 -- 172.21.15.22:0/609295807 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894003d80 con 0x7f68a4102740 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.887+0000 7f68a0c00640 1 -- 172.21.15.22:0/609295807 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68940348e0 con 0x7f68a4102740 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.887+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a4198600 con 0x7f68a4102740 2024-03-24T06:46:02.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.887+0000 7f68a0c00640 1 -- 172.21.15.22:0/609295807 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894004500 con 0x7f68a4102740 2024-03-24T06:46:02.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.887+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a41004a0 con 0x7f68a4102740 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.891+0000 7f68adf86740 1 Processor -- start 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.891+0000 7f68adf86740 1 -- start start 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761df410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555c761446e0 con 0x555c761df950 2024-03-24T06:46:02.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555c76141c10 con 0x555c761d6d70 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68adf86740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555c7613d3f0 con 0x555c761d6450 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a0c00640 1 -- 172.21.15.22:0/609295807 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f689403b050 con 0x7f68a4102740 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38332/0 (socket says 172.21.15.22:38332) 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 -- 172.21.15.22:0/263177981 learned_addr learned my addr 172.21.15.22:0/263177981 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a3400640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a3e00640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761df410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 -- 172.21.15.22:0/263177981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 msgr2=0x555c761df410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761df410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 -- 172.21.15.22:0/263177981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 msgr2=0x555c761e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a9200640 1 -- 172.21.15.22:0/263177981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c760fee40 con 0x555c761d6450 2024-03-24T06:46:02.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a3400640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:02.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.892+0000 7f68a3e00640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761df410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:46:02.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.893+0000 7f68a9200640 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f689c010dd0 tx=0x7f689c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=289a5767b757e75d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.893+0000 7f6899600640 1 -- 172.21.15.22:0/263177981 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c015050 con 0x555c761d6450 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.893+0000 7f6899600640 1 -- 172.21.15.22:0/263177981 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f689c016040 con 0x555c761d6450 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.893+0000 7f6899600640 1 -- 172.21.15.22:0/263177981 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c0199e0 con 0x555c761d6450 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 -- 172.21.15.22:0/263177981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 msgr2=0x555c761d6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f689c010dd0 tx=0x7f689c005f90 comp rx=0 tx=0).stop 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68a0c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 0x7f68880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 -- 172.21.15.22:0/263177981 shutdown_connections 2024-03-24T06:46:02.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761df410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 --2- 172.21.15.22:0/263177981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761d6830 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 -- 172.21.15.22:0/263177981 >> 172.21.15.22:0/263177981 conn(0x555c761d0b80 msgr2=0x555c761d0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68a0c00640 1 -- 172.21.15.22:0/609295807 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f68940f0400 con 0x7f68a4102740 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68a3e00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 0x7f68880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 -- 172.21.15.22:0/263177981 shutdown_connections 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.894+0000 7f68adf86740 1 -- 172.21.15.22:0/263177981 wait complete. 2024-03-24T06:46:02.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.895+0000 7f68a3e00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 0x7f68880a4770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x555c761d5f20 tx=0x7f68940095c0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.895+0000 7f68adf86740 1 -- start start 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.895+0000 7f68adf86740 1 Processor -- start 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 0x555c761dcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761dd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555c760e2060 con 0x555c761df950 2024-03-24T06:46:02.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555c760e1d00 con 0x555c761d6450 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555c75eab1c0 con 0x555c761d6d70 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68adf86740 10 client.0 _ll_register_callbacks cb 0x555c75e4dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 0x555c761dcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48512/0 (socket says 172.21.15.22:48512) 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a9200640 1 -- 172.21.15.22:0/655571239 learned_addr learned my addr 172.21.15.22:0/655571239 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a9200640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761dd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a3400640 1 -- 172.21.15.22:0/655571239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 msgr2=0x555c761dca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a3400640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.896+0000 7f68a3400640 1 -- 172.21.15.22:0/655571239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 msgr2=0x555c761dd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68a3400640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761dd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68a3400640 1 -- 172.21.15.22:0/655571239 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689c010a80 con 0x555c761d6d70 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68a9200640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68a3400640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 0x555c761dcf70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f68a410cf20 tx=0x7f688c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d95bde0ad7803c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f6892000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c017070 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555c75e3c7e0 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f688c004030 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f688c0041f0 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68adf86740 10 client.26728 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.897+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555c7613ba70 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555c75e3c7e0 con 0x555c761d6d70 2024-03-24T06:46:02.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f688c0039a0 con 0x555c761d6d70 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 10 client.26728 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 10 client.26728 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 20 client.26728 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c761efe70 0x555c761f2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 4 client.26728 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c75e79720 con 0x555c761efe70 2024-03-24T06:46:02.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.898+0000 7f68a9200640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c761efe70 0x555c761f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.899+0000 7f68a9200640 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c761efe70 0x555c761f2290 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a537668e9b81475a server_cookie=c0de49b17ffa7822 in_seq=0 out_seq=0 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.899+0000 7f6892000640 10 client.26728 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f6892000640 1 -- 172.21.15.22:0/655571239 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555c75e79720 con 0x555c761efe70 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f6892000640 10 client.26728 handle_command_reply: tid=0 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 client.26728 shutdown 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 msgr2=0x555c761dcf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 0x555c761dcf70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f68a410cf20 tx=0x7f688c012b60 comp rx=0 tx=0).stop 2024-03-24T06:46:02.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 shutdown_connections 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c761efe70 0x555c761f2290 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761dd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6d70 0x555c761dcf70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 --2- 172.21.15.22:0/655571239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6450 0x555c761dca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f6892000640 0 client.26728 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.900+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 >> 172.21.15.22:0/655571239 conn(0x555c761d0b80 msgr2=0x555c761d28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.901+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 shutdown_connections 2024-03-24T06:46:02.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.901+0000 7f68adf86740 1 -- 172.21.15.22:0/655571239 wait complete. 2024-03-24T06:46:02.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.901+0000 7f68adf86740 20 client.26728 trim_cache size 0 max 16384 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 Processor -- start 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 -- start start 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761df160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555c761cc4b0 con 0x555c761df950 2024-03-24T06:46:02.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555c761cc090 con 0x555c761d6d70 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68adf86740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555c761d60b0 con 0x555c761d6450 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68a3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68a3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48522/0 (socket says 172.21.15.22:48522) 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.922+0000 7f68a3400640 1 -- 172.21.15.22:0/124001003 learned_addr learned my addr 172.21.15.22:0/124001003 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 -- 172.21.15.22:0/124001003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 msgr2=0x555c761df160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761df160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 -- 172.21.15.22:0/124001003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 msgr2=0x555c761eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 -- 172.21.15.22:0/124001003 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c760d7b00 con 0x555c761d6d70 2024-03-24T06:46:02.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f68a3400640 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f688c003d30 tx=0x7f688c005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc19b8f0c917419 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f6898c00640 1 -- 172.21.15.22:0/124001003 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c017070 con 0x555c761d6d70 2024-03-24T06:46:02.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f6898c00640 1 -- 172.21.15.22:0/124001003 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c00f790 con 0x555c761d6d70 2024-03-24T06:46:02.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.923+0000 7f6898c00640 1 -- 172.21.15.22:0/124001003 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c005360 con 0x555c761d6d70 2024-03-24T06:46:02.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 -- 172.21.15.22:0/124001003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 msgr2=0x555c761cbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f688c003d30 tx=0x7f688c005db0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 -- 172.21.15.22:0/124001003 shutdown_connections 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761df950 0x555c761eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761d6d70 0x555c761cbb50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 --2- 172.21.15.22:0/124001003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761df160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 -- 172.21.15.22:0/124001003 >> 172.21.15.22:0/124001003 conn(0x555c761d0b80 msgr2=0x555c761cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 -- 172.21.15.22:0/124001003 shutdown_connections 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.924+0000 7f68adf86740 1 -- 172.21.15.22:0/124001003 wait complete. 2024-03-24T06:46:02.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 -- start start 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 Processor -- start 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761e7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 0x555c761e3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555c761e3e40 con 0x555c761d6d70 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555c761e3fb0 con 0x555c761df950 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555c761e4170 con 0x555c761d6450 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68adf86740 10 client.0 _ll_register_callbacks cb 0x7f688c009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:02.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68a3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68a3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38826/0 (socket says 172.21.15.22:38826) 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 -- 172.21.15.22:0/2723587069 learned_addr learned my addr 172.21.15.22:0/2723587069 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.925+0000 7f68a9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761e7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38358/0 (socket says 172.21.15.22:38358) 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a9200640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 0x555c761e3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 -- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 msgr2=0x555c761e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 -- 172.21.15.22:0/2723587069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 msgr2=0x555c761e3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 0x555c761e3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 -- 172.21.15.22:0/2723587069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c7613ba70 con 0x555c761d6d70 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a9200640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 0x555c761e3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f68a3400640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f688c015040 tx=0x7f688c0123e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7819d1b66dc8c5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.926+0000 7f6892000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c017040 con 0x555c761d6d70 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555c75ec8890 con 0x555c761d6d70 2024-03-24T06:46:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f688c000e40 con 0x555c761d6d70 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f688c003930 con 0x555c761d6d70 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f68adf86740 10 client.16338 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555c760fd6a0 con 0x555c761d6d70 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555c75ec8890 con 0x555c761d6d70 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.927+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f688c02f020 con 0x555c761d6d70 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 10 client.16338 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 10 client.16338 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 20 client.16338 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:02.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c76256f20 0x555c76259340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 4 client.16338 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c75de8700 con 0x555c76256f20 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68a9200640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c76256f20 0x555c76259340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f68a9200640 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c76256f20 0x555c76259340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=923cdadce998bc6e server_cookie=492413391a450b19 in_seq=0 out_seq=0 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.928+0000 7f6892000640 10 client.16338 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f6892000640 1 -- 172.21.15.22:0/2723587069 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c75de8700 con 0x555c76256f20 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f6892000640 10 client.16338 handle_command_reply: tid=0 2024-03-24T06:46:02.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 client.16338 shutdown 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 msgr2=0x555c761e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f688c015040 tx=0x7f688c0123e0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 shutdown_connections 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x555c76256f20 0x555c76259340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555c761df950 0x555c761e3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555c761d6d70 0x555c761e78d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f6892000640 0 client.16338 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:02.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 --2- 172.21.15.22:0/2723587069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555c761d6450 0x555c761e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 >> 172.21.15.22:0/2723587069 conn(0x555c761d0b80 msgr2=0x555c761cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 shutdown_connections 2024-03-24T06:46:02.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 1 -- 172.21.15.22:0/2723587069 wait complete. 2024-03-24T06:46:02.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.929+0000 7f68adf86740 20 client.16338 trim_cache size 0 max 16384 2024-03-24T06:46:02.938 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:02.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 msgr2=0x7f68880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 0x7f68880a4770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x555c761d5f20 tx=0x7f68940095c0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 msgr2=0x7f68a419d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6894005e30 tx=0x7f68940346f0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 shutdown_connections 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f68880a22b0 0x7f68880a4770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a410bb60 0x7f68a4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a4103060 0x7f68a419dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 --2- 172.21.15.22:0/609295807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a4102740 0x7f68a419d850 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 >> 172.21.15.22:0/609295807 conn(0x7f68a40fbf60 msgr2=0x7f68a40fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 shutdown_connections 2024-03-24T06:46:02.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:02.930+0000 7f68a9c00640 1 -- 172.21.15.22:0/609295807 wait complete. 2024-03-24T06:46:02.948 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:46:03.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 -- 172.21.15.22:0/2840778487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db41098c0 msgr2=0x7f3db4109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 --2- 172.21.15.22:0/2840778487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db41098c0 0x7f3db4109ca0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009930 tx=0x7f3da402bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:03.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 -- 172.21.15.22:0/2840778487 shutdown_connections 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 --2- 172.21.15.22:0/2840778487 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db4104270 0x7f3db4110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 --2- 172.21.15.22:0/2840778487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41038b0 0x7f3db4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 --2- 172.21.15.22:0/2840778487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db41098c0 0x7f3db4109ca0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 -- 172.21.15.22:0/2840778487 >> 172.21.15.22:0/2840778487 conn(0x7f3db40ff380 msgr2=0x7f3db41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 -- 172.21.15.22:0/2840778487 shutdown_connections 2024-03-24T06:46:03.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.150+0000 7f3dba600640 1 -- 172.21.15.22:0/2840778487 wait complete. 2024-03-24T06:46:03.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.151+0000 7f3dba600640 1 Processor -- start 2024-03-24T06:46:03.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.151+0000 7f3dba600640 1 -- start start 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 0x7f3db4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41098c0 0x7f3db419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3db419ee40 con 0x7f3db41038b0 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3db419efb0 con 0x7f3db41098c0 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3dba600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3db419f170 con 0x7f3db4104270 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 0x7f3db4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db2a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38376/0 (socket says 172.21.15.22:38376) 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db2a00640 1 -- 172.21.15.22:0/3527661653 learned_addr learned my addr 172.21.15.22:0/3527661653 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db3400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 0x7f3db4199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38848/0 (socket says 172.21.15.22:38848) 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.152+0000 7f3db3e00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41098c0 0x7f3db419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 -- 172.21.15.22:0/3527661653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41098c0 msgr2=0x7f3db419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41098c0 0x7f3db419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 -- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 msgr2=0x7f3db4199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 0x7f3db4199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 -- 172.21.15.22:0/3527661653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da4009560 con 0x7f3db4104270 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3db2a00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3da000e9f0 tx=0x7f3da000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3dabe00640 1 -- 172.21.15.22:0/3527661653 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0009700 con 0x7f3db4104270 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3dabe00640 1 -- 172.21.15.22:0/3527661653 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da000cd80 con 0x7f3db4104270 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db419f450 con 0x7f3db4104270 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3dabe00640 1 -- 172.21.15.22:0/3527661653 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0018920 con 0x7f3db4104270 2024-03-24T06:46:03.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.153+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db41a6cd0 con 0x7f3db4104270 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 Processor -- start 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 -- start start 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea12181660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea1218a780 0x55ea1218cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea120ef6e0 con 0x55ea12181280 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea120ecc10 con 0x55ea12181ba0 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3dbd216740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea120e83f0 con 0x55ea1218a780 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea12181660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3db3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3db3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48560/0 (socket says 172.21.15.22:48560) 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea12181660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38858/0 (socket says 172.21.15.22:38858) 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.157+0000 7f3db3400640 1 -- 172.21.15.22:0/2449183923 learned_addr learned my addr 172.21.15.22:0/2449183923 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 -- 172.21.15.22:0/2449183923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea1218a780 msgr2=0x55ea1218cb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:46:03.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea1218a780 0x55ea1218cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 -- 172.21.15.22:0/2449183923 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 msgr2=0x55ea12181660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea12181660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 -- 172.21.15.22:0/2449183923 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea120a9e40 con 0x55ea12181ba0 2024-03-24T06:46:03.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3db3400640 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3da4005ab0 tx=0x7f3da40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecfef235f7a76ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3da8c00640 1 -- 172.21.15.22:0/2449183923 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40048b0 con 0x55ea12181ba0 2024-03-24T06:46:03.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.158+0000 7f3da8c00640 1 -- 172.21.15.22:0/2449183923 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da4004a10 con 0x55ea12181ba0 2024-03-24T06:46:03.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3da8c00640 1 -- 172.21.15.22:0/2449183923 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40051e0 con 0x55ea12181ba0 2024-03-24T06:46:03.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dabe00640 1 -- 172.21.15.22:0/3527661653 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3da0018a80 con 0x7f3db4104270 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 -- 172.21.15.22:0/2449183923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 msgr2=0x55ea1218a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3da4005ab0 tx=0x7f3da40046c0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 -- 172.21.15.22:0/2449183923 shutdown_connections 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea1218a780 0x55ea1218cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218a240 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 --2- 172.21.15.22:0/2449183923 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea12181660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 -- 172.21.15.22:0/2449183923 >> 172.21.15.22:0/2449183923 conn(0x55ea1217b9b0 msgr2=0x55ea1217bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 -- 172.21.15.22:0/2449183923 shutdown_connections 2024-03-24T06:46:03.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.159+0000 7f3dbd216740 1 -- 172.21.15.22:0/2449183923 wait complete. 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.160+0000 7f3dbd216740 1 -- start start 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.160+0000 7f3dbd216740 1 Processor -- start 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea121878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea12187e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dabe00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 0x7f3d980a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 0x55ea12188370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea1208d060 con 0x55ea12181280 2024-03-24T06:46:03.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea1208cd00 con 0x55ea1218a780 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea11e561c0 con 0x55ea12181ba0 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dbd216740 10 client.0 _ll_register_callbacks cb 0x55ea11df8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea12187e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea121878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3dabe00640 1 -- 172.21.15.22:0/3527661653 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3da0014070 con 0x7f3db4104270 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea121878f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38870/0 (socket says 172.21.15.22:38870) 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db3e00640 1 -- 172.21.15.22:0/4236356608 learned_addr learned my addr 172.21.15.22:0/4236356608 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db2a00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 0x7f3d980a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.161+0000 7f3db3e00640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 0x55ea12188370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 -- 172.21.15.22:0/4236356608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 msgr2=0x55ea12188370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 0x55ea12188370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 -- 172.21.15.22:0/4236356608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 msgr2=0x55ea121878f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea121878f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 -- 172.21.15.22:0/4236356608 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db4111080 con 0x55ea12181ba0 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3e00640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 0x55ea12188370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:03.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db3400640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea12187e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3da402c170 tx=0x7f3da4005190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28954be6a0ff910e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3d9d600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3db2a00640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 0x7f3d980a4600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55ea1217cb40 tx=0x7f3da001f000 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da40048b0 con 0x55ea12181ba0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea11de77e0 con 0x55ea12181ba0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.162+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3da404a450 con 0x55ea12181ba0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3da404a610 con 0x55ea12181ba0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 10 client.26740 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ea120e6a70 con 0x55ea12181ba0 2024-03-24T06:46:03.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da405a080 con 0x55ea12181ba0 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f3da403c050 con 0x55ea12181ba0 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 10 client.26740 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 10 client.26740 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 20 client.26740 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea1219ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 4 client.26740 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea11e24720 con 0x55ea1219a8c0 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.163+0000 7f3db3e00640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea1219ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.164+0000 7f3db3e00640 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea1219ccb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d3ca1e9ba635459 server_cookie=a8477709b694ff0c in_seq=0 out_seq=0 2024-03-24T06:46:03.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.164+0000 7f3d9d600640 10 client.26740 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3d9d600640 1 -- 172.21.15.22:0/4236356608 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea11e24720 con 0x55ea1219a8c0 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3d9d600640 10 client.26740 handle_command_reply: tid=0 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 client.26740 shutdown 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 msgr2=0x55ea12187e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea12187e30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3da402c170 tx=0x7f3da4005190 comp rx=0 tx=0).stop 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 shutdown_connections 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea1219ccb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea1218a780 0x55ea12188370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3d9d600640 0 client.26740 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea12187e30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 --2- 172.21.15.22:0/4236356608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea12181280 0x55ea121878f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 >> 172.21.15.22:0/4236356608 conn(0x55ea1217b9b0 msgr2=0x55ea1217d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 shutdown_connections 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 1 -- 172.21.15.22:0/4236356608 wait complete. 2024-03-24T06:46:03.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.165+0000 7f3dbd216740 20 client.26740 trim_cache size 0 max 16384 2024-03-24T06:46:03.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.180+0000 7f3dbd216740 1 Processor -- start 2024-03-24T06:46:03.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.180+0000 7f3dbd216740 1 -- start start 2024-03-24T06:46:03.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea121925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea11e24720 con 0x55ea1218a780 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea1217c9b0 con 0x55ea12181280 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3dbd216740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea1217dd00 con 0x55ea12181ba0 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48568/0 (socket says 172.21.15.22:48568) 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38392/0 (socket says 172.21.15.22:38392) 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 -- 172.21.15.22:0/878835210 learned_addr learned my addr 172.21.15.22:0/878835210 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea121925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 -- 172.21.15.22:0/878835210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 msgr2=0x55ea121a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 -- 172.21.15.22:0/878835210 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 msgr2=0x55ea121925d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea121925d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3e00640 1 -- 172.21.15.22:0/878835210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea12082b00 con 0x55ea12181280 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.181+0000 7f3db3400640 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3db3e00640 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005b40 tx=0x7f3dac009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e38de149615b432 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3d9fe00640 1 -- 172.21.15.22:0/878835210 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac018070 con 0x55ea12181280 2024-03-24T06:46:03.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3d9fe00640 1 -- 172.21.15.22:0/878835210 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac023d20 con 0x55ea12181280 2024-03-24T06:46:03.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3d9fe00640 1 -- 172.21.15.22:0/878835210 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac003970 con 0x55ea12181280 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3dbd216740 1 -- 172.21.15.22:0/878835210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 msgr2=0x55ea12195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.182+0000 7f3dbd216740 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005b40 tx=0x7f3dac009bd0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 -- 172.21.15.22:0/878835210 shutdown_connections 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea121925d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181ba0 0x55ea121a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 --2- 172.21.15.22:0/878835210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181280 0x55ea12195fb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 -- 172.21.15.22:0/878835210 >> 172.21.15.22:0/878835210 conn(0x55ea1217b9b0 msgr2=0x55ea1217a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 -- 172.21.15.22:0/878835210 shutdown_connections 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 -- 172.21.15.22:0/878835210 wait complete. 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 -- start start 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.183+0000 7f3dbd216740 1 Processor -- start 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181280 0x55ea121927d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea1218faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea12190030 con 0x55ea1218a780 2024-03-24T06:46:03.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea121901a0 con 0x55ea12181ba0 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea12190360 con 0x55ea12181280 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3dbd216740 10 client.0 _ll_register_callbacks cb 0x7f3dac009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181280 0x55ea121927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea1218faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3db3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea1218faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38890/0 (socket says 172.21.15.22:38890) 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.184+0000 7f3db3e00640 1 -- 172.21.15.22:0/3167018167 learned_addr learned my addr 172.21.15.22:0/3167018167 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 -- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181280 msgr2=0x55ea121927d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181280 0x55ea121927d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 -- 172.21.15.22:0/3167018167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 msgr2=0x55ea1218faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea1218faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 -- 172.21.15.22:0/3167018167 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea120e6a70 con 0x55ea12181ba0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3db3400640 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218f5b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3da403f7b0 tx=0x7f3da4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d7b69b4ad025a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3d9d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da403fa30 con 0x55ea12181ba0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da4004a10 con 0x55ea12181ba0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3da4000e40 con 0x55ea12181ba0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3da4005850 con 0x55ea12181ba0 2024-03-24T06:46:03.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3dbd216740 10 client.26723 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.185+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x7f3dac01b040 con 0x55ea12181ba0 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da4004a10 con 0x55ea12181ba0 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f3da403c070 con 0x55ea12181ba0 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 10 client.26723 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 10 client.26723 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 20 client.26723 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea12203e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 4 client.26723 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea11e73890 con 0x55ea1219a8c0 2024-03-24T06:46:03.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.186+0000 7f3db3e00640 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea12203e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3db3e00640 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea12203e10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5961ae05524a805f server_cookie=525e8f23b295e5d9 in_seq=0 out_seq=0 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3d9d600640 10 client.26723 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3d9d600640 1 -- 172.21.15.22:0/3167018167 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea11e73890 con 0x55ea1219a8c0 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3d9d600640 10 client.26723 handle_command_reply: tid=0 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 client.26723 shutdown 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 msgr2=0x55ea1218f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218f5b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3da403f7b0 tx=0x7f3da4005740 comp rx=0 tx=0).stop 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 shutdown_connections 2024-03-24T06:46:03.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ea1219a8c0 0x55ea12203e10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea1218a780 0x55ea1218faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea12181ba0 0x55ea1218f5b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 --2- 172.21.15.22:0/3167018167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea12181280 0x55ea121927d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 >> 172.21.15.22:0/3167018167 conn(0x55ea1217b9b0 msgr2=0x55ea12188140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3d9d600640 0 client.26723 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 shutdown_connections 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.187+0000 7f3dbd216740 1 -- 172.21.15.22:0/3167018167 wait complete. 2024-03-24T06:46:03.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dbd216740 20 client.26723 trim_cache size 0 max 16384 2024-03-24T06:46:03.196 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:03.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 msgr2=0x7f3d980a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 0x7f3d980a4600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x55ea1217cb40 tx=0x7f3da001f000 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 msgr2=0x7f3db419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3da000e9f0 tx=0x7f3da000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 shutdown_connections 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3d980a2140 0x7f3d980a4600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db41098c0 0x7f3db419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3db4104270 0x7f3db419a4c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 --2- 172.21.15.22:0/3527661653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3db41038b0 0x7f3db4199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.188+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 >> 172.21.15.22:0/3527661653 conn(0x7f3db40ff380 msgr2=0x7f3db40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.189+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 shutdown_connections 2024-03-24T06:46:03.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.189+0000 7f3dba600640 1 -- 172.21.15.22:0/3527661653 wait complete. 2024-03-24T06:46:03.206 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:46:03.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 -- 172.21.15.22:0/731692884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4101f50 msgr2=0x7fdde410e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 --2- 172.21.15.22:0/731692884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4101f50 0x7fdde410e410 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009cd0 tx=0x7fdddc02c0f0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 -- 172.21.15.22:0/731692884 shutdown_connections 2024-03-24T06:46:03.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 --2- 172.21.15.22:0/731692884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4101f50 0x7fdde410e410 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 --2- 172.21.15.22:0/731692884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde4101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 --2- 172.21.15.22:0/731692884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4107650 0x7fdde4107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 -- 172.21.15.22:0/731692884 >> 172.21.15.22:0/731692884 conn(0x7fdde40fd080 msgr2=0x7fdde40ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.424+0000 7fdde8a00640 1 -- 172.21.15.22:0/731692884 shutdown_connections 2024-03-24T06:46:03.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.425+0000 7fdde8a00640 1 -- 172.21.15.22:0/731692884 wait complete. 2024-03-24T06:46:03.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 Processor -- start 2024-03-24T06:46:03.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 -- start start 2024-03-24T06:46:03.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4101f50 0x7fdde419dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4107650 0x7fdde4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4197f60 con 0x7fdde4101f50 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdde41980d0 con 0x7fdde4107650 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4198290 con 0x7fdde4101590 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.426+0000 7fdde2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38424/0 (socket says 172.21.15.22:38424) 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 -- 172.21.15.22:0/1255432822 learned_addr learned my addr 172.21.15.22:0/1255432822 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdddbe00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4101f50 0x7fdde419dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde3200640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4107650 0x7fdde4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 -- 172.21.15.22:0/1255432822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4107650 msgr2=0x7fdde4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4107650 0x7fdde4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 -- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4101f50 msgr2=0x7fdde419dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4101f50 0x7fdde419dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 -- 172.21.15.22:0/1255432822 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc009900 con 0x7fdde4101590 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.427+0000 7fdde2800640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e9f0 tx=0x7fddd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.428+0000 7fdddb400640 1 -- 172.21.15.22:0/1255432822 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4009700 con 0x7fdde4101590 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.428+0000 7fdddb400640 1 -- 172.21.15.22:0/1255432822 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddd400cd80 con 0x7fdde4101590 2024-03-24T06:46:03.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.428+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde4198570 con 0x7fdde4101590 2024-03-24T06:46:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.428+0000 7fdddb400640 1 -- 172.21.15.22:0/1255432822 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4018920 con 0x7fdde4101590 2024-03-24T06:46:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.428+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde41a4800 con 0x7fdde4101590 2024-03-24T06:46:03.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.431+0000 7fddec1d4740 1 Processor -- start 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.431+0000 7fddec1d4740 1 -- start start 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 0x560faf09f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560faf0006e0 con 0x560faf09fd60 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560faeffdc10 con 0x560faf096860 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fddec1d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560faeff93f0 con 0x560faf097180 2024-03-24T06:46:03.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdde3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdde3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48608/0 (socket says 172.21.15.22:48608) 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdde3200640 1 -- 172.21.15.22:0/4048714806 learned_addr learned my addr 172.21.15.22:0/4048714806 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdddb400640 1 -- 172.21.15.22:0/1255432822 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fddd4018a80 con 0x7fdde4101590 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdddbe00640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.432+0000 7fdde3200640 1 -- 172.21.15.22:0/4048714806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 msgr2=0x560faf09f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde2800640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 0x560faf09f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde3200640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 0x560faf09f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde3200640 1 -- 172.21.15.22:0/4048714806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 msgr2=0x560faf0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde3200640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde3200640 1 -- 172.21.15.22:0/4048714806 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560faefbae40 con 0x560faf096860 2024-03-24T06:46:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdddbe00640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:03.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde2800640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 0x560faf09f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:46:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fdde3200640 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdddc004af0 tx=0x7fdddc005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9d916366f356dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fddd3400640 1 -- 172.21.15.22:0/4048714806 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc034440 con 0x560faf096860 2024-03-24T06:46:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fddd3400640 1 -- 172.21.15.22:0/4048714806 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc0345a0 con 0x560faf096860 2024-03-24T06:46:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.433+0000 7fddd3400640 1 -- 172.21.15.22:0/4048714806 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc003bd0 con 0x560faf096860 2024-03-24T06:46:03.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fdddb400640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 0x7fddc80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fdddb400640 1 -- 172.21.15.22:0/1255432822 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fddd4014070 con 0x7fdde4101590 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 -- 172.21.15.22:0/4048714806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 msgr2=0x560faf096c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdddc004af0 tx=0x7fdddc005f80 comp rx=0 tx=0).stop 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 -- 172.21.15.22:0/4048714806 shutdown_connections 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fdde2800640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 0x7fddc80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf097180 0x560faf09f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 --2- 172.21.15.22:0/4048714806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf096c40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 -- 172.21.15.22:0/4048714806 >> 172.21.15.22:0/4048714806 conn(0x560faf090f90 msgr2=0x560faf091370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 -- 172.21.15.22:0/4048714806 shutdown_connections 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.434+0000 7fddec1d4740 1 -- 172.21.15.22:0/4048714806 wait complete. 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.435+0000 7fddec1d4740 1 -- start start 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.435+0000 7fdde2800640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 0x7fddc80a4770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x560faf096330 tx=0x7fddd4000bc0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.435+0000 7fddec1d4740 1 Processor -- start 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 0x560faf09d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560faef9e060 con 0x560faf097180 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560faef9dd00 con 0x560faf09fd60 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560faed671c0 con 0x560faf096860 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fddec1d4740 10 client.0 _ll_register_callbacks cb 0x560faed09be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fdde3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fdde3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38444/0 (socket says 172.21.15.22:38444) 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fdde3200640 1 -- 172.21.15.22:0/802614051 learned_addr learned my addr 172.21.15.22:0/802614051 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fdddbe00640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.436+0000 7fdde3200640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 0x560faf09d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 -- 172.21.15.22:0/802614051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 msgr2=0x560faf09ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 -- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 msgr2=0x560faf09d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 0x560faf09d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 -- 172.21.15.22:0/802614051 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde410ed10 con 0x560faf097180 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdde3200640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 0x560faf09d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:03.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fdddbe00640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09d360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdde4106cc0 tx=0x7fddcc012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dac870c5d228ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fddc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc017070 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fddcc00fa30 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560faecf87e0 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.437+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fddcc00fbf0 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 10 client.16368 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560faeff7a70 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560faecf87e0 con 0x560faf097180 2024-03-24T06:46:03.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fddcc003a60 con 0x560faf097180 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 10 client.16368 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 10 client.16368 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 20 client.16368 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 4 client.16368 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.438+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560faed35720 con 0x560faf0b0250 2024-03-24T06:46:03.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.439+0000 7fdde3200640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.439+0000 7fddc7e00640 10 client.16368 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.439+0000 7fdde3200640 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf0b2670 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df9273c31195f6d5 server_cookie=17f6ff4cd20a8d7e in_seq=0 out_seq=0 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddc7e00640 1 -- 172.21.15.22:0/802614051 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560faed35720 con 0x560faf0b0250 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddc7e00640 10 client.16368 handle_command_reply: tid=0 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 client.16368 shutdown 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 msgr2=0x560faf09d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09d360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdde4106cc0 tx=0x7fddcc012b60 comp rx=0 tx=0).stop 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 shutdown_connections 2024-03-24T06:46:03.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf0b2670 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddc7e00640 0 client.16368 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.440+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf09fd60 0x560faf09d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09d360 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 1 --2- 172.21.15.22:0/802614051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 >> 172.21.15.22:0/802614051 conn(0x560faf090f90 msgr2=0x560faf08bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 shutdown_connections 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 1 -- 172.21.15.22:0/802614051 wait complete. 2024-03-24T06:46:03.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.441+0000 7fddec1d4740 20 client.16368 trim_cache size 0 max 16384 2024-03-24T06:46:03.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 Processor -- start 2024-03-24T06:46:03.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 -- start start 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0b7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.456+0000 7fddec1d4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560faf0884b0 con 0x560faf09fd60 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fddec1d4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560faf0abad0 con 0x560faf097180 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fddec1d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560faf090ac0 con 0x560faf096860 2024-03-24T06:46:03.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdde3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48624/0 (socket says 172.21.15.22:48624) 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 -- 172.21.15.22:0/2346996018 learned_addr learned my addr 172.21.15.22:0/2346996018 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdde3200640 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0b7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 -- 172.21.15.22:0/2346996018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 msgr2=0x560faf09f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 -- 172.21.15.22:0/2346996018 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 msgr2=0x560faf0b7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0b7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdddbe00640 1 -- 172.21.15.22:0/2346996018 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560faef93b00 con 0x560faf097180 2024-03-24T06:46:03.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.457+0000 7fdde3200640 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fdddbe00640 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fddcc000c00 tx=0x7fddcc005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32ab23ca38f3ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddd2a00640 1 -- 172.21.15.22:0/2346996018 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc017040 con 0x560faf097180 2024-03-24T06:46:03.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddd2a00640 1 -- 172.21.15.22:0/2346996018 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddcc00f790 con 0x560faf097180 2024-03-24T06:46:03.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddd2a00640 1 -- 172.21.15.22:0/2346996018 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddcc003850 con 0x560faf097180 2024-03-24T06:46:03.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 -- 172.21.15.22:0/2346996018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 msgr2=0x560faf0ab590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fddcc000c00 tx=0x7fddcc005db0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 -- 172.21.15.22:0/2346996018 shutdown_connections 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf09fd60 0x560faf0b7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf097180 0x560faf0ab590 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.458+0000 7fddec1d4740 1 --2- 172.21.15.22:0/2346996018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf096860 0x560faf09f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.459+0000 7fddec1d4740 1 -- 172.21.15.22:0/2346996018 >> 172.21.15.22:0/2346996018 conn(0x560faf090f90 msgr2=0x560faf08bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.459+0000 7fddec1d4740 1 -- 172.21.15.22:0/2346996018 shutdown_connections 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.459+0000 7fddec1d4740 1 -- 172.21.15.22:0/2346996018 wait complete. 2024-03-24T06:46:03.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.459+0000 7fddec1d4740 1 -- start start 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.459+0000 7fddec1d4740 1 Processor -- start 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf09fd60 0x560faf09a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x560faf1156e0 con 0x560faf097180 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560faf115850 con 0x560faf096860 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560faf115a10 con 0x560faf09fd60 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fddec1d4740 10 client.0 _ll_register_callbacks cb 0x7fddcc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:48638/0 (socket says 172.21.15.22:48638) 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 -- 172.21.15.22:0/224485148 learned_addr learned my addr 172.21.15.22:0/224485148 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdddbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38946/0 (socket says 172.21.15.22:38946) 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf09fd60 0x560faf09a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 -- 172.21.15.22:0/224485148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf09fd60 msgr2=0x560faf09a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf09fd60 0x560faf09a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 -- 172.21.15.22:0/224485148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 msgr2=0x560faf09a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdde3200640 1 -- 172.21.15.22:0/224485148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560faefbae40 con 0x560faf096860 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.460+0000 7fdddbe00640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fdde3200640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdddc004af0 tx=0x7fdddc04c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b408e3e6973772c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc04f040 con 0x560faf096860 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560faed84890 con 0x560faf096860 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdddc058b40 con 0x560faf096860 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdddc058d00 con 0x560faf096860 2024-03-24T06:46:03.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 10 client.26741 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x7fddcc021040 con 0x560faf096860 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc066020 con 0x560faf096860 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fdddc0582c0 con 0x560faf096860 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 10 client.26741 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 10 client.26741 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.461+0000 7fddec1d4740 20 client.26741 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf118e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fddec1d4740 4 client.26741 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560faeca4700 con 0x560faf0b0250 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fdddbe00640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf118e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fdddbe00640 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf118e00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9cf4e89db1d68437 server_cookie=26689da822eb3542 in_seq=0 out_seq=0 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fddc7e00640 10 client.26741 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.462+0000 7fddc7e00640 1 -- 172.21.15.22:0/224485148 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560faeca4700 con 0x560faf0b0250 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddc7e00640 10 client.26741 handle_command_reply: tid=0 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 client.26741 shutdown 2024-03-24T06:46:03.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 msgr2=0x560faf099c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdddc004af0 tx=0x7fdddc04c040 comp rx=0 tx=0).stop 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 shutdown_connections 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560faf09fd60 0x560faf09a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x560faf097180 0x560faf09a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560faf0b0250 0x560faf118e00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 --2- 172.21.15.22:0/224485148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560faf096860 0x560faf099c60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 >> 172.21.15.22:0/224485148 conn(0x560faf090f90 msgr2=0x560faf09cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddc7e00640 0 client.26741 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:03.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 shutdown_connections 2024-03-24T06:46:03.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 1 -- 172.21.15.22:0/224485148 wait complete. 2024-03-24T06:46:03.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.463+0000 7fddec1d4740 20 client.26741 trim_cache size 0 max 16384 2024-03-24T06:46:03.472 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 msgr2=0x7fddc80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 0x7fddc80a4770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x560faf096330 tx=0x7fddd4000bc0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 msgr2=0x7fdde419d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fddd400e9f0 tx=0x7fddd400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 shutdown_connections 2024-03-24T06:46:03.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fddc80a22b0 0x7fddc80a4770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde4107650 0x7fdde4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdde4101f50 0x7fdde419dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 --2- 172.21.15.22:0/1255432822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdde4101590 0x7fdde419d760 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 >> 172.21.15.22:0/1255432822 conn(0x7fdde40fd080 msgr2=0x7fdde40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 shutdown_connections 2024-03-24T06:46:03.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.464+0000 7fdde8a00640 1 -- 172.21.15.22:0/1255432822 wait complete. 2024-03-24T06:46:03.479 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:03 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4132454700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:03.481 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:46:03.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:03 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4132454700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:03 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4132454700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:03.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 -- 172.21.15.22:0/2852375082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41038b0 msgr2=0x7fdfb4103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/2852375082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41038b0 0x7fdfb4103d30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009900 tx=0x7fdf9c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 -- 172.21.15.22:0/2852375082 shutdown_connections 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/2852375082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb4104270 0x7fdfb4110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/2852375082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41038b0 0x7fdfb4103d30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/2852375082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb41098c0 0x7fdfb4109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 -- 172.21.15.22:0/2852375082 >> 172.21.15.22:0/2852375082 conn(0x7fdfb40ff360 msgr2=0x7fdfb4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 -- 172.21.15.22:0/2852375082 shutdown_connections 2024-03-24T06:46:03.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.680+0000 7fdfb9a00640 1 -- 172.21.15.22:0/2852375082 wait complete. 2024-03-24T06:46:03.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.681+0000 7fdfb9a00640 1 Processor -- start 2024-03-24T06:46:03.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 -- start start 2024-03-24T06:46:03.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb41038b0 0x7fdfb4199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 0x7fdfb419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb419edd0 con 0x7fdfb41038b0 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb419ef40 con 0x7fdfb41098c0 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb419f100 con 0x7fdfb4104270 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.682+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb41038b0 0x7fdfb4199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38480/0 (socket says 172.21.15.22:38480) 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 0x7fdfb419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb2000640 1 -- 172.21.15.22:0/1120158586 learned_addr learned my addr 172.21.15.22:0/1120158586 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 -- 172.21.15.22:0/1120158586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 msgr2=0x7fdfb419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 -- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb41038b0 msgr2=0x7fdfb4199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb41038b0 0x7fdfb4199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 -- 172.21.15.22:0/1120158586 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c009560 con 0x7fdfb41098c0 2024-03-24T06:46:03.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 0x7fdfb419e7d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00e970 tx=0x7fdfac00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.683+0000 7fdfab400640 1 -- 172.21.15.22:0/1120158586 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac009620 con 0x7fdfb41098c0 2024-03-24T06:46:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.684+0000 7fdfab400640 1 -- 172.21.15.22:0/1120158586 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfac00ccc0 con 0x7fdfb41098c0 2024-03-24T06:46:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.684+0000 7fdfab400640 1 -- 172.21.15.22:0/1120158586 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac004a00 con 0x7fdfb41098c0 2024-03-24T06:46:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.684+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb419f3e0 con 0x7fdfb41098c0 2024-03-24T06:46:03.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.684+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb41a6c60 con 0x7fdfb41098c0 2024-03-24T06:46:03.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 Processor -- start 2024-03-24T06:46:03.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfab400640 1 -- 172.21.15.22:0/1120158586 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fdfac005020 con 0x7fdfb41098c0 2024-03-24T06:46:03.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 -- start start 2024-03-24T06:46:03.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 0x56426c351660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56426c2bf6e0 con 0x56426c351ba0 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56426c2bcc10 con 0x56426c35a780 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.687+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56426c2b83f0 con 0x56426c351280 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 0x56426c351660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38966/0 (socket says 172.21.15.22:38966) 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 0x56426c351660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38484/0 (socket says 172.21.15.22:38484) 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 -- 172.21.15.22:0/3150620445 learned_addr learned my addr 172.21.15.22:0/3150620445 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2000640 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 -- 172.21.15.22:0/3150620445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 msgr2=0x56426c351660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 0x56426c351660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 -- 172.21.15.22:0/3150620445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 msgr2=0x56426c35cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.688+0000 7fdfb2a00640 1 -- 172.21.15.22:0/3150620445 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56426c279e40 con 0x56426c351ba0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfab400640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 0x7fdf980a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009fd0 tx=0x7fdfa40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9bde3053c5b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfab400640 1 -- 172.21.15.22:0/1120158586 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdfac014070 con 0x7fdfb41098c0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 0x7fdf980a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfa3e00640 1 -- 172.21.15.22:0/3150620445 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4003c30 con 0x56426c351ba0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfa3e00640 1 -- 172.21.15.22:0/3150620445 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa4003d90 con 0x56426c351ba0 2024-03-24T06:46:03.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.689+0000 7fdfa3e00640 1 -- 172.21.15.22:0/3150620445 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400fb20 con 0x56426c351ba0 2024-03-24T06:46:03.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 -- 172.21.15.22:0/3150620445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 msgr2=0x56426c35a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009fd0 tx=0x7fdfa40118d0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 0x7fdf980a4770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4108ee0 tx=0x7fdf9c02bc30 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 -- 172.21.15.22:0/3150620445 shutdown_connections 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c35a240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/3150620445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351280 0x56426c351660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 -- 172.21.15.22:0/3150620445 >> 172.21.15.22:0/3150620445 conn(0x56426c34b9b0 msgr2=0x56426c34bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 -- 172.21.15.22:0/3150620445 shutdown_connections 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.690+0000 7fdfbc64e740 1 -- 172.21.15.22:0/3150620445 wait complete. 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 -- start start 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 Processor -- start 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351ba0 0x56426c357d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c358270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56426c25d060 con 0x56426c351280 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56426c25cd00 con 0x56426c351ba0 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.691+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56426c0261c0 con 0x56426c35a780 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfbc64e740 10 client.0 _ll_register_callbacks cb 0x56426bfc8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38968/0 (socket says 172.21.15.22:38968) 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 -- 172.21.15.22:0/1986104624 learned_addr learned my addr 172.21.15.22:0/1986104624 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351ba0 0x56426c357d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c358270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 -- 172.21.15.22:0/1986104624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 msgr2=0x56426c358270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c358270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 -- 172.21.15.22:0/1986104624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351ba0 msgr2=0x56426c357d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351ba0 0x56426c357d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb2a00640 1 -- 172.21.15.22:0/1986104624 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa4009c00 con 0x56426c351280 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.692+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c358270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40099d0 tx=0x7fdfa4004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486ae3dd218145c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400fc30 con 0x56426c351280 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdfa400fd90 con 0x56426c351280 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56426bfb77e0 con 0x56426c351280 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdfa400f4e0 con 0x56426c351280 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfbc64e740 10 client.16386 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56426c2b6a70 con 0x56426c351280 2024-03-24T06:46:03.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.693+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56426bfb77e0 con 0x56426c351280 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fdfa4005500 con 0x56426c351280 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 10 client.16386 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 10 client.16386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 20 client.16386 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c36abb0 0x56426c36d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 4 client.16386 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:03.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56426bff4720 con 0x56426c36abb0 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.694+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c36abb0 0x56426c36d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.695+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c36abb0 0x56426c36d010 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e9f831c5a6bf57 server_cookie=eb9c4e3d173b9872 in_seq=0 out_seq=0 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.695+0000 7fdfa0c00640 10 client.16386 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.695+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1986104624 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56426bff4720 con 0x56426c36abb0 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.695+0000 7fdfa0c00640 10 client.16386 handle_command_reply: tid=0 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 client.16386 shutdown 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 msgr2=0x56426c3577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40099d0 tx=0x7fdfa4004cf0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 shutdown_connections 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c358270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c36abb0 0x56426c36d010 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351ba0 0x56426c357d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1986104624 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c3577f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 >> 172.21.15.22:0/1986104624 conn(0x56426c34b9b0 msgr2=0x56426c34d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfa0c00640 0 client.16386 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:03.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 shutdown_connections 2024-03-24T06:46:03.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1986104624 wait complete. 2024-03-24T06:46:03.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.696+0000 7fdfbc64e740 20 client.16386 trim_cache size 0 max 16384 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 Processor -- start 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 -- start start 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 0x56426c366010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c357d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56426c345eb0 con 0x56426c351ba0 2024-03-24T06:46:03.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56426c36a8e0 con 0x56426c351280 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.711+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56426c355d40 con 0x56426c35a780 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38980/0 (socket says 172.21.15.22:38980) 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 0x56426c366010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 -- 172.21.15.22:0/1615102722 learned_addr learned my addr 172.21.15.22:0/1615102722 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c357d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 -- 172.21.15.22:0/1615102722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 msgr2=0x56426c357d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c357d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 -- 172.21.15.22:0/1615102722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 msgr2=0x56426c366010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 0x56426c366010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 -- 172.21.15.22:0/1615102722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56426c252b00 con 0x56426c351ba0 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 0x56426c366010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.712+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02c240 tx=0x7fdf9c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7edc3a515eee80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.713+0000 7fdfa3400640 1 -- 172.21.15.22:0/1615102722 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c02dea0 con 0x56426c351ba0 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.713+0000 7fdfa3400640 1 -- 172.21.15.22:0/1615102722 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c0056a0 con 0x56426c351ba0 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.713+0000 7fdfa3400640 1 -- 172.21.15.22:0/1615102722 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c003eb0 con 0x56426c351ba0 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.713+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1615102722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 msgr2=0x56426c372920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.713+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02c240 tx=0x7fdf9c005a20 comp rx=0 tx=0).stop 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1615102722 shutdown_connections 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c35a780 0x56426c357d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351ba0 0x56426c372920 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1615102722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c351280 0x56426c366010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1615102722 >> 172.21.15.22:0/1615102722 conn(0x56426c34b9b0 msgr2=0x56426c34aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1615102722 shutdown_connections 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1615102722 wait complete. 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 -- start start 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.714+0000 7fdfbc64e740 1 Processor -- start 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56426c362e80 con 0x56426c351280 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56426c3cff50 con 0x56426c35a780 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56426c3d00c0 con 0x56426c351ba0 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfbc64e740 10 client.0 _ll_register_callbacks cb 0x7fdfa401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38510/0 (socket says 172.21.15.22:38510) 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 -- 172.21.15.22:0/1585944666 learned_addr learned my addr 172.21.15.22:0/1585944666 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:03.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38986/0 (socket says 172.21.15.22:38986) 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 -- 172.21.15.22:0/1585944666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 msgr2=0x56426c35e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 -- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 msgr2=0x56426c35deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2000640 1 -- 172.21.15.22:0/1585944666 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56426c2b6a70 con 0x56426c351ba0 2024-03-24T06:46:03.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb3400640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.715+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfb2000640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c003a30 tx=0x7fdf9c02cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0a7ad1cde47849 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0053c0 con 0x56426c351ba0 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfa4003d90 con 0x56426c351ba0 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fdf9c04a040 con 0x56426c351ba0 2024-03-24T06:46:03.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.716+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdf9c005520 con 0x56426c351ba0 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 10 client.26779 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x7fdfb4111080 con 0x56426c351ba0 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa4003d90 con 0x56426c351ba0 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7fdf9c068080 con 0x56426c351ba0 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 10 client.26779 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 10 client.26779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 20 client.26779 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:03.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c3d15a0 0x56426c3d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 4 client.26779 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56426c043890 con 0x56426c3d15a0 2024-03-24T06:46:03.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.717+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c3d15a0 0x56426c3d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:03.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfb2a00640 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c3d15a0 0x56426c3d39c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7badfb1d47cc7623 server_cookie=cfa3d6ac5c39b8d7 in_seq=0 out_seq=0 2024-03-24T06:46:03.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfa0c00640 10 client.26779 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfa0c00640 1 -- 172.21.15.22:0/1585944666 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56426c043890 con 0x56426c3d15a0 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfa0c00640 10 client.26779 handle_command_reply: tid=0 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 client.26779 shutdown 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 msgr2=0x56426c35e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c003a30 tx=0x7fdf9c02cc30 comp rx=0 tx=0).stop 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 shutdown_connections 2024-03-24T06:46:03.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56426c3d15a0 0x56426c3d39c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56426c35a780 0x56426c35e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56426c351ba0 0x56426c35e3f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfa0c00640 0 client.26779 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 --2- 172.21.15.22:0/1585944666 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56426c351280 0x56426c35deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.718+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 >> 172.21.15.22:0/1585944666 conn(0x56426c34b9b0 msgr2=0x56426c34caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 shutdown_connections 2024-03-24T06:46:03.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfbc64e740 1 -- 172.21.15.22:0/1585944666 wait complete. 2024-03-24T06:46:03.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfbc64e740 20 client.26779 trim_cache size 0 max 16384 2024-03-24T06:46:03.729 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:03.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 msgr2=0x7fdf980a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 0x7fdf980a4770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4108ee0 tx=0x7fdf9c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 msgr2=0x7fdfb419e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.719+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 0x7fdfb419e7d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00e970 tx=0x7fdfac00ee30 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 shutdown_connections 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fdf980a22b0 0x7fdf980a4770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfb41098c0 0x7fdfb419e7d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfb4104270 0x7fdfb419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 --2- 172.21.15.22:0/1120158586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdfb41038b0 0x7fdfb4199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:03.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 >> 172.21.15.22:0/1120158586 conn(0x7fdfb40ff360 msgr2=0x7fdfb40ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:03.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 shutdown_connections 2024-03-24T06:46:03.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:03.720+0000 7fdfb9a00640 1 -- 172.21.15.22:0/1120158586 wait complete. 2024-03-24T06:46:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:04 smithi155 ceph-mon[36262]: pgmap v186: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-24T06:46:04.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:04 smithi022 ceph-mon[32003]: pgmap v186: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-24T06:46:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:04 smithi143 ceph-mon[36520]: pgmap v186: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-24T06:46:04.739 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:46:04.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:46:04.739 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:46:04.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 -- 172.21.15.22:0/1062377063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 msgr2=0x7f4140106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:04.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 --2- 172.21.15.22:0/1062377063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f4140106930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4134009930 tx=0x7f413402bc20 comp rx=0 tx=0).stop 2024-03-24T06:46:04.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 -- 172.21.15.22:0/1062377063 shutdown_connections 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 --2- 172.21.15.22:0/1062377063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100e50 0x7f414010d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 --2- 172.21.15.22:0/1062377063 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100490 0x7f4140100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 --2- 172.21.15.22:0/1062377063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f4140106930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.982+0000 7f4146c00640 1 -- 172.21.15.22:0/1062377063 >> 172.21.15.22:0/1062377063 conn(0x7f41400fbf60 msgr2=0x7f41400fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.983+0000 7f4146c00640 1 -- 172.21.15.22:0/1062377063 shutdown_connections 2024-03-24T06:46:04.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.983+0000 7f4146c00640 1 -- 172.21.15.22:0/1062377063 wait complete. 2024-03-24T06:46:04.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 Processor -- start 2024-03-24T06:46:04.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 -- start start 2024-03-24T06:46:04.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 0x7f4140199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100e50 0x7f414019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f414019ec30 con 0x7f4140100e50 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f414019eda0 con 0x7f4140106550 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.984+0000 7f4146c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f414019ef60 con 0x7f4140100490 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 0x7f4140199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:04.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:33426/0 (socket says 172.21.15.22:33426) 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 -- 172.21.15.22:0/2891076377 learned_addr learned my addr 172.21.15.22:0/2891076377 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 0x7f4140199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45384/0 (socket says 172.21.15.22:45384) 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4144e00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100e50 0x7f414019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 -- 172.21.15.22:0/2891076377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 msgr2=0x7f4140199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 0x7f4140199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 -- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100e50 msgr2=0x7f414019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100e50 0x7f414019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:04.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 -- 172.21.15.22:0/2891076377 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4138009840 con 0x7f4140106550 2024-03-24T06:46:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f4146200640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f413800efd0 tx=0x7f413800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.985+0000 7f413e000640 1 -- 172.21.15.22:0/2891076377 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138009b40 con 0x7f4140106550 2024-03-24T06:46:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.986+0000 7f413e000640 1 -- 172.21.15.22:0/2891076377 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4138003b70 con 0x7f4140106550 2024-03-24T06:46:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.986+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4134009560 con 0x7f4140106550 2024-03-24T06:46:04.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.986+0000 7f413e000640 1 -- 172.21.15.22:0/2891076377 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41380108f0 con 0x7f4140106550 2024-03-24T06:46:04.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.986+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41401a68e0 con 0x7f4140106550 2024-03-24T06:46:04.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.990+0000 7f414ae77740 1 Processor -- start 2024-03-24T06:46:04.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.990+0000 7f413e000640 1 -- 172.21.15.22:0/2891076377 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f4138003ce0 con 0x7f4140106550 2024-03-24T06:46:04.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 -- start start 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e32b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e336920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c06e2992d0 con 0x55c06e334530 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c06e296800 con 0x55c06e32b030 2024-03-24T06:46:04.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f414ae77740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c06e291fe0 con 0x55c06e32b950 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f4144e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e32b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45394/0 (socket says 172.21.15.22:45394) 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f4145800640 1 -- 172.21.15.22:0/3754445212 learned_addr learned my addr 172.21.15.22:0/3754445212 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.991+0000 7f4144e00640 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e336920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 -- 172.21.15.22:0/3754445212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 msgr2=0x55c06e32b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e32b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 -- 172.21.15.22:0/3754445212 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 msgr2=0x55c06e336920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e336920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 -- 172.21.15.22:0/3754445212 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06e253e90 con 0x55c06e32b950 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4145800640 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4134005ab0 tx=0x7f41340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa2679bcdf6da35b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4132a00640 1 -- 172.21.15.22:0/3754445212 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413402dea0 con 0x55c06e32b950 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f413e000640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 0x7f41280a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4132a00640 1 -- 172.21.15.22:0/3754445212 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41340048b0 con 0x55c06e32b950 2024-03-24T06:46:05.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.992+0000 7f4132a00640 1 -- 172.21.15.22:0/3754445212 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134005050 con 0x55c06e32b950 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f413e000640 1 -- 172.21.15.22:0/2891076377 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4138014070 con 0x7f4140106550 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f4144e00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 0x7f41280a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 -- 172.21.15.22:0/3754445212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 msgr2=0x55c06e333ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4134005ab0 tx=0x7f41340046c0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 -- 172.21.15.22:0/3754445212 shutdown_connections 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e336920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e333ff0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 --2- 172.21.15.22:0/3754445212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e32b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 -- 172.21.15.22:0/3754445212 >> 172.21.15.22:0/3754445212 conn(0x55c06e325760 msgr2=0x55c06e325b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f4144e00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 0x7f41280a4770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55c06e3268f0 tx=0x7f412c00b390 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.993+0000 7f414ae77740 1 -- 172.21.15.22:0/3754445212 shutdown_connections 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.994+0000 7f414ae77740 1 -- 172.21.15.22:0/3754445212 wait complete. 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.994+0000 7f414ae77740 1 -- start start 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.994+0000 7f414ae77740 1 Processor -- start 2024-03-24T06:46:05.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e331a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e331fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c06e236fc0 con 0x55c06e334530 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c06e236c60 con 0x55c06e32b950 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c06dfda0c0 con 0x55c06e32b030 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f414ae77740 10 client.0 _ll_register_callbacks cb 0x55c06e24ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:05.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f4144e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e331a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f4145800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45404/0 (socket says 172.21.15.22:45404) 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f4145800640 1 -- 172.21.15.22:0/3790740903 learned_addr learned my addr 172.21.15.22:0/3790740903 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.995+0000 7f4146200640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e331fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 -- 172.21.15.22:0/3790740903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 msgr2=0x55c06e331a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e331a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 -- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 msgr2=0x55c06e331fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e331fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 -- 172.21.15.22:0/3790740903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414010dc10 con 0x55c06e32b030 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4145800640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4134036040 tx=0x7f413402cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d850008f777f3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4127400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340050c0 con 0x55c06e32b030 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c06e2b6e90 con 0x55c06e32b030 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4134005220 con 0x55c06e32b030 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f41340053e0 con 0x55c06e32b030 2024-03-24T06:46:05.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f414ae77740 10 client.26791 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.996+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c06e290660 con 0x55c06e32b030 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c06e2b6e90 con 0x55c06e32b030 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f413403c050 con 0x55c06e32b030 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 10 client.26791 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 10 client.26791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 20 client.26791 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:05.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e344640 0x55c06e346a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 4 client.26791 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c06e288210 con 0x55c06e344640 2024-03-24T06:46:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.997+0000 7f4144e00640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e344640 0x55c06e346a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.998+0000 7f4144e00640 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e344640 0x55c06e346a30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c39e7f02a6d2991 server_cookie=3be7941b51bba332 in_seq=0 out_seq=0 2024-03-24T06:46:05.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.998+0000 7f4127400640 10 client.26791 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f4127400640 1 -- 172.21.15.22:0/3790740903 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c06e288210 con 0x55c06e344640 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f4127400640 10 client.26791 handle_command_reply: tid=0 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 client.26791 shutdown 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 msgr2=0x55c06e331530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4134036040 tx=0x7f413402cd20 comp rx=0 tx=0).stop 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 shutdown_connections 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e331fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e344640 0x55c06e346a30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e331a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 --2- 172.21.15.22:0/3790740903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b030 0x55c06e331530 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 >> 172.21.15.22:0/3790740903 conn(0x55c06e325760 msgr2=0x55c06e3274b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f4127400640 0 client.26791 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 shutdown_connections 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:04.999+0000 7f414ae77740 1 -- 172.21.15.22:0/3790740903 wait complete. 2024-03-24T06:46:05.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.000+0000 7f414ae77740 20 client.26791 trim_cache size 0 max 16384 2024-03-24T06:46:05.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 Processor -- start 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 -- start start 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e334530 0x55c06e324640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c06e326760 con 0x55c06e32b030 2024-03-24T06:46:05.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c06e3413a0 con 0x55c06e32b950 2024-03-24T06:46:05.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.019+0000 7f414ae77740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c06e336920 con 0x55c06e334530 2024-03-24T06:46:05.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4145800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:33460/0 (socket says 172.21.15.22:33460) 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 -- 172.21.15.22:0/804197922 learned_addr learned my addr 172.21.15.22:0/804197922 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4145800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51846/0 (socket says 172.21.15.22:51846) 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4146200640 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e334530 0x55c06e324640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 -- 172.21.15.22:0/804197922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e334530 msgr2=0x55c06e324640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e334530 0x55c06e324640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 -- 172.21.15.22:0/804197922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 msgr2=0x55c06e333c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 -- 172.21.15.22:0/804197922 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c06e22cae0 con 0x55c06e32b950 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4145800640 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.020+0000 7f4144e00640 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55c06e330d90 tx=0x7f412c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd10533641506480 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f4132000640 1 -- 172.21.15.22:0/804197922 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c025040 con 0x55c06e32b950 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f4132000640 1 -- 172.21.15.22:0/804197922 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f412c02bd90 con 0x55c06e32b950 2024-03-24T06:46:05.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f4132000640 1 -- 172.21.15.22:0/804197922 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c01fcd0 con 0x55c06e32b950 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 -- 172.21.15.22:0/804197922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 msgr2=0x55c06e33fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55c06e330d90 tx=0x7f412c0046c0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 -- 172.21.15.22:0/804197922 shutdown_connections 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e334530 0x55c06e324640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b950 0x55c06e33fd60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 --2- 172.21.15.22:0/804197922 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e32b030 0x55c06e333c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 -- 172.21.15.22:0/804197922 >> 172.21.15.22:0/804197922 conn(0x55c06e325760 msgr2=0x55c06e323f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.021+0000 7f414ae77740 1 -- 172.21.15.22:0/804197922 shutdown_connections 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.022+0000 7f414ae77740 1 -- 172.21.15.22:0/804197922 wait complete. 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.022+0000 7f414ae77740 1 -- start start 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.022+0000 7f414ae77740 1 Processor -- start 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e3389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e3a9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c06e3a9e70 con 0x55c06e334530 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c06e3a9fb0 con 0x55c06e32b030 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c06e3aa0f0 con 0x55c06e32b950 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f414ae77740 10 client.0 _ll_register_callbacks cb 0x7f412c00ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4144e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e3389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:33466/0 (socket says 172.21.15.22:33466) 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 -- 172.21.15.22:0/4099277358 learned_addr learned my addr 172.21.15.22:0/4099277358 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4146200640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e3a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 -- 172.21.15.22:0/4099277358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 msgr2=0x55c06e3389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e3389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 -- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 msgr2=0x55c06e3a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e3a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4145800640 1 -- 172.21.15.22:0/4099277358 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c06e290660 con 0x55c06e32b030 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4144e00640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e3389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.023+0000 7f4146200640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e3a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4145800640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4134036040 tx=0x7f4134005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25eaab050b59a8a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4127400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413402cb50 con 0x55c06e32b030 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41340048b0 con 0x55c06e32b030 2024-03-24T06:46:05.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f4134000e40 con 0x55c06e32b030 2024-03-24T06:46:05.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f413403f640 con 0x55c06e32b030 2024-03-24T06:46:05.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f414ae77740 10 client.26777 fetch_fsmap learned FSMap version 36 2024-03-24T06:46:05.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x7f412c012450 con 0x55c06e32b030 2024-03-24T06:46:05.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.024+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41340048b0 con 0x55c06e32b030 2024-03-24T06:46:05.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 36) ==== 8366+0+0 (secure 0 0 0) 0x7f413403c070 con 0x55c06e32b030 2024-03-24T06:46:05.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 10 client.26777 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T06:46:05.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 10 client.26777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 20 client.26777 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e3ab580 0x55c06e3ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 4 client.26777 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c06df91860 con 0x55c06e3ab580 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.025+0000 7f4144e00640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e3ab580 0x55c06e3ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.026+0000 7f4144e00640 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e3ab580 0x55c06e3ad9a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06bb4e79cb78392 server_cookie=91ecbbde52938b0b in_seq=0 out_seq=0 2024-03-24T06:46:05.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.026+0000 7f4127400640 10 client.26777 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.026+0000 7f4127400640 1 -- 172.21.15.22:0/4099277358 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c06df91860 con 0x55c06e3ab580 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.026+0000 7f4127400640 10 client.26777 handle_command_reply: tid=0 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.026+0000 7f414ae77740 1 client.26777 shutdown 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 msgr2=0x55c06e338460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4134036040 tx=0x7f4134005b50 comp rx=0 tx=0).stop 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 shutdown_connections 2024-03-24T06:46:05.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c06e334530 0x55c06e3a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c06e32b950 0x55c06e3389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c06e3ab580 0x55c06e3ad9a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 --2- 172.21.15.22:0/4099277358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c06e32b030 0x55c06e338460 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 >> 172.21.15.22:0/4099277358 conn(0x55c06e325760 msgr2=0x55c06e3247c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f4127400640 0 client.26777 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 shutdown_connections 2024-03-24T06:46:05.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 1 -- 172.21.15.22:0/4099277358 wait complete. 2024-03-24T06:46:05.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.027+0000 7f414ae77740 20 client.26777 trim_cache size 0 max 16384 2024-03-24T06:46:05.038 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:05.038 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:46:05.038 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "92753aca-a0f6-47c8-9537-1aabe8b972c0", 2024-03-24T06:46:05.038 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:46:05.039 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:05.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.028+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 msgr2=0x7f41280a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.028+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 0x7f41280a4770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55c06e3268f0 tx=0x7f412c00b390 comp rx=0 tx=0).stop 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.028+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 msgr2=0x7f414019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.028+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f413800efd0 tx=0x7f413800c6a0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 shutdown_connections 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f41280a22b0 0x7f41280a4770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4140106550 0x7f414019e630 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4140100e50 0x7f414019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 --2- 172.21.15.22:0/2891076377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4140100490 0x7f4140199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 >> 172.21.15.22:0/2891076377 conn(0x7f41400fbf60 msgr2=0x7f41400fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 shutdown_connections 2024-03-24T06:46:05.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.029+0000 7f4146c00640 1 -- 172.21.15.22:0/2891076377 wait complete. 2024-03-24T06:46:05.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:46:05.052 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:46:05.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.286+0000 7f03b1c00640 1 -- 172.21.15.22:0/589561816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac100530 msgr2=0x7f03ac100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.286+0000 7f03b1c00640 1 --2- 172.21.15.22:0/589561816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac100530 0x7f03ac100910 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f039c009900 tx=0x7f039c02bd30 comp rx=0 tx=0).stop 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 -- 172.21.15.22:0/589561816 shutdown_connections 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 --2- 172.21.15.22:0/589561816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac105930 0x7f03ac105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 --2- 172.21.15.22:0/589561816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100e50 0x7f03ac105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 --2- 172.21.15.22:0/589561816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac100530 0x7f03ac100910 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 -- 172.21.15.22:0/589561816 >> 172.21.15.22:0/589561816 conn(0x7f03ac0fbf60 msgr2=0x7f03ac0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 -- 172.21.15.22:0/589561816 shutdown_connections 2024-03-24T06:46:05.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.287+0000 7f03b1c00640 1 -- 172.21.15.22:0/589561816 wait complete. 2024-03-24T06:46:05.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 Processor -- start 2024-03-24T06:46:05.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 -- start start 2024-03-24T06:46:05.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100530 0x7f03ac19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac105930 0x7f03ac1a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a30b0 con 0x7f03ac100530 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a3220 con 0x7f03ac105930 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.288+0000 7f03b1c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a33e0 con 0x7f03ac100e50 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03abe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100530 0x7f03ac19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45442/0 (socket says 172.21.15.22:45442) 2024-03-24T06:46:05.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 -- 172.21.15.22:0/3052343611 learned_addr learned my addr 172.21.15.22:0/3052343611 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03b1200640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac105930 0x7f03ac1a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 -- 172.21.15.22:0/3052343611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac105930 msgr2=0x7f03ac1a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac105930 0x7f03ac1a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 -- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100530 msgr2=0x7f03ac19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100530 0x7f03ac19e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.289+0000 7f03a3e00640 1 -- 172.21.15.22:0/3052343611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c009560 con 0x7f03ac100e50 2024-03-24T06:46:05.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f039400e9f0 tx=0x7f039400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03a9600640 1 -- 172.21.15.22:0/3052343611 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394009650 con 0x7f03ac100e50 2024-03-24T06:46:05.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03a9600640 1 -- 172.21.15.22:0/3052343611 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039400ccc0 con 0x7f03ac100e50 2024-03-24T06:46:05.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac1a36c0 con 0x7f03ac100e50 2024-03-24T06:46:05.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03a9600640 1 -- 172.21.15.22:0/3052343611 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394018860 con 0x7f03ac100e50 2024-03-24T06:46:05.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.290+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac1aaf40 con 0x7f03ac100e50 2024-03-24T06:46:05.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.293+0000 7f03a9600640 1 -- 172.21.15.22:0/3052343611 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0394010aa0 con 0x7f03ac100e50 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.293+0000 7f03b5fa1740 1 Processor -- start 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 -- start start 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x55620612a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5562061342d0 0x5562061366c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556206099790 con 0x5562061342d0 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556206096cc0 con 0x55620612af60 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5562060924a0 con 0x55620612bc40 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03abe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03abe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:33492/0 (socket says 172.21.15.22:33492) 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03abe00640 1 -- 172.21.15.22:0/3897897926 learned_addr learned my addr 172.21.15.22:0/3897897926 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03b1200640 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x55620612a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03a9600640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 0x7f03900a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.294+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5562061342d0 0x5562061366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 -- 172.21.15.22:0/3897897926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 msgr2=0x55620612a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x55620612a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 -- 172.21.15.22:0/3897897926 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5562061342d0 msgr2=0x5562061366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5562061342d0 0x5562061366c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 -- 172.21.15.22:0/3897897926 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556206053c20 con 0x55620612af60 2024-03-24T06:46:05.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03a9600640 1 -- 172.21.15.22:0/3052343611 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f0394014070 con 0x7f03ac100e50 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03b1200640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 0x7f03900a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03abe00640 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f039c000c00 tx=0x7f039c02c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d57ed2199c46896 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03a1600640 1 -- 172.21.15.22:0/3897897926 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c004880 con 0x55620612af60 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03a1600640 1 -- 172.21.15.22:0/3897897926 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c0049e0 con 0x55620612af60 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.295+0000 7f03a1600640 1 -- 172.21.15.22:0/3897897926 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c005050 con 0x55620612af60 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3897897926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 msgr2=0x55620612b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f039c000c00 tx=0x7f039c02c710 comp rx=0 tx=0).stop 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b1200640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 0x7f03900a4770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f03a4005fd0 tx=0x7f03a4014040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3897897926 shutdown_connections 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5562061342d0 0x5562061366c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612af60 0x55620612b3e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3897897926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x55620612a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3897897926 >> 172.21.15.22:0/3897897926 conn(0x5562060f1b30 msgr2=0x5562061259b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3897897926 shutdown_connections 2024-03-24T06:46:05.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.296+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3897897926 wait complete. 2024-03-24T06:46:05.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.297+0000 7f03b5fa1740 1 -- start start 2024-03-24T06:46:05.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.297+0000 7f03b5fa1740 1 Processor -- start 2024-03-24T06:46:05.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 0x556206131ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556206036f20 con 0x55620612af60 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556206036bc0 con 0x5562061342d0 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556205e001c0 con 0x55620612bc40 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b5fa1740 10 client.0 _ll_register_callbacks cb 0x556205da2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03abe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45458/0 (socket says 172.21.15.22:45458) 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03abe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51878/0 (socket says 172.21.15.22:51878) 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 -- 172.21.15.22:0/2033502183 learned_addr learned my addr 172.21.15.22:0/2033502183 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03b1200640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 0x556206131ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 -- 172.21.15.22:0/2033502183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 msgr2=0x556206131ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 0x556206131ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 -- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 msgr2=0x556206131260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.298+0000 7f03a3e00640 1 -- 172.21.15.22:0/2033502183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac115290 con 0x55620612bc40 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03b1200640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 0x556206131ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03abe00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03a3e00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0394097150 tx=0x7f039400cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1567b21fc02034dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f039a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:05.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03940cbe70 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55620608a160 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f03940ce060 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f039401ae80 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03b5fa1740 10 client.26809 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556206090b20 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.299+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55620608a160 con 0x55620612bc40 2024-03-24T06:46:05.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f039409a200 con 0x55620612bc40 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 10 client.26809 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 10 client.26809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 20 client.26809 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556206144710 0x556206146b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 4 client.26809 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556205d917e0 con 0x556206144710 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.300+0000 7f03abe00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556206144710 0x556206146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.301+0000 7f03abe00640 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556206144710 0x556206146b00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636b03007f79cdde server_cookie=3cb80cfcbbd9f9b7 in_seq=0 out_seq=0 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.301+0000 7f039a000640 10 client.26809 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.301+0000 7f039a000640 1 -- 172.21.15.22:0/2033502183 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556205d917e0 con 0x556206144710 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.301+0000 7f039a000640 10 client.26809 handle_command_reply: tid=0 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.301+0000 7f03b5fa1740 1 client.26809 shutdown 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 msgr2=0x5562061317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0394097150 tx=0x7f039400cef0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 shutdown_connections 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556206144710 0x556206146b00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5562061342d0 0x556206131ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55620612bc40 0x5562061317a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/2033502183 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x556206131260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f039a000640 0 client.26809 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 >> 172.21.15.22:0/2033502183 conn(0x5562060f1b30 msgr2=0x556206127700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 shutdown_connections 2024-03-24T06:46:05.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 1 -- 172.21.15.22:0/2033502183 wait complete. 2024-03-24T06:46:05.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.302+0000 7f03b5fa1740 20 client.26809 trim_cache size 0 max 16384 2024-03-24T06:46:05.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.329+0000 7f03b5fa1740 1 Processor -- start 2024-03-24T06:46:05.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 -- start start 2024-03-24T06:46:05.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x5562061301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5562061215f0 con 0x55620612af60 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556206129f90 con 0x55620612bc40 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556206121270 con 0x5562061342d0 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45472/0 (socket says 172.21.15.22:45472) 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03b1200640 1 -- 172.21.15.22:0/3651891162 learned_addr learned my addr 172.21.15.22:0/3651891162 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.330+0000 7f03abe00640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x5562061301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 -- 172.21.15.22:0/3651891162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 msgr2=0x55620612c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 -- 172.21.15.22:0/3651891162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 msgr2=0x5562061301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x5562061301e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 -- 172.21.15.22:0/3651891162 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620602c9a0 con 0x5562061342d0 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03b1200640 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009ee0 tx=0x7f03a4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772359878709fa1e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03a0c00640 1 -- 172.21.15.22:0/3651891162 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4023070 con 0x5562061342d0 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03a0c00640 1 -- 172.21.15.22:0/3651891162 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a4010b30 con 0x5562061342d0 2024-03-24T06:46:05.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.331+0000 7f03a0c00640 1 -- 172.21.15.22:0/3651891162 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a401b450 con 0x5562061342d0 2024-03-24T06:46:05.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3651891162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 msgr2=0x55620613fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009ee0 tx=0x7f03a4016000 comp rx=0 tx=0).stop 2024-03-24T06:46:05.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3651891162 shutdown_connections 2024-03-24T06:46:05.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x55620613fbf0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3651891162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x5562061301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3651891162 >> 172.21.15.22:0/3651891162 conn(0x5562060f1b30 msgr2=0x556206124bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3651891162 shutdown_connections 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3651891162 wait complete. 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.332+0000 7f03b5fa1740 1 -- start start 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 Processor -- start 2024-03-24T06:46:05.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x55620612e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55620612f200 con 0x55620612af60 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5562061a9f80 con 0x55620612bc40 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5562061aa140 con 0x5562061342d0 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b5fa1740 10 client.0 _ll_register_callbacks cb 0x7f03940968c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03a3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:33522/0 (socket says 172.21.15.22:33522) 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03a3e00640 1 -- 172.21.15.22:0/3806512999 learned_addr learned my addr 172.21.15.22:0/3806512999 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03abe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x55620612e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.333+0000 7f03b1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:45474/0 (socket says 172.21.15.22:45474) 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 -- 172.21.15.22:0/3806512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 msgr2=0x55620612ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 -- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 msgr2=0x55620612e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x55620612e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 -- 172.21.15.22:0/3806512999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a4004640 con 0x5562061342d0 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03a3e00640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03abe00640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x55620612e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f03b1200640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f03a40108c0 tx=0x7f03a401d6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd2487159db2c1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f039a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:05.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.334+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4023040 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55620612cc00 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f03a4028820 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f03a4005ac0 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 10 client.26815 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556206052480 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a4038080 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f03a401baa0 con 0x5562061342d0 2024-03-24T06:46:05.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 10 client.26815 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 10 client.26815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 20 client.26815 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5562061ab920 0x5562061add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 4 client.26815 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.335+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f039c0049e0 con 0x5562061ab920 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.336+0000 7f03abe00640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5562061ab920 0x5562061add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.336+0000 7f03abe00640 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5562061ab920 0x5562061add40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4801a3f0275e83d5 server_cookie=9830d628fbb74a81 in_seq=0 out_seq=0 2024-03-24T06:46:05.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.336+0000 7f039a000640 10 client.26815 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f039a000640 1 -- 172.21.15.22:0/3806512999 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f039c0049e0 con 0x5562061ab920 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f039a000640 10 client.26815 handle_command_reply: tid=0 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 client.26815 shutdown 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 msgr2=0x5562061a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f03a40108c0 tx=0x7f03a401d6c0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 shutdown_connections 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562061342d0 0x5562061a9a40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5562061ab920 0x5562061add40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55620612bc40 0x55620612ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 --2- 172.21.15.22:0/3806512999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55620612af60 0x55620612e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f039a000640 0 client.26815 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 >> 172.21.15.22:0/3806512999 conn(0x5562060f1b30 msgr2=0x556206126ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 shutdown_connections 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 1 -- 172.21.15.22:0/3806512999 wait complete. 2024-03-24T06:46:05.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.337+0000 7f03b5fa1740 20 client.26815 trim_cache size 0 max 16384 2024-03-24T06:46:05.348 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:46:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:46:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "92753aca-a0f6-47c8-9537-1aabe8b972c0": { 2024-03-24T06:46:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:46:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "92753aca-a0f6-47c8-9537-1aabe8b972c0", 2024-03-24T06:46:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:46:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:46:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:46:05.349 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:05.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 msgr2=0x7f03900a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 0x7f03900a4770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f03a4005fd0 tx=0x7f03a4014040 comp rx=0 tx=0).stop 2024-03-24T06:46:05.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 msgr2=0x7f03ac19e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:05.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f039400e9f0 tx=0x7f039400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 shutdown_connections 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03900a22b0 0x7f03900a4770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03ac105930 0x7f03ac1a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03ac100e50 0x7f03ac19e780 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 --2- 172.21.15.22:0/3052343611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03ac100530 0x7f03ac19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 >> 172.21.15.22:0/3052343611 conn(0x7f03ac0fbf60 msgr2=0x7f03ac1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 shutdown_connections 2024-03-24T06:46:05.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:05.338+0000 7f03b1c00640 1 -- 172.21.15.22:0/3052343611 wait complete. 2024-03-24T06:46:05.356 INFO:tasks.cephfs.filesystem:scrub status for tag:92753aca-a0f6-47c8-9537-1aabe8b972c0 - {'path': '/', 'tag': '92753aca-a0f6-47c8-9537-1aabe8b972c0', 'options': 'recursive,force'} 2024-03-24T06:46:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:05 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:05.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:05 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:05 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:05.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:05 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:05.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:05 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:05.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:05 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: pgmap v187: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-24T06:46:06.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:46:06.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:06.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:06 smithi155 ceph-mon[36262]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:46:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: pgmap v187: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:06.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:06 smithi022 ceph-mon[32003]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:46:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: pgmap v187: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:06.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:06.681 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:06 smithi143 ceph-mon[36520]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T06:46:07.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:07 smithi022 ceph-mon[32003]: pgmap v188: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 KiB/s wr, 27 op/s 2024-03-24T06:46:07.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:07 smithi143 ceph-mon[36520]: pgmap v188: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 KiB/s wr, 27 op/s 2024-03-24T06:46:07.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:07 smithi155 ceph-mon[36262]: pgmap v188: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 KiB/s wr, 27 op/s 2024-03-24T06:46:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:09 smithi155 ceph-mon[36262]: pgmap v189: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 24 op/s 2024-03-24T06:46:10.351 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:09 smithi022 ceph-mon[32003]: pgmap v189: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 24 op/s 2024-03-24T06:46:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:09 smithi143 ceph-mon[36520]: pgmap v189: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 KiB/s wr, 24 op/s 2024-03-24T06:46:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:11 smithi155 ceph-mon[36262]: pgmap v190: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:12.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:11 smithi022 ceph-mon[32003]: pgmap v190: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:11 smithi143 ceph-mon[36520]: pgmap v190: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:13 smithi155 ceph-mon[36262]: pgmap v191: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 28 op/s 2024-03-24T06:46:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:13 smithi022 ceph-mon[32003]: pgmap v191: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 28 op/s 2024-03-24T06:46:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:13 smithi143 ceph-mon[36520]: pgmap v191: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 28 op/s 2024-03-24T06:46:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:15 smithi155 ceph-mon[36262]: pgmap v192: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:15 smithi022 ceph-mon[32003]: pgmap v192: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:15 smithi143 ceph-mon[36520]: pgmap v192: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 KiB/s wr, 26 op/s 2024-03-24T06:46:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:17 smithi155 ceph-mon[36262]: pgmap v193: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-24T06:46:18.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:17 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:18.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:17 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:18.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:17 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:17 smithi022 ceph-mon[32003]: pgmap v193: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-24T06:46:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:17 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:17 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:17 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:17 smithi143 ceph-mon[36520]: pgmap v193: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 14 op/s 2024-03-24T06:46:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:17 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:17 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:17 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:19.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:18 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:19.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:18 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:19.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:18 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:19.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:18 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:19.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:18 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:19.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:18 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:20.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:20 smithi155 ceph-mon[36262]: pgmap v194: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 10 op/s 2024-03-24T06:46:20.329 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:20 smithi143 ceph-mon[36520]: pgmap v194: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 10 op/s 2024-03-24T06:46:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:20 smithi022 ceph-mon[32003]: pgmap v194: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 10 op/s 2024-03-24T06:46:22.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:22 smithi155 ceph-mon[36262]: pgmap v195: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-03-24T06:46:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:22 smithi022 ceph-mon[32003]: pgmap v195: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-03-24T06:46:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:22 smithi143 ceph-mon[36520]: pgmap v195: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 KiB/s wr, 11 op/s 2024-03-24T06:46:24.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:24 smithi155 ceph-mon[36262]: pgmap v196: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-03-24T06:46:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:24 smithi022 ceph-mon[32003]: pgmap v196: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-03-24T06:46:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:24 smithi143 ceph-mon[36520]: pgmap v196: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.8 KiB/s wr, 13 op/s 2024-03-24T06:46:26.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:26 smithi155 ceph-mon[36262]: pgmap v197: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-24T06:46:26.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:26 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:46:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:26 smithi022 ceph-mon[32003]: pgmap v197: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-24T06:46:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:26 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:46:26.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:26 smithi143 ceph-mon[36520]: pgmap v197: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 10 op/s 2024-03-24T06:46:26.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:26 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:46:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:46:27.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:46:27.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:27 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:46:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:46:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:46:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:27 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:46:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:46:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:46:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:27 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:46:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:28 smithi143 ceph-mon[36520]: pgmap v198: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-24T06:46:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:28 smithi155 ceph-mon[36262]: pgmap v198: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-24T06:46:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:28 smithi022 ceph-mon[32003]: pgmap v198: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-24T06:46:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:30 smithi143 ceph-mon[36520]: pgmap v199: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:30 smithi155 ceph-mon[36262]: pgmap v199: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:30.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:30 smithi022 ceph-mon[32003]: pgmap v199: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:32 smithi143 ceph-mon[36520]: pgmap v200: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:32 smithi155 ceph-mon[36262]: pgmap v200: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:32 smithi022 ceph-mon[32003]: pgmap v200: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:34 smithi143 ceph-mon[36520]: pgmap v201: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T06:46:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:34 smithi155 ceph-mon[36262]: pgmap v201: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T06:46:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:34 smithi022 ceph-mon[32003]: pgmap v201: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T06:46:35.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:46:35.359 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:46:35.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 -- 172.21.15.22:0/2256542029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 msgr2=0x7f1be0107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 --2- 172.21.15.22:0/2256542029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0107a40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009930 tx=0x7f1bd002bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:35.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 -- 172.21.15.22:0/2256542029 shutdown_connections 2024-03-24T06:46:35.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 --2- 172.21.15.22:0/2256542029 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 0x7f1be010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 --2- 172.21.15.22:0/2256542029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 --2- 172.21.15.22:0/2256542029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0107a40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 -- 172.21.15.22:0/2256542029 >> 172.21.15.22:0/2256542029 conn(0x7f1be00fd070 msgr2=0x7f1be00ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 -- 172.21.15.22:0/2256542029 shutdown_connections 2024-03-24T06:46:35.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.596+0000 7f1bdf200640 1 -- 172.21.15.22:0/2256542029 wait complete. 2024-03-24T06:46:35.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.598+0000 7f1bdf200640 1 Processor -- start 2024-03-24T06:46:35.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.598+0000 7f1bdf200640 1 -- start start 2024-03-24T06:46:35.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 0x7f1be019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0197fc0 con 0x7f1be0101f60 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0198130 con 0x7f1be0107660 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1be01982f0 con 0x7f1be01015a0 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48704/0 (socket says 172.21.15.22:48704) 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdde00640 1 -- 172.21.15.22:0/1197266094 learned_addr learned my addr 172.21.15.22:0/1197266094 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bdd400640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 0x7f1be019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.599+0000 7f1bde800640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 -- 172.21.15.22:0/1197266094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 msgr2=0x7f1be0197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 -- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 msgr2=0x7f1be019dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 0x7f1be019dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 -- 172.21.15.22:0/1197266094 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd0009560 con 0x7f1be01015a0 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdde00640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000c00 tx=0x7f1bd002c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bd6a00640 1 -- 172.21.15.22:0/1197266094 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00042d0 con 0x7f1be01015a0 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be0198570 con 0x7f1be01015a0 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bd6a00640 1 -- 172.21.15.22:0/1197266094 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd0004430 con 0x7f1be01015a0 2024-03-24T06:46:35.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.600+0000 7f1bd6a00640 1 -- 172.21.15.22:0/1197266094 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0003c50 con 0x7f1be01015a0 2024-03-24T06:46:35.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.601+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be01a4860 con 0x7f1be01015a0 2024-03-24T06:46:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.604+0000 7f1be7fcf740 1 Processor -- start 2024-03-24T06:46:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.604+0000 7f1be7fcf740 1 -- start start 2024-03-24T06:46:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6bcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6bd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556faa627790 con 0x556faa6be050 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556faa624cc0 con 0x556faa6c66e0 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556faa6204a0 con 0x556faa6bd370 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bd6a00640 1 -- 172.21.15.22:0/1197266094 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1bd0004df0 con 0x7f1be01015a0 2024-03-24T06:46:35.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bde800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6bcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bde800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6bcda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52782/0 (socket says 172.21.15.22:52782) 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34826/0 (socket says 172.21.15.22:34826) 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bde800640 1 -- 172.21.15.22:0/3500702658 learned_addr learned my addr 172.21.15.22:0/3500702658 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.605+0000 7f1bdde00640 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6bd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 -- 172.21.15.22:0/3500702658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 msgr2=0x556faa6bd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6bd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 -- 172.21.15.22:0/3500702658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 msgr2=0x556faa6bcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6bcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 -- 172.21.15.22:0/3500702658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556faa5e1c20 con 0x556faa6c66e0 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bdd400640 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1be0199160 tx=0x7f1bcc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e500b959913373c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bcb400640 1 -- 172.21.15.22:0/3500702658 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc004050 con 0x556faa6c66e0 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bcb400640 1 -- 172.21.15.22:0/3500702658 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc005500 con 0x556faa6c66e0 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.606+0000 7f1bcb400640 1 -- 172.21.15.22:0/3500702658 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc004d40 con 0x556faa6c66e0 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1be7fcf740 1 -- 172.21.15.22:0/3500702658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 msgr2=0x556faa6c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1be0199160 tx=0x7f1bcc011810 comp rx=0 tx=0).stop 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1bd6a00640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 0x7f1bc40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1bd6a00640 1 -- 172.21.15.22:0/1197266094 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1bd00f0940 con 0x7f1be01015a0 2024-03-24T06:46:35.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1be7fcf740 1 -- 172.21.15.22:0/3500702658 shutdown_connections 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.607+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c8ad0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6bd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/3500702658 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6bcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 -- 172.21.15.22:0/3500702658 >> 172.21.15.22:0/3500702658 conn(0x556faa67fb30 msgr2=0x556faa6b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 -- 172.21.15.22:0/3500702658 shutdown_connections 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1bde800640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 0x7f1bc40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 -- 172.21.15.22:0/3500702658 wait complete. 2024-03-24T06:46:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1be7fcf740 1 -- start start 2024-03-24T06:46:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.608+0000 7f1bde800640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 0x7f1bc40a4770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8005fd0 tx=0x7f1bd8009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.609+0000 7f1be7fcf740 1 Processor -- start 2024-03-24T06:46:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.609+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.609+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.609+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.609+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556faa5c4f20 con 0x556faa6bd370 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556faa5c4bc0 con 0x556faa6c66e0 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556faa38e1c0 con 0x556faa6be050 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1be7fcf740 10 client.0 _ll_register_callbacks cb 0x556faa330be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:35.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52786/0 (socket says 172.21.15.22:52786) 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 -- 172.21.15.22:0/539117167 learned_addr learned my addr 172.21.15.22:0/539117167 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48712/0 (socket says 172.21.15.22:48712) 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bde800640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 -- 172.21.15.22:0/539117167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 msgr2=0x556faa6c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 -- 172.21.15.22:0/539117167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 msgr2=0x556faa6c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdde00640 1 -- 172.21.15.22:0/539117167 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bcc009850 con 0x556faa6bd370 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.610+0000 7f1bdd400640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bdde00640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd00b9320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf5ff3576b4844c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:35.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00bc9e0 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1bd00bd780 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556faa618160 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1bd00bd080 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1be7fcf740 10 client.16440 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556faa61eb20 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.611+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556faa618160 con 0x556faa6bd370 2024-03-24T06:46:35.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1bd003c070 con 0x556faa6bd370 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 10 client.16440 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 10 client.16440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 20 client.16440 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa6d6a60 0x556faa6d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 4 client.16440 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556faa31f7e0 con 0x556faa6d6a60 2024-03-24T06:46:35.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.612+0000 7f1bdd400640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa6d6a60 0x556faa6d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.613+0000 7f1bdd400640 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa6d6a60 0x556faa6d8e70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9287b0b3cad3ef1 server_cookie=138020c2b3cd9706 in_seq=0 out_seq=0 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.613+0000 7f1bbfe00640 10 client.16440 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.613+0000 7f1bbfe00640 1 -- 172.21.15.22:0/539117167 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556faa31f7e0 con 0x556faa6d6a60 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.613+0000 7f1bbfe00640 10 client.16440 handle_command_reply: tid=0 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 client.16440 shutdown 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 msgr2=0x556faa6c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009900 tx=0x7f1bd00b9320 comp rx=0 tx=0).stop 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 shutdown_connections 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa6d6a60 0x556faa6d8e70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6c66e0 0x556faa6c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6be050 0x556faa6c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/539117167 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6bd370 0x556faa6c3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 >> 172.21.15.22:0/539117167 conn(0x556faa67fb30 msgr2=0x556faa6b9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1bbfe00640 0 client.16440 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:35.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 shutdown_connections 2024-03-24T06:46:35.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 1 -- 172.21.15.22:0/539117167 wait complete. 2024-03-24T06:46:35.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.614+0000 7f1be7fcf740 20 client.16440 trim_cache size 0 max 16384 2024-03-24T06:46:35.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.633+0000 7f1be7fcf740 1 Processor -- start 2024-03-24T06:46:35.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 -- start start 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6be050 0x556faa6de710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556faa6af270 con 0x556faa6c66e0 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556faa6b7650 con 0x556faa6be050 2024-03-24T06:46:35.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556faa6c6160 con 0x556faa6bd370 2024-03-24T06:46:35.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1bde800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6be050 0x556faa6de710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1bde800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52796/0 (socket says 172.21.15.22:52796) 2024-03-24T06:46:35.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.634+0000 7f1bde800640 1 -- 172.21.15.22:0/2969263973 learned_addr learned my addr 172.21.15.22:0/2969263973 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 -- 172.21.15.22:0/2969263973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6be050 msgr2=0x556faa6de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6be050 0x556faa6de710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 -- 172.21.15.22:0/2969263973 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 msgr2=0x556faa6ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 -- 172.21.15.22:0/2969263973 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556faa5ba9a0 con 0x556faa6bd370 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bde800640 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:35.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6c3f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0005ec0 tx=0x7f1bd00b9450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce6ad54a5615b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bcaa00640 1 -- 172.21.15.22:0/2969263973 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00bc9e0 con 0x556faa6bd370 2024-03-24T06:46:35.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bcaa00640 1 -- 172.21.15.22:0/2969263973 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd00f6020 con 0x556faa6bd370 2024-03-24T06:46:35.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.635+0000 7f1bcaa00640 1 -- 172.21.15.22:0/2969263973 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00c58b0 con 0x556faa6bd370 2024-03-24T06:46:35.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2969263973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 msgr2=0x556faa6c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6c3f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0005ec0 tx=0x7f1bd00b9450 comp rx=0 tx=0).stop 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2969263973 shutdown_connections 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6c66e0 0x556faa6ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6be050 0x556faa6de710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2969263973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6bd370 0x556faa6c3f30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2969263973 >> 172.21.15.22:0/2969263973 conn(0x556faa67fb30 msgr2=0x556faa6c1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2969263973 shutdown_connections 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2969263973 wait complete. 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.636+0000 7f1be7fcf740 1 -- start start 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 Processor -- start 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556faa73bdc0 con 0x556faa6be050 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556faa73bf30 con 0x556faa6bd370 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556faa73c0f0 con 0x556faa6c66e0 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1be7fcf740 10 client.0 _ll_register_callbacks cb 0x7f1bd00045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bde800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bdd400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bdde00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34850/0 (socket says 172.21.15.22:34850) 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bde800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48744/0 (socket says 172.21.15.22:48744) 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.637+0000 7f1bdde00640 1 -- 172.21.15.22:0/2538867287 learned_addr learned my addr 172.21.15.22:0/2538867287 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 -- 172.21.15.22:0/2538867287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 msgr2=0x556faa6c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 -- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 msgr2=0x556faa6c0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6c0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 -- 172.21.15.22:0/2538867287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd8003b30 con 0x556faa6bd370 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bde800640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdd400640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bdde00640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00c56b0 tx=0x7f1bd0009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=243df73c2f6dd3d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd009bea0 con 0x556faa6bd370 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.638+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1bd00050a0 con 0x556faa6bd370 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bcc005500 con 0x556faa6bd370 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1bd00c5890 con 0x556faa6bd370 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 10 client.26813 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556faa61eb20 con 0x556faa6bd370 2024-03-24T06:46:35.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc005500 con 0x556faa6bd370 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1bd003c070 con 0x556faa6bd370 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 10 client.26813 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 10 client.26813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 20 client.26813 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa73d3b0 0x556faa73f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 4 client.26813 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:46:35.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.639+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556faa35c720 con 0x556faa73d3b0 2024-03-24T06:46:35.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1bdd400640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa73d3b0 0x556faa73f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1bdd400640 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa73d3b0 0x556faa73f7a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ad876dfa7a2e8d server_cookie=281bb3cd45048cb2 in_seq=0 out_seq=0 2024-03-24T06:46:35.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1bbfe00640 10 client.26813 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:35.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1bbfe00640 1 -- 172.21.15.22:0/2538867287 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556faa35c720 con 0x556faa73d3b0 2024-03-24T06:46:35.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1bbfe00640 10 client.26813 handle_command_reply: tid=0 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.640+0000 7f1be7fcf740 1 client.26813 shutdown 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 msgr2=0x556faa6c03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00c56b0 tx=0x7f1bd0009880 comp rx=0 tx=0).stop 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 shutdown_connections 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556faa6c66e0 0x556faa6c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556faa6be050 0x556faa6c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556faa73d3b0 0x556faa73f7a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 --2- 172.21.15.22:0/2538867287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556faa6bd370 0x556faa6c03c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1bbfe00640 0 client.26813 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:35.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 >> 172.21.15.22:0/2538867287 conn(0x556faa67fb30 msgr2=0x556faa6b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 shutdown_connections 2024-03-24T06:46:35.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 1 -- 172.21.15.22:0/2538867287 wait complete. 2024-03-24T06:46:35.653 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.641+0000 7f1be7fcf740 20 client.26813 trim_cache size 0 max 16384 2024-03-24T06:46:35.653 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:46:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:46:35.654 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 msgr2=0x7f1bc40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 0x7f1bc40a4770 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8005fd0 tx=0x7f1bd8009500 comp rx=0 tx=0).stop 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 msgr2=0x7f1be019d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0000c00 tx=0x7f1bd002c5f0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 shutdown_connections 2024-03-24T06:46:35.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1bc40a22b0 0x7f1bc40a4770 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be0107660 0x7f1be0197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1be0101f60 0x7f1be019dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 --2- 172.21.15.22:0/1197266094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1be01015a0 0x7f1be019d7c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 >> 172.21.15.22:0/1197266094 conn(0x7f1be00fd070 msgr2=0x7f1be00fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 shutdown_connections 2024-03-24T06:46:35.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.642+0000 7f1bdf200640 1 -- 172.21.15.22:0/1197266094 wait complete. 2024-03-24T06:46:35.660 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:46:35.660 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:46:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 -- 172.21.15.22:0/745773937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 msgr2=0x7efbf81079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 --2- 172.21.15.22:0/745773937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf81079e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efbec009930 tx=0x7efbec02bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 -- 172.21.15.22:0/745773937 shutdown_connections 2024-03-24T06:46:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 --2- 172.21.15.22:0/745773937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8069230 0x7efbf810e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 --2- 172.21.15.22:0/745773937 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8068870 0x7efbf8068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 --2- 172.21.15.22:0/745773937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf81079e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 -- 172.21.15.22:0/745773937 >> 172.21.15.22:0/745773937 conn(0x7efbf8076020 msgr2=0x7efbf8076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 -- 172.21.15.22:0/745773937 shutdown_connections 2024-03-24T06:46:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.913+0000 7efbfe000640 1 -- 172.21.15.22:0/745773937 wait complete. 2024-03-24T06:46:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.914+0000 7efbfe000640 1 Processor -- start 2024-03-24T06:46:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 -- start start 2024-03-24T06:46:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 0x7efbf8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8069230 0x7efbf819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efbf819ec20 con 0x7efbf8069230 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efbf819ed90 con 0x7efbf8107600 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfe000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efbf819ef50 con 0x7efbf8068870 2024-03-24T06:46:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfcc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 0x7efbf8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbf7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8069230 0x7efbf819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34870/0 (socket says 172.21.15.22:34870) 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.915+0000 7efbfd600640 1 -- 172.21.15.22:0/1422291647 learned_addr learned my addr 172.21.15.22:0/1422291647 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 -- 172.21.15.22:0/1422291647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 msgr2=0x7efbf8199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 0x7efbf8199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 -- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8069230 msgr2=0x7efbf819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8069230 0x7efbf819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 -- 172.21.15.22:0/1422291647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbec009560 con 0x7efbf8107600 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfcc00640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 0x7efbf8199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfd600640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7efbf000eb10 tx=0x7efbf000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0004b20 con 0x7efbf8107600 2024-03-24T06:46:35.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf819f230 con 0x7efbf8107600 2024-03-24T06:46:35.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.916+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbf0004c80 con 0x7efbf8107600 2024-03-24T06:46:35.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.917+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0010670 con 0x7efbf8107600 2024-03-24T06:46:35.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.917+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81a6ab0 con 0x7efbf8107600 2024-03-24T06:46:35.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.918+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf8068420 con 0x7efbf8107600 2024-03-24T06:46:35.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.920+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7efbf0010890 con 0x7efbf8107600 2024-03-24T06:46:35.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.921+0000 7efbf5600640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 0x7efbe00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:35.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.921+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7efbf0014070 con 0x7efbf8107600 2024-03-24T06:46:35.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.921+0000 7efbfcc00640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 0x7efbe00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:35.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.922+0000 7efbfcc00640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 0x7efbe00a46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efbec009900 tx=0x7efbec005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:35.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:35.923+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efbf0097260 con 0x7efbf8107600 2024-03-24T06:46:36.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.165+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efbf8111750 con 0x7efbf8107600 2024-03-24T06:46:36.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.167+0000 7efbf5600640 1 -- 172.21.15.22:0/1422291647 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7efbf009a920 con 0x7efbf8107600 2024-03-24T06:46:36.167 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:46:36.167 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:46:36.167 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:46:36.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 msgr2=0x7efbe00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 0x7efbe00a46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efbec009900 tx=0x7efbec005a20 comp rx=0 tx=0).stop 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 msgr2=0x7efbf819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7efbf000eb10 tx=0x7efbf000c2d0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 shutdown_connections 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbe00a2230 0x7efbe00a46f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efbf8107600 0x7efbf819e620 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efbf8069230 0x7efbf819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 --2- 172.21.15.22:0/1422291647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf8068870 0x7efbf8199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 >> 172.21.15.22:0/1422291647 conn(0x7efbf8076020 msgr2=0x7efbf810c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 shutdown_connections 2024-03-24T06:46:36.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.170+0000 7efbfe000640 1 -- 172.21.15.22:0/1422291647 wait complete. 2024-03-24T06:46:36.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:46:36.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 -- 172.21.15.22:0/1690994573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f851410bb60 msgr2=0x7f851410e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 --2- 172.21.15.22:0/1690994573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f851410bb60 0x7f851410e000 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8510009ca0 tx=0x7f851002c180 comp rx=0 tx=0).stop 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 -- 172.21.15.22:0/1690994573 shutdown_connections 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 --2- 172.21.15.22:0/1690994573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f851410bb60 0x7f851410e000 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 --2- 172.21.15.22:0/1690994573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 0x7f851410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 --2- 172.21.15.22:0/1690994573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8514102740 0x7f8514102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 -- 172.21.15.22:0/1690994573 >> 172.21.15.22:0/1690994573 conn(0x7f85140fbf60 msgr2=0x7f85140fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 -- 172.21.15.22:0/1690994573 shutdown_connections 2024-03-24T06:46:36.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.410+0000 7f851b200640 1 -- 172.21.15.22:0/1690994573 wait complete. 2024-03-24T06:46:36.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 Processor -- start 2024-03-24T06:46:36.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 -- start start 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 0x7f851419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851410bb60 0x7f851419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f851419ebe0 con 0x7f8514103060 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f851419ed50 con 0x7f851410bb60 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f851b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f851419ef10 con 0x7f8514102740 2024-03-24T06:46:36.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.412+0000 7f8519e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 0x7f851419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48782/0 (socket says 172.21.15.22:48782) 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 -- 172.21.15.22:0/1317426704 learned_addr learned my addr 172.21.15.22:0/1317426704 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f851a800640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851410bb60 0x7f851419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 -- 172.21.15.22:0/1317426704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851410bb60 msgr2=0x7f851419e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851410bb60 0x7f851419e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 -- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 msgr2=0x7f851419a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 0x7f851419a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 -- 172.21.15.22:0/1317426704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8504009490 con 0x7f8514102740 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f8519e00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f850800e9f0 tx=0x7f850800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.413+0000 7f850ea00640 1 -- 172.21.15.22:0/1317426704 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508009700 con 0x7f8514102740 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.414+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8510009900 con 0x7f8514102740 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.414+0000 7f850ea00640 1 -- 172.21.15.22:0/1317426704 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f850800cd80 con 0x7f8514102740 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.414+0000 7f850ea00640 1 -- 172.21.15.22:0/1317426704 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508018920 con 0x7f8514102740 2024-03-24T06:46:36.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.414+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85141a6890 con 0x7f8514102740 2024-03-24T06:46:36.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f850ea00640 1 -- 172.21.15.22:0/1317426704 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8508018a80 con 0x7f8514102740 2024-03-24T06:46:36.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 Processor -- start 2024-03-24T06:46:36.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 -- start start 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894927410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894929d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56489488c6e0 con 0x56489491ed70 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564894889c10 con 0x564894927950 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851f5a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648948853f0 con 0x56489491e450 2024-03-24T06:46:36.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.417+0000 7f851a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48788/0 (socket says 172.21.15.22:48788) 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 -- 172.21.15.22:0/3119785457 learned_addr learned my addr 172.21.15.22:0/3119785457 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f8519400640 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894929d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f8519e00640 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894927410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 -- 172.21.15.22:0/3119785457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 msgr2=0x564894929d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894929d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 -- 172.21.15.22:0/3119785457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 msgr2=0x564894927410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894927410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 -- 172.21.15.22:0/3119785457 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564894846e40 con 0x56489491e450 2024-03-24T06:46:36.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f850ea00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 0x7f84fc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.418+0000 7f851a800640 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f851002c790 tx=0x7f8510005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b212e64d26289c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f850ea00640 1 -- 172.21.15.22:0/1317426704 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8508014070 con 0x7f8514102740 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f8503000640 1 -- 172.21.15.22:0/3119785457 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510034440 con 0x56489491e450 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f8503000640 1 -- 172.21.15.22:0/3119785457 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8510003970 con 0x56489491e450 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f8503000640 1 -- 172.21.15.22:0/3119785457 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510032e90 con 0x56489491e450 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f8519e00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 0x7f84fc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 -- 172.21.15.22:0/3119785457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 msgr2=0x56489491e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f851002c790 tx=0x7f8510005f80 comp rx=0 tx=0).stop 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f8519e00640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 0x7f84fc0a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x56489491df20 tx=0x7f8508000bc0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 -- 172.21.15.22:0/3119785457 shutdown_connections 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894929d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894927410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 --2- 172.21.15.22:0/3119785457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x56489491e830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.419+0000 7f851f5a6740 1 -- 172.21.15.22:0/3119785457 >> 172.21.15.22:0/3119785457 conn(0x564894918b80 msgr2=0x564894918f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.420+0000 7f851f5a6740 1 -- 172.21.15.22:0/3119785457 shutdown_connections 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.420+0000 7f851f5a6740 1 -- 172.21.15.22:0/3119785457 wait complete. 2024-03-24T06:46:36.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.420+0000 7f851f5a6740 1 -- start start 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.420+0000 7f851f5a6740 1 Processor -- start 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x5648949249e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894925460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56489482a060 con 0x56489491ed70 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564894829d00 con 0x564894927950 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648945f31c0 con 0x56489491e450 2024-03-24T06:46:36.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851f5a6740 10 client.0 _ll_register_callbacks cb 0x564894595be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f8519400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x5648949249e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f8519400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52860/0 (socket says 172.21.15.22:52860) 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f8519400640 1 -- 172.21.15.22:0/796762396 learned_addr learned my addr 172.21.15.22:0/796762396 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.421+0000 7f851a800640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894925460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 -- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 msgr2=0x5648949249e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x5648949249e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 -- 172.21.15.22:0/796762396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 msgr2=0x564894925460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894925460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 -- 172.21.15.22:0/796762396 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851410ec80 con 0x56489491ed70 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f851a800640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894925460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f8519400640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8504012820 tx=0x7f8504012ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f5d82d12b2edea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f84f7400640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504017070 con 0x56489491ed70 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f850400f920 con 0x56489491ed70 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.422+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648945847e0 con 0x56489491ed70 2024-03-24T06:46:36.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f850400fb20 con 0x56489491ed70 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f851f5a6740 10 client.16464 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564894883a70 con 0x56489491ed70 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5648945847e0 con 0x56489491ed70 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f850401c420 con 0x56489491ed70 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f851f5a6740 10 client.16464 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f851f5a6740 10 client.16464 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.423+0000 7f851f5a6740 20 client.16464 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.424+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x564894937e30 0x56489493a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.424+0000 7f851f5a6740 4 client.16464 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.424+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648945c1720 con 0x564894937e30 2024-03-24T06:46:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.424+0000 7f851a800640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x564894937e30 0x56489493a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.425+0000 7f851a800640 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x564894937e30 0x56489493a250 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d196c946be98eb00 server_cookie=dc5a37cfdca704a4 in_seq=0 out_seq=0 2024-03-24T06:46:36.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.425+0000 7f84f7400640 10 client.16464 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:36.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:36 smithi143 ceph-mon[36520]: pgmap v202: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-03-24T06:46:36.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1422291647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.425+0000 7f84f7400640 1 -- 172.21.15.22:0/796762396 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648945c1720 con 0x564894937e30 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.425+0000 7f84f7400640 10 client.16464 handle_command_reply: tid=0 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.425+0000 7f851f5a6740 1 client.16464 shutdown 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 msgr2=0x564894924f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8504012820 tx=0x7f8504012ce0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 shutdown_connections 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x564894937e30 0x56489493a250 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x564894925460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491ed70 0x564894924f20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 --2- 172.21.15.22:0/796762396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491e450 0x5648949249e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f84f7400640 0 client.16464 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 >> 172.21.15.22:0/796762396 conn(0x564894918b80 msgr2=0x56489491a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 shutdown_connections 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 1 -- 172.21.15.22:0/796762396 wait complete. 2024-03-24T06:46:36.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.426+0000 7f851f5a6740 20 client.16464 trim_cache size 0 max 16384 2024-03-24T06:46:36.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 Processor -- start 2024-03-24T06:46:36.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 -- start start 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 0x564894925280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x56489493fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564894934890 con 0x56489491e450 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648949186b0 con 0x564894927950 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.441+0000 7f851f5a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56489491f7f0 con 0x56489491ed70 2024-03-24T06:46:36.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f851a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 0x564894925280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f851a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52870/0 (socket says 172.21.15.22:52870) 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f851a800640 1 -- 172.21.15.22:0/926115228 learned_addr learned my addr 172.21.15.22:0/926115228 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f851a800640 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x56489493fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 -- 172.21.15.22:0/926115228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 msgr2=0x56489493fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x56489493fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 -- 172.21.15.22:0/926115228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 msgr2=0x564894927110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 -- 172.21.15.22:0/926115228 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489481fb00 con 0x56489491ed70 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f851a800640 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.442+0000 7f8519400640 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 0x564894925280 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f850401ca40 tx=0x7f8504005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b324b73a126269b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f8502600640 1 -- 172.21.15.22:0/926115228 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504017070 con 0x56489491ed70 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f8502600640 1 -- 172.21.15.22:0/926115228 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f850401c420 con 0x56489491ed70 2024-03-24T06:46:36.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f8502600640 1 -- 172.21.15.22:0/926115228 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504005310 con 0x56489491ed70 2024-03-24T06:46:36.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 -- 172.21.15.22:0/926115228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 msgr2=0x564894925280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 0x564894925280 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f850401ca40 tx=0x7f8504005e30 comp rx=0 tx=0).stop 2024-03-24T06:46:36.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 -- 172.21.15.22:0/926115228 shutdown_connections 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564894927950 0x56489493fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56489491ed70 0x564894925280 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 --2- 172.21.15.22:0/926115228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x564894927110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.443+0000 7f851f5a6740 1 -- 172.21.15.22:0/926115228 >> 172.21.15.22:0/926115228 conn(0x564894918b80 msgr2=0x564894919db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.444+0000 7f851f5a6740 1 -- 172.21.15.22:0/926115228 shutdown_connections 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.444+0000 7f851f5a6740 1 -- 172.21.15.22:0/926115228 wait complete. 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.444+0000 7f851f5a6740 1 -- start start 2024-03-24T06:46:36.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.444+0000 7f851f5a6740 1 Processor -- start 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x56489492b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564894927950 0x56489492bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5648949a5920 con 0x56489491e450 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5648949a5a90 con 0x56489491ed70 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648949a5c50 con 0x564894927950 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851f5a6740 10 client.0 _ll_register_callbacks cb 0x7f85040099d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34942/0 (socket says 172.21.15.22:34942) 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 -- 172.21.15.22:0/1045956259 learned_addr learned my addr 172.21.15.22:0/1045956259 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x56489492b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851a800640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564894927950 0x56489492bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 -- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564894927950 msgr2=0x56489492bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564894927950 0x56489492bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 -- 172.21.15.22:0/1045956259 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 msgr2=0x56489492b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x56489492b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f8519400640 1 -- 172.21.15.22:0/1045956259 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564894846e40 con 0x56489491ed70 2024-03-24T06:46:36.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.445+0000 7f851a800640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x56489492b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f8519400640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f85040127f0 tx=0x7f8504009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b367946b5ce6aeca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f84f7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504017070 con 0x56489491ed70 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f850400fe60 con 0x56489491ed70 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8510003970 con 0x56489491ed70 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8504005240 con 0x56489491ed70 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f851f5a6740 10 client.26837 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.446+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564894883a70 con 0x56489491ed70 2024-03-24T06:46:36.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8510003970 con 0x56489491ed70 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8504030030 con 0x56489491ed70 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 10 client.26837 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 10 client.26837 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 20 client.26837 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5648949a71d0 0x5648949a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 4 client.26837 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564894610890 con 0x5648949a71d0 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.447+0000 7f851a800640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5648949a71d0 0x5648949a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851a800640 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5648949a71d0 0x5648949a95f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f2ef23c218706767 server_cookie=1ce4214516f88de1 in_seq=0 out_seq=0 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f84f7400640 10 client.26837 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f84f7400640 1 -- 172.21.15.22:0/1045956259 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564894610890 con 0x5648949a71d0 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f84f7400640 10 client.26837 handle_command_reply: tid=0 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 client.26837 shutdown 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 msgr2=0x56489492b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f85040127f0 tx=0x7f8504009ec0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 shutdown_connections 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5648949a71d0 0x5648949a95f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564894927950 0x56489492bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56489491ed70 0x56489492b740 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 --2- 172.21.15.22:0/1045956259 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56489491e450 0x56489492b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 >> 172.21.15.22:0/1045956259 conn(0x564894918b80 msgr2=0x56489491a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f84f7400640 0 client.26837 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 shutdown_connections 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 1 -- 172.21.15.22:0/1045956259 wait complete. 2024-03-24T06:46:36.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.448+0000 7f851f5a6740 20 client.26837 trim_cache size 0 max 16384 2024-03-24T06:46:36.456 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:36.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 msgr2=0x7f84fc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 0x7f84fc0a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x56489491df20 tx=0x7f8508000bc0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 msgr2=0x7f8514199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f850800e9f0 tx=0x7f850800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 shutdown_connections 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f84fc0a2230 0x7f84fc0a46f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851410bb60 0x7f851419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8514103060 0x7f851419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 --2- 172.21.15.22:0/1317426704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8514102740 0x7f8514199dc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 >> 172.21.15.22:0/1317426704 conn(0x7f85140fbf60 msgr2=0x7f851410ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 shutdown_connections 2024-03-24T06:46:36.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.449+0000 7f851b200640 1 -- 172.21.15.22:0/1317426704 wait complete. 2024-03-24T06:46:36.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:36 smithi022 ceph-mon[32003]: pgmap v202: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-03-24T06:46:36.464 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1422291647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:36.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:46:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:36 smithi155 ceph-mon[36262]: pgmap v202: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-03-24T06:46:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1422291647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 -- 172.21.15.22:0/2675337103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164100e50 msgr2=0x7f3164105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 --2- 172.21.15.22:0/2675337103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164100e50 0x7f3164105310 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f314c009900 tx=0x7f314c02bd10 comp rx=0 tx=0).stop 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 -- 172.21.15.22:0/2675337103 shutdown_connections 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 --2- 172.21.15.22:0/2675337103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164105930 0x7f3164105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 --2- 172.21.15.22:0/2675337103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164100e50 0x7f3164105310 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 --2- 172.21.15.22:0/2675337103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f3164100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 -- 172.21.15.22:0/2675337103 >> 172.21.15.22:0/2675337103 conn(0x7f31640fbf60 msgr2=0x7f31640fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 -- 172.21.15.22:0/2675337103 shutdown_connections 2024-03-24T06:46:36.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.676+0000 7f3169e00640 1 -- 172.21.15.22:0/2675337103 wait complete. 2024-03-24T06:46:36.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 Processor -- start 2024-03-24T06:46:36.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 -- start start 2024-03-24T06:46:36.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164100e50 0x7f316419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164105930 0x7f31641a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f31641a30a0 con 0x7f3164100e50 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f31641a3210 con 0x7f3164105930 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.678+0000 7f3169e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f31641a33d0 con 0x7f3164100530 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48836/0 (socket says 172.21.15.22:48836) 2024-03-24T06:46:36.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 -- 172.21.15.22:0/3061937664 learned_addr learned my addr 172.21.15.22:0/3061937664 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f315be00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164100e50 0x7f316419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3169400640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164105930 0x7f31641a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 -- 172.21.15.22:0/3061937664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164105930 msgr2=0x7f31641a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164105930 0x7f31641a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 -- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164100e50 msgr2=0x7f316419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164100e50 0x7f316419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 -- 172.21.15.22:0/3061937664 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c009560 con 0x7f3164100530 2024-03-24T06:46:36.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.679+0000 7f3168a00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9f0 tx=0x7f315400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.680+0000 7f3162000640 1 -- 172.21.15.22:0/3061937664 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154009700 con 0x7f3164100530 2024-03-24T06:46:36.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.680+0000 7f3162000640 1 -- 172.21.15.22:0/3061937664 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315400cd80 con 0x7f3164100530 2024-03-24T06:46:36.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.680+0000 7f3162000640 1 -- 172.21.15.22:0/3061937664 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154018920 con 0x7f3164100530 2024-03-24T06:46:36.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.680+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31641a36b0 con 0x7f3164100530 2024-03-24T06:46:36.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.680+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31641aaf30 con 0x7f3164100530 2024-03-24T06:46:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.683+0000 7f316e219740 1 Processor -- start 2024-03-24T06:46:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.683+0000 7f316e219740 1 -- start start 2024-03-24T06:46:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.683+0000 7f3162000640 1 -- 172.21.15.22:0/3061937664 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3154018a80 con 0x7f3164100530 2024-03-24T06:46:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c836410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c838d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55727c79b6e0 con 0x55727c82d450 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55727c798c10 con 0x55727c836950 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f316e219740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55727c7943f0 con 0x55727c82dd70 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f3169400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f3168a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c836410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f3169400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52906/0 (socket says 172.21.15.22:52906) 2024-03-24T06:46:36.686 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f3169400640 1 -- 172.21.15.22:0/4184715622 learned_addr learned my addr 172.21.15.22:0/4184715622 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.684+0000 7f315be00640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c838d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 -- 172.21.15.22:0/4184715622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 msgr2=0x55727c836410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c836410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 -- 172.21.15.22:0/4184715622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 msgr2=0x55727c838d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c838d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 -- 172.21.15.22:0/4184715622 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727c755e40 con 0x55727c82d450 2024-03-24T06:46:36.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3168a00640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c836410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f315be00640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c838d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3169400640 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f315c009f50 tx=0x7f315c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a7e6e90db34866 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3159c00640 1 -- 172.21.15.22:0/4184715622 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c004d20 con 0x55727c82d450 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3162000640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 0x7f31480a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3159c00640 1 -- 172.21.15.22:0/4184715622 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c004e80 con 0x55727c82d450 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.685+0000 7f3159c00640 1 -- 172.21.15.22:0/4184715622 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c005280 con 0x55727c82d450 2024-03-24T06:46:36.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f3162000640 1 -- 172.21.15.22:0/3061937664 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f3154014070 con 0x7f3164100530 2024-03-24T06:46:36.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f3168a00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 0x7f31480a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 -- 172.21.15.22:0/4184715622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 msgr2=0x55727c82d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f315c009f50 tx=0x7f315c0118d0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 -- 172.21.15.22:0/4184715622 shutdown_connections 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c838d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c836410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 --2- 172.21.15.22:0/4184715622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c82d830 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 -- 172.21.15.22:0/4184715622 >> 172.21.15.22:0/4184715622 conn(0x55727c827b80 msgr2=0x55727c827f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 -- 172.21.15.22:0/4184715622 shutdown_connections 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.686+0000 7f316e219740 1 -- 172.21.15.22:0/4184715622 wait complete. 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.687+0000 7f3168a00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 0x7f31480a4770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31540cbb80 tx=0x7f31540d40e0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.687+0000 7f316e219740 1 -- start start 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.687+0000 7f316e219740 1 Processor -- start 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c833990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c836950 0x55727c834410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55727c739060 con 0x55727c836950 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55727c738d00 con 0x55727c82d450 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55727c5021c0 con 0x55727c82dd70 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f316e219740 10 client.0 _ll_register_callbacks cb 0x55727c4a4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f315be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f3169400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c833990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f315be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48862/0 (socket says 172.21.15.22:48862) 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.688+0000 7f315be00640 1 -- 172.21.15.22:0/2485537176 learned_addr learned my addr 172.21.15.22:0/2485537176 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f3169400640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c836950 0x55727c834410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 -- 172.21.15.22:0/2485537176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 msgr2=0x55727c833990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c833990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 -- 172.21.15.22:0/2485537176 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c836950 msgr2=0x55727c834410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c836950 0x55727c834410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 -- 172.21.15.22:0/2485537176 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c009c00 con 0x55727c82dd70 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f315be00640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f314c009230 tx=0x7f314c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d79f02b4b29126 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.689+0000 7f3152000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:36.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c02c860 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f314c02c9c0 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55727c4d0720 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f314c02cb80 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f316e219740 10 client.26869 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55727c792a70 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55727c4d0720 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.690+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f314c046020 con 0x55727c82dd70 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 10 client.26869 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 10 client.26869 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 20 client.26869 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c846e90 0x55727c8492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 4 client.26869 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55727c51f890 con 0x55727c846e90 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f3169400640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c846e90 0x55727c8492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f3169400640 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c846e90 0x55727c8492b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=514f7fd22f64c2b4 server_cookie=52f0f3c4c665a19f in_seq=0 out_seq=0 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.691+0000 7f3152000640 10 client.26869 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:36.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.692+0000 7f3152000640 1 -- 172.21.15.22:0/2485537176 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55727c51f890 con 0x55727c846e90 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.692+0000 7f3152000640 10 client.26869 handle_command_reply: tid=0 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.692+0000 7f316e219740 1 client.26869 shutdown 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.692+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 msgr2=0x55727c833ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.692+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f314c009230 tx=0x7f314c004270 comp rx=0 tx=0).stop 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 shutdown_connections 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c846e90 0x55727c8492b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c836950 0x55727c834410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c833ed0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 --2- 172.21.15.22:0/2485537176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c833990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f3152000640 0 client.26869 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 >> 172.21.15.22:0/2485537176 conn(0x55727c827b80 msgr2=0x55727c8298d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 shutdown_connections 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 1 -- 172.21.15.22:0/2485537176 wait complete. 2024-03-24T06:46:36.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.693+0000 7f316e219740 20 client.26869 trim_cache size 0 max 16384 2024-03-24T06:46:36.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.707+0000 7f316e219740 1 Processor -- start 2024-03-24T06:46:36.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 -- start start 2024-03-24T06:46:36.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c8360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c836950 0x55727c822b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55727c4d0720 con 0x55727c82dd70 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55727c843880 con 0x55727c82d450 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f316e219740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55727c8234b0 con 0x55727c836950 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f315be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f315be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52934/0 (socket says 172.21.15.22:52934) 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f315be00640 1 -- 172.21.15.22:0/403732009 learned_addr learned my addr 172.21.15.22:0/403732009 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f3169400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c8360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.708+0000 7f3169400640 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c836950 0x55727c822b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 -- 172.21.15.22:0/403732009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c836950 msgr2=0x55727c822b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c836950 0x55727c822b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 -- 172.21.15.22:0/403732009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 msgr2=0x55727c8360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c8360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 -- 172.21.15.22:0/403732009 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727c72eb00 con 0x55727c82dd70 2024-03-24T06:46:36.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f315be00640 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f314c006010 tx=0x7f314c02bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351006cd8a1dd9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f3159200640 1 -- 172.21.15.22:0/403732009 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0042e0 con 0x55727c82dd70 2024-03-24T06:46:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f3159200640 1 -- 172.21.15.22:0/403732009 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f314c0035f0 con 0x55727c82dd70 2024-03-24T06:46:36.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.709+0000 7f3159200640 1 -- 172.21.15.22:0/403732009 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0348d0 con 0x55727c82dd70 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- 172.21.15.22:0/403732009 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 msgr2=0x55727c8421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f314c006010 tx=0x7f314c02bd10 comp rx=0 tx=0).stop 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- 172.21.15.22:0/403732009 shutdown_connections 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c836950 0x55727c822b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82dd70 0x55727c8421e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 --2- 172.21.15.22:0/403732009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c82d450 0x55727c8360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- 172.21.15.22:0/403732009 >> 172.21.15.22:0/403732009 conn(0x55727c827b80 msgr2=0x55727c829810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- 172.21.15.22:0/403732009 shutdown_connections 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- 172.21.15.22:0/403732009 wait complete. 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.710+0000 7f316e219740 1 -- start start 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 Processor -- start 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c83a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c8ac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55727c8ac6b0 con 0x55727c82d450 2024-03-24T06:46:36.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55727c8ac7f0 con 0x55727c836950 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55727c8ac980 con 0x55727c82dd70 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f316e219740 10 client.0 _ll_register_callbacks cb 0x7f3154005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f315be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f315be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48882/0 (socket says 172.21.15.22:48882) 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f315be00640 1 -- 172.21.15.22:0/445089879 learned_addr learned my addr 172.21.15.22:0/445089879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.711+0000 7f3169400640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c83a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f3169400640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c8ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 -- 172.21.15.22:0/445089879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 msgr2=0x55727c8ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c8ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 -- 172.21.15.22:0/445089879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 msgr2=0x55727c83a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c83a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 -- 172.21.15.22:0/445089879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727c792a70 con 0x55727c82dd70 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f315be00640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f314c02c800 tx=0x7f314c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe23ac1db78a136 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f3152000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c004b20 con 0x55727c82dd70 2024-03-24T06:46:36.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f314c000e40 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.712+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55727c4937e0 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f314c003ed0 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 10 client.26875 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55727c7546a0 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55727c4937e0 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f314c046040 con 0x55727c82dd70 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 10 client.26875 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 10 client.26875 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 20 client.26875 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c8ade30 0x55727c8b0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 4 client.26875 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55727c43f700 con 0x55727c8ade30 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.713+0000 7f3169400640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c8ade30 0x55727c8b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.714+0000 7f3152000640 10 client.26875 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.714+0000 7f3169400640 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c8ade30 0x55727c8b0250 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea808f61128bd7b7 server_cookie=d3f512fb2d915775 in_seq=0 out_seq=0 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.714+0000 7f3152000640 1 -- 172.21.15.22:0/445089879 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55727c43f700 con 0x55727c8ade30 2024-03-24T06:46:36.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.714+0000 7f3152000640 10 client.26875 handle_command_reply: tid=0 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.714+0000 7f316e219740 1 client.26875 shutdown 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 msgr2=0x55727c83ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f314c02c800 tx=0x7f314c009800 comp rx=0 tx=0).stop 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 shutdown_connections 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55727c8ade30 0x55727c8b0250 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55727c836950 0x55727c8ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55727c82dd70 0x55727c83ae40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 --2- 172.21.15.22:0/445089879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55727c82d450 0x55727c83a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f3152000640 0 client.26875 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 >> 172.21.15.22:0/445089879 conn(0x55727c827b80 msgr2=0x55727c833930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 shutdown_connections 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 1 -- 172.21.15.22:0/445089879 wait complete. 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f316e219740 20 client.26875 trim_cache size 0 max 16384 2024-03-24T06:46:36.723 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:36.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 msgr2=0x7f31480a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.715+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 0x7f31480a4770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31540cbb80 tx=0x7f31540d40e0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 msgr2=0x7f316419e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f315400e9f0 tx=0x7f315400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 shutdown_connections 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31480a22b0 0x7f31480a4770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3164105930 0x7f31641a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3164100e50 0x7f316419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 --2- 172.21.15.22:0/3061937664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3164100530 0x7f316419e1e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 >> 172.21.15.22:0/3061937664 conn(0x7f31640fbf60 msgr2=0x7f31641032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 shutdown_connections 2024-03-24T06:46:36.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.716+0000 7f3169e00640 1 -- 172.21.15.22:0/3061937664 wait complete. 2024-03-24T06:46:36.734 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:46:36.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 -- 172.21.15.22:0/420171999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36081097a0 msgr2=0x7f3608109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 --2- 172.21.15.22:0/420171999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36081097a0 0x7f3608109b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009930 tx=0x7f35f802bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:36.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 -- 172.21.15.22:0/420171999 shutdown_connections 2024-03-24T06:46:36.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 --2- 172.21.15.22:0/420171999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104150 0x7f3608110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 --2- 172.21.15.22:0/420171999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3608103790 0x7f3608103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 --2- 172.21.15.22:0/420171999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f36081097a0 0x7f3608109b80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 -- 172.21.15.22:0/420171999 >> 172.21.15.22:0/420171999 conn(0x7f36080ff240 msgr2=0x7f3608101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 -- 172.21.15.22:0/420171999 shutdown_connections 2024-03-24T06:46:36.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.951+0000 7f360e400640 1 -- 172.21.15.22:0/420171999 wait complete. 2024-03-24T06:46:36.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.952+0000 7f360e400640 1 Processor -- start 2024-03-24T06:46:36.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 -- start start 2024-03-24T06:46:36.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3608104150 0x7f360819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f360819ed00 con 0x7f3608104150 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f360819ee70 con 0x7f36081097a0 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f360819f030 con 0x7f3608103790 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3608104150 0x7f360819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48890/0 (socket says 172.21.15.22:48890) 2024-03-24T06:46:36.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360d000640 1 -- 172.21.15.22:0/221731274 learned_addr learned my addr 172.21.15.22:0/221731274 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.953+0000 7f360da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35004/0 (socket says 172.21.15.22:35004) 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 -- 172.21.15.22:0/221731274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 msgr2=0x7f360819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 -- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3608104150 msgr2=0x7f360819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3608104150 0x7f360819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 -- 172.21.15.22:0/221731274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f8009560 con 0x7f3608103790 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360da00640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360d000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f35f8000c00 tx=0x7f35f802c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f3606000640 1 -- 172.21.15.22:0/221731274 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f80042d0 con 0x7f3608103790 2024-03-24T06:46:36.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f3606000640 1 -- 172.21.15.22:0/221731274 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35f8004430 con 0x7f3608103790 2024-03-24T06:46:36.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360819f2b0 con 0x7f3608103790 2024-03-24T06:46:36.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.954+0000 7f3606000640 1 -- 172.21.15.22:0/221731274 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8003c20 con 0x7f3608103790 2024-03-24T06:46:36.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.955+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36081a6b90 con 0x7f3608103790 2024-03-24T06:46:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3606000640 1 -- 172.21.15.22:0/221731274 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f35f8004df0 con 0x7f3608103790 2024-03-24T06:46:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 Processor -- start 2024-03-24T06:46:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 -- start start 2024-03-24T06:46:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e35410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17e37d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ac17d9a6e0 con 0x55ac17e35950 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac17d97c10 con 0x55ac17e2cd70 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.958+0000 7f3612794740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ac17d933f0 con 0x55ac17e2c450 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48896/0 (socket says 172.21.15.22:48896) 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 -- 172.21.15.22:0/3560281054 learned_addr learned my addr 172.21.15.22:0/3560281054 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360d000640 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e35410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f35ffe00640 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17e37d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 -- 172.21.15.22:0/3560281054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 msgr2=0x55ac17e35410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e35410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 -- 172.21.15.22:0/3560281054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 msgr2=0x55ac17e37d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17e37d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 -- 172.21.15.22:0/3560281054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac17d54e40 con 0x55ac17e2c450 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f360da00640 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3600011790 tx=0x7f3600011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b338036c77f447e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.959+0000 7f3606000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 0x7f35ec0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f35fe000640 1 -- 172.21.15.22:0/3560281054 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600011ea0 con 0x55ac17e2c450 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f3606000640 1 -- 172.21.15.22:0/221731274 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f35f80f08f0 con 0x7f3608103790 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f35fe000640 1 -- 172.21.15.22:0/3560281054 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f360000fe80 con 0x55ac17e2c450 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f35fe000640 1 -- 172.21.15.22:0/3560281054 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600004840 con 0x55ac17e2c450 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f360d000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 0x7f35ec0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f3612794740 1 -- 172.21.15.22:0/3560281054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 msgr2=0x55ac17e2c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f3612794740 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3600011790 tx=0x7f3600011c50 comp rx=0 tx=0).stop 2024-03-24T06:46:36.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.960+0000 7f360d000640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 0x7f35ec0a4770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55ac17e2bf20 tx=0x7f35f8009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 -- 172.21.15.22:0/3560281054 shutdown_connections 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17e37d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e35410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 --2- 172.21.15.22:0/3560281054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2c450 0x55ac17e2c830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 -- 172.21.15.22:0/3560281054 >> 172.21.15.22:0/3560281054 conn(0x55ac17e26b80 msgr2=0x55ac17e26f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 -- 172.21.15.22:0/3560281054 shutdown_connections 2024-03-24T06:46:36.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 -- 172.21.15.22:0/3560281054 wait complete. 2024-03-24T06:46:36.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 -- start start 2024-03-24T06:46:36.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.961+0000 7f3612794740 1 Processor -- start 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e32940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e35950 0x55ac17e333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ac17d38060 con 0x55ac17e2c450 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac17d37d00 con 0x55ac17e35950 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ac17b011c0 con 0x55ac17e2cd70 2024-03-24T06:46:36.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f3612794740 10 client.0 _ll_register_callbacks cb 0x55ac17aa3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f360da00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e32940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48900/0 (socket says 172.21.15.22:48900) 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f360da00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e32940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52990/0 (socket says 172.21.15.22:52990) 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.962+0000 7f35ffe00640 1 -- 172.21.15.22:0/4059004378 learned_addr learned my addr 172.21.15.22:0/4059004378 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f360da00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e35950 0x55ac17e333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 -- 172.21.15.22:0/4059004378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e35950 msgr2=0x55ac17e333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e35950 0x55ac17e333c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 -- 172.21.15.22:0/4059004378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 msgr2=0x55ac17e32940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e32940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 -- 172.21.15.22:0/4059004378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3600011440 con 0x55ac17e2cd70 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35ffe00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f35f0012700 tx=0x7f35f0012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e7efb9ecb93f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0017070 con 0x55ac17e2cd70 2024-03-24T06:46:36.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.963+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f35f0004050 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac17a927e0 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35f0004210 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 10 client.26887 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ac17d91a70 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ac17a927e0 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35f0003950 con 0x55ac17e2cd70 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 10 client.26887 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 10 client.26887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:46:36.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 20 client.26887 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17e45de0 0x55ac17e48220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.964+0000 7f3612794740 4 client.26887 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.965+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac17acf720 con 0x55ac17e45de0 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.965+0000 7f360da00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17e45de0 0x55ac17e48220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.965+0000 7f360da00640 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17e45de0 0x55ac17e48220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21ff30a269a79aff server_cookie=3c60268fa78796db in_seq=0 out_seq=0 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.965+0000 7f35f6a00640 10 client.26887 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f35f6a00640 1 -- 172.21.15.22:0/4059004378 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac17acf720 con 0x55ac17e45de0 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f35f6a00640 10 client.26887 handle_command_reply: tid=0 2024-03-24T06:46:36.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f3612794740 1 client.26887 shutdown 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 msgr2=0x55ac17e32e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f35f0012700 tx=0x7f35f0012bc0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 shutdown_connections 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.966+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17e45de0 0x55ac17e48220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e35950 0x55ac17e333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e32e80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 --2- 172.21.15.22:0/4059004378 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e32940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 >> 172.21.15.22:0/4059004378 conn(0x55ac17e26b80 msgr2=0x55ac17e288d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f35f6a00640 0 client.26887 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 shutdown_connections 2024-03-24T06:46:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 1 -- 172.21.15.22:0/4059004378 wait complete. 2024-03-24T06:46:36.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.967+0000 7f3612794740 20 client.26887 trim_cache size 0 max 16384 2024-03-24T06:46:36.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 Processor -- start 2024-03-24T06:46:36.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 -- start start 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e35070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 0x55ac17e4db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ac17e224b0 con 0x55ac17e2c450 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac17e2c0b0 con 0x55ac17e2cd70 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f3612794740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ac17e33c90 con 0x55ac17e35950 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35040/0 (socket says 172.21.15.22:35040) 2024-03-24T06:46:36.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.982+0000 7f35ffe00640 1 -- 172.21.15.22:0/1733704679 learned_addr learned my addr 172.21.15.22:0/1733704679 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f360da00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e35070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 -- 172.21.15.22:0/1733704679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 msgr2=0x55ac17e4db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f360da00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 0x55ac17e4db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 0x55ac17e4db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 -- 172.21.15.22:0/1733704679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 msgr2=0x55ac17e35070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e35070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 -- 172.21.15.22:0/1733704679 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac17d2db00 con 0x55ac17e2cd70 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f360da00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 0x55ac17e4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35ffe00640 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009560 tx=0x7f35f000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67613eb6a56b10c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35fd600640 1 -- 172.21.15.22:0/1733704679 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0017070 con 0x55ac17e2cd70 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35fd600640 1 -- 172.21.15.22:0/1733704679 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35f0003ad0 con 0x55ac17e2cd70 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.983+0000 7f35fd600640 1 -- 172.21.15.22:0/1733704679 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0005650 con 0x55ac17e2cd70 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 -- 172.21.15.22:0/1733704679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 msgr2=0x55ac17e41210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009560 tx=0x7f35f000f790 comp rx=0 tx=0).stop 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 -- 172.21.15.22:0/1733704679 shutdown_connections 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e35950 0x55ac17e4db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2cd70 0x55ac17e41210 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 --2- 172.21.15.22:0/1733704679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e2c450 0x55ac17e35070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 -- 172.21.15.22:0/1733704679 >> 172.21.15.22:0/1733704679 conn(0x55ac17e26b80 msgr2=0x55ac17e287f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 -- 172.21.15.22:0/1733704679 shutdown_connections 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.984+0000 7f3612794740 1 -- 172.21.15.22:0/1733704679 wait complete. 2024-03-24T06:46:36.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 -- start start 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 Processor -- start 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 0x55ac17e2f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17eab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ac17e30790 con 0x55ac17e35950 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac17eab6a0 con 0x55ac17e2c450 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ac17eab7e0 con 0x55ac17e2cd70 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.985+0000 7f3612794740 10 client.0 _ll_register_callbacks cb 0x7f35f00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f360da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 0x55ac17e2f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48912/0 (socket says 172.21.15.22:48912) 2024-03-24T06:46:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f360da00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 0x55ac17e2f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35050/0 (socket says 172.21.15.22:35050) 2024-03-24T06:46:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 -- 172.21.15.22:0/2218779524 learned_addr learned my addr 172.21.15.22:0/2218779524 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f360da00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17eab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 -- 172.21.15.22:0/2218779524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 msgr2=0x55ac17e2f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 0x55ac17e2f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 -- 172.21.15.22:0/2218779524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 msgr2=0x55ac17eab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17eab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 -- 172.21.15.22:0/2218779524 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f0021040 con 0x55ac17e2cd70 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f360da00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17eab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35ffe00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35f0003d60 tx=0x7f35f00096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2294d28226be0659 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.986+0000 7f35f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:36.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0017070 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f35f00037f0 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360000fe80 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35f000f920 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f3612794740 10 client.26890 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ac17d536a0 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.987+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f360000fe80 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35f0030070 con 0x55ac17e2cd70 2024-03-24T06:46:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 10 client.26890 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 10 client.26890 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 20 client.26890 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17eacb20 0x55ac17eaef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 4 client.26890 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac17b1e890 con 0x55ac17eacb20 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f360da00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17eacb20 0x55ac17eaef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f360da00640 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17eacb20 0x55ac17eaef40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e0c72088df5ff44 server_cookie=aa50c4778fe4acef in_seq=0 out_seq=0 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.988+0000 7f35f6a00640 10 client.26890 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:36.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f35f6a00640 1 -- 172.21.15.22:0/2218779524 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac17b1e890 con 0x55ac17eacb20 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f35f6a00640 10 client.26890 handle_command_reply: tid=0 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 client.26890 shutdown 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 msgr2=0x55ac17e2fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35f0003d60 tx=0x7f35f00096c0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 shutdown_connections 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55ac17eacb20 0x55ac17eaef40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ac17e35950 0x55ac17eab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ac17e2cd70 0x55ac17e2fe90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f35f6a00640 0 client.26890 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 --2- 172.21.15.22:0/2218779524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac17e2c450 0x55ac17e2f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 >> 172.21.15.22:0/2218779524 conn(0x55ac17e26b80 msgr2=0x55ac17e32880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 shutdown_connections 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 1 -- 172.21.15.22:0/2218779524 wait complete. 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.989+0000 7f3612794740 20 client.26890 trim_cache size 0 max 16384 2024-03-24T06:46:36.996 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:36.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 msgr2=0x7f35ec0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 0x7f35ec0a4770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55ac17e2bf20 tx=0x7f35f8009260 comp rx=0 tx=0).stop 2024-03-24T06:46:36.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 msgr2=0x7f3608199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:36.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f35f8000c00 tx=0x7f35f802c5f0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 shutdown_connections 2024-03-24T06:46:36.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35ec0a22b0 0x7f35ec0a4770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36081097a0 0x7f360819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3608104150 0x7f360819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 --2- 172.21.15.22:0/221731274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608103790 0x7f3608199e40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:36.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 >> 172.21.15.22:0/221731274 conn(0x7f36080ff240 msgr2=0x7f36080ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:36.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 shutdown_connections 2024-03-24T06:46:36.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:36.990+0000 7f360e400640 1 -- 172.21.15.22:0/221731274 wait complete. 2024-03-24T06:46:37.009 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:46:37.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 -- 172.21.15.22:0/2009752121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b8102740 msgr2=0x7fe2b8102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2009752121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b8102740 0x7fe2b8102b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009900 tx=0x7fe2a802bcd0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 -- 172.21.15.22:0/2009752121 shutdown_connections 2024-03-24T06:46:37.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2009752121 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b810bb60 0x7fe2b810e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2009752121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b810b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2009752121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b8102740 0x7fe2b8102b20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 -- 172.21.15.22:0/2009752121 >> 172.21.15.22:0/2009752121 conn(0x7fe2b80fbf60 msgr2=0x7fe2b80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 -- 172.21.15.22:0/2009752121 shutdown_connections 2024-03-24T06:46:37.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.217+0000 7fe2bce00640 1 -- 172.21.15.22:0/2009752121 wait complete. 2024-03-24T06:46:37.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.218+0000 7fe2bce00640 1 Processor -- start 2024-03-24T06:46:37.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 -- start start 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 0x7fe2b8100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b810bb60 0x7fe2b8100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81a5fe0 con 0x7fe2b8102740 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81a6150 con 0x7fe2b810bb60 2024-03-24T06:46:37.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2bce00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b81a6310 con 0x7fe2b8103060 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 0x7fe2b8100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48932/0 (socket says 172.21.15.22:48932) 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b810bb60 0x7fe2b8100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b6a00640 1 -- 172.21.15.22:0/2999839422 learned_addr learned my addr 172.21.15.22:0/2999839422 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.219+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 0x7fe2b8100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53036/0 (socket says 172.21.15.22:53036) 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 -- 172.21.15.22:0/2999839422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b810bb60 msgr2=0x7fe2b8100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.221 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b810bb60 0x7fe2b8100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 -- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 msgr2=0x7fe2b8100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 0x7fe2b8100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 -- 172.21.15.22:0/2999839422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a8009560 con 0x7fe2b8103060 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.220+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400e9f0 tx=0x7fe2a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.221+0000 7fe2afe00640 1 -- 172.21.15.22:0/2999839422 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4009670 con 0x7fe2b8103060 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.221+0000 7fe2afe00640 1 -- 172.21.15.22:0/2999839422 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a4005500 con 0x7fe2b8103060 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.221+0000 7fe2afe00640 1 -- 172.21.15.22:0/2999839422 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4010430 con 0x7fe2b8103060 2024-03-24T06:46:37.222 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.221+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b81a65f0 con 0x7fe2b8103060 2024-03-24T06:46:37.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.221+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b81a6b00 con 0x7fe2b8103060 2024-03-24T06:46:37.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 Processor -- start 2024-03-24T06:46:37.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 -- start start 2024-03-24T06:46:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 0x55a5a57a2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a57a2950 0x55a5a57a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a5a57076e0 con 0x55a5a57a2950 2024-03-24T06:46:37.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5a5704c10 con 0x55a5a5799d70 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.224+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a5a57003f0 con 0x55a5a5799450 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48944/0 (socket says 172.21.15.22:48944) 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1650572617 learned_addr learned my addr 172.21.15.22:0/1650572617 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7400640 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 0x55a5a57a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1650572617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 msgr2=0x55a5a57a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 0x55a5a57a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.226 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1650572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a57a2950 msgr2=0x55a5a57a4d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a57a2950 0x55a5a57a4d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1650572617 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a56c1e40 con 0x55a5a5799450 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7400640 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 0x55a5a57a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2afe00640 1 -- 172.21.15.22:0/2999839422 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe2a4005020 con 0x7fe2b8103060 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.225+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009f70 tx=0x7fe2b0010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96a6b8645c8f20a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.226+0000 7fe2acc00640 1 -- 172.21.15.22:0/1650572617 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0004d20 con 0x55a5a5799450 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.226+0000 7fe2acc00640 1 -- 172.21.15.22:0/1650572617 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b0004e80 con 0x55a5a5799450 2024-03-24T06:46:37.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.226+0000 7fe2acc00640 1 -- 172.21.15.22:0/1650572617 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00129b0 con 0x55a5a5799450 2024-03-24T06:46:37.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.226+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1650572617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 msgr2=0x55a5a5799830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.226+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009f70 tx=0x7fe2b0010800 comp rx=0 tx=0).stop 2024-03-24T06:46:37.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1650572617 shutdown_connections 2024-03-24T06:46:37.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a57a2950 0x55a5a57a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799d70 0x55a5a57a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1650572617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a5799830 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1650572617 >> 172.21.15.22:0/1650572617 conn(0x55a5a5793b80 msgr2=0x55a5a5793f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1650572617 shutdown_connections 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1650572617 wait complete. 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2afe00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 0x7fe29c0a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2afe00640 1 -- 172.21.15.22:0/2999839422 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2a4014070 con 0x7fe2b8103060 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2b7400640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 0x7fe29c0a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.227+0000 7fe2c11e2740 1 -- start start 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 Processor -- start 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799450 0x55a5a579fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2b7400640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 0x7fe29c0a4680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8005ab0 tx=0x7fe2a8005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a5a56a5060 con 0x55a5a5799450 2024-03-24T06:46:37.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5a56a4d00 con 0x55a5a57a2950 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a5a546e1c0 con 0x55a5a5799d70 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.228+0000 7fe2c11e2740 10 client.0 _ll_register_callbacks cb 0x55a5a5410be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799450 0x55a5a579fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48958/0 (socket says 172.21.15.22:48958) 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 -- 172.21.15.22:0/3882152676 learned_addr learned my addr 172.21.15.22:0/3882152676 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 -- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 msgr2=0x55a5a57a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 -- 172.21.15.22:0/3882152676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799450 msgr2=0x55a5a579fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799450 0x55a5a579fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 -- 172.21.15.22:0/3882152676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b0009bd0 con 0x55a5a5799d70 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40cbb40 tx=0x7fe2a40057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f83710d26d8a6517 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.229+0000 7fe2a1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a409eed0 con 0x55a5a5799d70 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2a40a3520 con 0x55a5a5799d70 2024-03-24T06:46:37.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5a543c720 con 0x55a5a5799d70 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe2a4099080 con 0x55a5a5799d70 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 10 client.26902 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a5a56fea70 con 0x55a5a5799d70 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a5a543c720 con 0x55a5a5799d70 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe2a401f060 con 0x55a5a5799d70 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 10 client.26902 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 10 client.26902 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 20 client.26902 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:37.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a57b2df0 0x55a5a57b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 4 client.26902 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.230+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a548b890 con 0x55a5a57b2df0 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.231+0000 7fe2b7400640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a57b2df0 0x55a5a57b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.231+0000 7fe2b7400640 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a57b2df0 0x55a5a57b5210 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5743ad0edb201372 server_cookie=2fa35daf5e2a9765 in_seq=0 out_seq=0 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.231+0000 7fe2a1600640 10 client.26902 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.232+0000 7fe2a1600640 1 -- 172.21.15.22:0/3882152676 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5a548b890 con 0x55a5a57b2df0 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.232+0000 7fe2a1600640 10 client.26902 handle_command_reply: tid=0 2024-03-24T06:46:37.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.232+0000 7fe2c11e2740 1 client.26902 shutdown 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.232+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 msgr2=0x55a5a579ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.232+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40cbb40 tx=0x7fe2a40057b0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 shutdown_connections 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a57b2df0 0x55a5a57b5210 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799d70 0x55a5a579ff90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/3882152676 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799450 0x55a5a579fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 >> 172.21.15.22:0/3882152676 conn(0x55a5a5793b80 msgr2=0x55a5a57958d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2a1600640 0 client.26902 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:37.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 shutdown_connections 2024-03-24T06:46:37.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 1 -- 172.21.15.22:0/3882152676 wait complete. 2024-03-24T06:46:37.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.233+0000 7fe2c11e2740 20 client.26902 trim_cache size 0 max 16384 2024-03-24T06:46:37.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 Processor -- start 2024-03-24T06:46:37.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 -- start start 2024-03-24T06:46:37.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 0x55a5a57bab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a5a5793440 con 0x55a5a5799d70 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5a579b6c0 con 0x55a5a5799450 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a5a578f4b0 con 0x55a5a57a2950 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 0x55a5a57bab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35122/0 (socket says 172.21.15.22:35122) 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2b7400640 1 -- 172.21.15.22:0/1152917722 learned_addr learned my addr 172.21.15.22:0/1152917722 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.248+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1152917722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 msgr2=0x55a5a57a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1152917722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 msgr2=0x55a5a57ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 -- 172.21.15.22:0/1152917722 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a569ab00 con 0x55a5a57a2950 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7400640 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:37.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 0x55a5a57bab20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0010d00 tx=0x7fe2b0012590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa8b973bbabfe5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2a3e00640 1 -- 172.21.15.22:0/1152917722 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0004d00 con 0x55a5a57a2950 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2a3e00640 1 -- 172.21.15.22:0/1152917722 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2b001f040 con 0x55a5a57a2950 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.249+0000 7fe2a3e00640 1 -- 172.21.15.22:0/1152917722 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b00056b0 con 0x55a5a57a2950 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1152917722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 msgr2=0x55a5a57bab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 0x55a5a57bab20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0010d00 tx=0x7fe2b0012590 comp rx=0 tx=0).stop 2024-03-24T06:46:37.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1152917722 shutdown_connections 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a57a2950 0x55a5a57bab20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/1152917722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a5799450 0x55a5a57a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.250+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1152917722 >> 172.21.15.22:0/1152917722 conn(0x55a5a5793b80 msgr2=0x55a5a57a02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.251+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1152917722 shutdown_connections 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.251+0000 7fe2c11e2740 1 -- 172.21.15.22:0/1152917722 wait complete. 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.251+0000 7fe2c11e2740 1 -- start start 2024-03-24T06:46:37.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.251+0000 7fe2c11e2740 1 Processor -- start 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a5a57aadd0 con 0x55a5a5799d70 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5a57aaf40 con 0x55a5a57a2950 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a5a57a8260 con 0x55a5a5799450 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2c11e2740 10 client.0 _ll_register_callbacks cb 0x7fe2a400cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.252+0000 7fe2b7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48970/0 (socket says 172.21.15.22:48970) 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 -- 172.21.15.22:0/2711287909 learned_addr learned my addr 172.21.15.22:0/2711287909 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7e00640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 -- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 msgr2=0x55a5a57a7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 -- 172.21.15.22:0/2711287909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 msgr2=0x55a5a57a77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57a77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 -- 172.21.15.22:0/2711287909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b810ec80 con 0x55a5a5799450 2024-03-24T06:46:37.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2b7400640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe2a802c210 tx=0x7fe2a802e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80de29fe6fb76d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2a1600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.253+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8054040 con 0x55a5a5799450 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5a53ff7e0 con 0x55a5a5799450 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fe2a8054a30 con 0x55a5a5799450 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe2a8054c30 con 0x55a5a5799450 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 10 client.26908 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:37.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a5a56c1e40 con 0x55a5a5799450 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a5a53ff7e0 con 0x55a5a5799450 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe2a805a020 con 0x55a5a5799450 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 10 client.26908 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 10 client.26908 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 20 client.26908 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a581a390 0x55a5a581c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 4 client.26908 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.254+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5a53ab700 con 0x55a5a581a390 2024-03-24T06:46:37.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.255+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a581a390 0x55a5a581c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.255+0000 7fe2b6a00640 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a581a390 0x55a5a581c7b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac66f45a84d074b server_cookie=231889bbb39369ff in_seq=0 out_seq=0 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.255+0000 7fe2a1600640 10 client.26908 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.255+0000 7fe2a1600640 1 -- 172.21.15.22:0/2711287909 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5a53ab700 con 0x55a5a581a390 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.255+0000 7fe2a1600640 10 client.26908 handle_command_reply: tid=0 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 client.26908 shutdown 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 msgr2=0x55a5a57aa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe2a802c210 tx=0x7fe2a802e050 comp rx=0 tx=0).stop 2024-03-24T06:46:37.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 shutdown_connections 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55a5a581a390 0x55a5a581c7b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5a57a2950 0x55a5a57a7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a5a5799d70 0x55a5a57a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 --2- 172.21.15.22:0/2711287909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a5a5799450 0x55a5a57aa810 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 >> 172.21.15.22:0/2711287909 conn(0x55a5a5793b80 msgr2=0x55a5a579fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2a1600640 0 client.26908 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 shutdown_connections 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 1 -- 172.21.15.22:0/2711287909 wait complete. 2024-03-24T06:46:37.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.256+0000 7fe2c11e2740 20 client.26908 trim_cache size 0 max 16384 2024-03-24T06:46:37.263 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:37.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 msgr2=0x7fe29c0a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 0x7fe29c0a4680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8005ab0 tx=0x7fe2a8005a20 comp rx=0 tx=0).stop 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 msgr2=0x7fe2b81008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400e9f0 tx=0x7fe2a400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 shutdown_connections 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe29c0a21c0 0x7fe29c0a4680 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2b810bb60 0x7fe2b8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2b8103060 0x7fe2b81008d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 --2- 172.21.15.22:0/2999839422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe2b8102740 0x7fe2b8100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 >> 172.21.15.22:0/2999839422 conn(0x7fe2b80fbf60 msgr2=0x7fe2b80fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 shutdown_connections 2024-03-24T06:46:37.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.257+0000 7fe2bce00640 1 -- 172.21.15.22:0/2999839422 wait complete. 2024-03-24T06:46:37.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:46:37.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.486+0000 7f852b200640 1 -- 172.21.15.22:0/371027879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c1015a0 msgr2=0x7f852c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.486+0000 7f852b200640 1 --2- 172.21.15.22:0/371027879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c1015a0 0x7f852c101a20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8518009930 tx=0x7f851802bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 -- 172.21.15.22:0/371027879 shutdown_connections 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 --2- 172.21.15.22:0/371027879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 --2- 172.21.15.22:0/371027879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c1015a0 0x7f852c101a20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 --2- 172.21.15.22:0/371027879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c107660 0x7f852c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 -- 172.21.15.22:0/371027879 >> 172.21.15.22:0/371027879 conn(0x7f852c0fd0b0 msgr2=0x7f852c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 -- 172.21.15.22:0/371027879 shutdown_connections 2024-03-24T06:46:37.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.487+0000 7f852b200640 1 -- 172.21.15.22:0/371027879 wait complete. 2024-03-24T06:46:37.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.488+0000 7f852b200640 1 Processor -- start 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.488+0000 7f852b200640 1 -- start start 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c1015a0 0x7f852c19d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c107660 0x7f852c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f852c197fe0 con 0x7f852c1015a0 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f852c198150 con 0x7f852c107660 2024-03-24T06:46:37.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f852c198310 con 0x7f852c101f60 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f8529400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f8529e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c1015a0 0x7f852c19d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f8529400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48988/0 (socket says 172.21.15.22:48988) 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f8529400640 1 -- 172.21.15.22:0/1023620192 learned_addr learned my addr 172.21.15.22:0/1023620192 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.489+0000 7f852a800640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c107660 0x7f852c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 -- 172.21.15.22:0/1023620192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c107660 msgr2=0x7f852c197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c107660 0x7f852c197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 -- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c1015a0 msgr2=0x7f852c19d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c1015a0 0x7f852c19d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 -- 172.21.15.22:0/1023620192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8518009560 con 0x7f852c101f60 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8529400640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8518000c00 tx=0x7f851802c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8522a00640 1 -- 172.21.15.22:0/1023620192 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85180042d0 con 0x7f852c101f60 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8522a00640 1 -- 172.21.15.22:0/1023620192 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8518004430 con 0x7f852c101f60 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f8522a00640 1 -- 172.21.15.22:0/1023620192 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518003c20 con 0x7f852c101f60 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c198590 con 0x7f852c101f60 2024-03-24T06:46:37.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.490+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c1a4880 con 0x7f852c101f60 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8522a00640 1 -- 172.21.15.22:0/1023620192 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8518004df0 con 0x7f852c101f60 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 Processor -- start 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 -- start start 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2d6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5612df2446e0 con 0x5612df2d6260 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5612df241c10 con 0x5612df2d6c10 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.494+0000 7f8533e0f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612df23d3f0 con 0x5612df2e3d60 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f852a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2d6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35152/0 (socket says 172.21.15.22:35152) 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 -- 172.21.15.22:0/2123329871 learned_addr learned my addr 172.21.15.22:0/2123329871 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 -- 172.21.15.22:0/2123329871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 msgr2=0x5612df2e6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 -- 172.21.15.22:0/2123329871 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 msgr2=0x5612df2d6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2d6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 -- 172.21.15.22:0/2123329871 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612df1fee40 con 0x5612df2d6c10 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8522a00640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 0x7f85100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.495+0000 7f8529e00640 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f851c009cc0 tx=0x7f851c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6caf778c34e6ea32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8522a00640 1 -- 172.21.15.22:0/1023620192 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f85180f0970 con 0x7f852c101f60 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8517400640 1 -- 172.21.15.22:0/2123329871 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c005220 con 0x5612df2d6c10 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8517400640 1 -- 172.21.15.22:0/2123329871 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851c004c80 con 0x5612df2d6c10 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8517400640 1 -- 172.21.15.22:0/2123329871 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c0198d0 con 0x5612df2d6c10 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f852a800640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 0x7f85100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8533e0f740 1 -- 172.21.15.22:0/2123329871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 msgr2=0x5612df2e3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.496+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f851c009cc0 tx=0x7f851c011810 comp rx=0 tx=0).stop 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 -- 172.21.15.22:0/2123329871 shutdown_connections 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e3820 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2123329871 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2d6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 -- 172.21.15.22:0/2123329871 >> 172.21.15.22:0/2123329871 conn(0x5612df2d0b80 msgr2=0x5612df2d0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f852a800640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 0x7f85100a46f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8524005fd0 tx=0x7f8524009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 -- 172.21.15.22:0/2123329871 shutdown_connections 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 -- 172.21.15.22:0/2123329871 wait complete. 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 -- start start 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.497+0000 7f8533e0f740 1 Processor -- start 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6c10 0x5612df2e1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2e3d60 0x5612df2e1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5612df1e2060 con 0x5612df2e3d60 2024-03-24T06:46:37.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5612df1e1d00 con 0x5612df2d6260 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612defab1c0 con 0x5612df2d6c10 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8533e0f740 10 client.0 _ll_register_callbacks cb 0x5612def4dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8529e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f852a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2e3d60 0x5612df2e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8529400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6c10 0x5612df2e1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8529e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35164/0 (socket says 172.21.15.22:35164) 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.498+0000 7f8529e00640 1 -- 172.21.15.22:0/1700984640 learned_addr learned my addr 172.21.15.22:0/1700984640 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 -- 172.21.15.22:0/1700984640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6c10 msgr2=0x5612df2e1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6c10 0x5612df2e1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 -- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2e3d60 msgr2=0x5612df2e1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2e3d60 0x5612df2e1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 -- 172.21.15.22:0/1700984640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851c0098f0 con 0x5612df2d6260 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8529e00640 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f851c005020 tx=0x7f851c005050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69059578b71b924d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f850be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c0055a0 con 0x5612df2d6260 2024-03-24T06:46:37.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612def3c7e0 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f851c0195d0 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.499+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f851c004ad0 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 10 client.26900 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5612df23ba70 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5612def3c7e0 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f851c0245c0 con 0x5612df2d6260 2024-03-24T06:46:37.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 10 client.26900 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 10 client.26900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 20 client.26900 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df2f4130 0x5612df2f6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 4 client.26900 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612def79720 con 0x5612df2f4130 2024-03-24T06:46:37.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.500+0000 7f8529400640 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df2f4130 0x5612df2f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.501+0000 7f8529400640 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df2f4130 0x5612df2f6540 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4063cbafae14f7e0 server_cookie=2365dc3bdb24c34e in_seq=0 out_seq=0 2024-03-24T06:46:37.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.501+0000 7f850be00640 10 client.26900 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f850be00640 1 -- 172.21.15.22:0/1700984640 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612def79720 con 0x5612df2f4130 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f850be00640 10 client.26900 handle_command_reply: tid=0 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 client.26900 shutdown 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 msgr2=0x5612df2e0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f851c005020 tx=0x7f851c005050 comp rx=0 tx=0).stop 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 shutdown_connections 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2e3d60 0x5612df2e1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6c10 0x5612df2e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df2f4130 0x5612df2f6540 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1700984640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6260 0x5612df2e0e00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f850be00640 0 client.26900 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 >> 172.21.15.22:0/1700984640 conn(0x5612df2d0b80 msgr2=0x5612df2d28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 shutdown_connections 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.502+0000 7f8533e0f740 1 -- 172.21.15.22:0/1700984640 wait complete. 2024-03-24T06:46:37.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.503+0000 7f8533e0f740 20 client.26900 trim_cache size 0 max 16384 2024-03-24T06:46:37.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.517+0000 7f8533e0f740 1 Processor -- start 2024-03-24T06:46:37.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.517+0000 7f8533e0f740 1 -- start start 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 0x5612df2ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2e3d60 0x5612df2fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5612df2e1dd0 con 0x5612df2d6c10 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5612df2d57b0 con 0x5612df2e3d60 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8533e0f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612df2f0ce0 con 0x5612df2d6260 2024-03-24T06:46:37.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 0x5612df2ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49004/0 (socket says 172.21.15.22:49004) 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 -- 172.21.15.22:0/2704766885 learned_addr learned my addr 172.21.15.22:0/2704766885 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f852a800640 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2e3d60 0x5612df2fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 -- 172.21.15.22:0/2704766885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2e3d60 msgr2=0x5612df2fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2e3d60 0x5612df2fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.518+0000 7f8529e00640 1 -- 172.21.15.22:0/2704766885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 msgr2=0x5612df2ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8529e00640 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 0x5612df2ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8529e00640 1 -- 172.21.15.22:0/2704766885 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612df1d7b00 con 0x5612df2d6260 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8529400640 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 0x5612df2ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8529e00640 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f851c005460 tx=0x7f851c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59885e8ca9cede32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8516a00640 1 -- 172.21.15.22:0/2704766885 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c009e70 con 0x5612df2d6260 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8516a00640 1 -- 172.21.15.22:0/2704766885 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851c005550 con 0x5612df2d6260 2024-03-24T06:46:37.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.519+0000 7f8516a00640 1 -- 172.21.15.22:0/2704766885 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c019aa0 con 0x5612df2d6260 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- 172.21.15.22:0/2704766885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 msgr2=0x5612df2e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f851c005460 tx=0x7f851c005fd0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- 172.21.15.22:0/2704766885 shutdown_connections 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2e3d60 0x5612df2fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6c10 0x5612df2ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 --2- 172.21.15.22:0/2704766885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2d6260 0x5612df2e3530 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- 172.21.15.22:0/2704766885 >> 172.21.15.22:0/2704766885 conn(0x5612df2d0b80 msgr2=0x5612df2cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- 172.21.15.22:0/2704766885 shutdown_connections 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- 172.21.15.22:0/2704766885 wait complete. 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 -- start start 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.520+0000 7f8533e0f740 1 Processor -- start 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5612df359440 con 0x5612df2d6260 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5612df359580 con 0x5612df2d6c10 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612df3596c0 con 0x5612df2e3d60 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8533e0f740 10 client.0 _ll_register_callbacks cb 0x7f8524009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8529e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8529e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53110/0 (socket says 172.21.15.22:53110) 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8529e00640 1 -- 172.21.15.22:0/1150229875 learned_addr learned my addr 172.21.15.22:0/1150229875 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:37.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f852a800640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.521+0000 7f8529400640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 -- 172.21.15.22:0/1150229875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 msgr2=0x5612df2e7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 -- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 msgr2=0x5612df2e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 -- 172.21.15.22:0/1150229875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c1a4880 con 0x5612df2e3d60 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f8529e00640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f852a800640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e7f80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x5612df2f6a30 tx=0x7f8524004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb1550ea9cab96d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:37.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f850be00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524024070 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f851c004c80 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f8524004b20 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8524016070 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f8533e0f740 10 client.26926 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.522+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5612df23ba70 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f851c004c80 con 0x5612df2e3d60 2024-03-24T06:46:37.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8524005490 con 0x5612df2e3d60 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 10 client.26926 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 10 client.26926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 20 client.26926 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df35af90 0x5612df35d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 4 client.26926 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612defc8890 con 0x5612df35af90 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.523+0000 7f8529e00640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df35af90 0x5612df35d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:37.530 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8529e00640 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df35af90 0x5612df35d3b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fca2ea6465163f3 server_cookie=6b4f05e074db4ab6 in_seq=0 out_seq=0 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f850be00640 10 client.26926 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f850be00640 1 -- 172.21.15.22:0/1150229875 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612defc8890 con 0x5612df35af90 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f850be00640 10 client.26926 handle_command_reply: tid=0 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 client.26926 shutdown 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 msgr2=0x5612df2e7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e7f80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x5612df2f6a30 tx=0x7f8524004a40 comp rx=0 tx=0).stop 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 shutdown_connections 2024-03-24T06:46:37.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612df2e3d60 0x5612df2e7f80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5612df35af90 0x5612df35d3b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5612df2d6c10 0x5612df2e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 --2- 172.21.15.22:0/1150229875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5612df2d6260 0x5612df2e7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f850be00640 0 client.26926 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.524+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 >> 172.21.15.22:0/1150229875 conn(0x5612df2d0b80 msgr2=0x5612df2cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 shutdown_connections 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f8533e0f740 1 -- 172.21.15.22:0/1150229875 wait complete. 2024-03-24T06:46:37.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f8533e0f740 20 client.26926 trim_cache size 0 max 16384 2024-03-24T06:46:37.533 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:46:37.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 msgr2=0x7f85100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 0x7f85100a46f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8524005fd0 tx=0x7f8524009500 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 msgr2=0x7f852c19dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8518000c00 tx=0x7f851802c5f0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 shutdown_connections 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f85100a2230 0x7f85100a46f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f852c107660 0x7f852c197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f852c101f60 0x7f852c19dd20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 --2- 172.21.15.22:0/1023620192 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f852c1015a0 0x7f852c19d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:37.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.525+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 >> 172.21.15.22:0/1023620192 conn(0x7f852c0fd0b0 msgr2=0x7f852c0fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:37.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.526+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 shutdown_connections 2024-03-24T06:46:37.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:37.526+0000 7f852b200640 1 -- 172.21.15.22:0/1023620192 wait complete. 2024-03-24T06:46:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:38 smithi143 ceph-mon[36520]: pgmap v203: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-03-24T06:46:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:38 smithi155 ceph-mon[36262]: pgmap v203: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-03-24T06:46:38.544 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:46:38.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:46:38.545 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:46:38.632 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:38 smithi022 ceph-mon[32003]: pgmap v203: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-03-24T06:46:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 -- 172.21.15.22:0/910374458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481076b0 msgr2=0x7f1048107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 --2- 172.21.15.22:0/910374458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481076b0 0x7f1048107a90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1038009930 tx=0x7f103802bc30 comp rx=0 tx=0).stop 2024-03-24T06:46:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 -- 172.21.15.22:0/910374458 shutdown_connections 2024-03-24T06:46:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 --2- 172.21.15.22:0/910374458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f104810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 --2- 172.21.15.22:0/910374458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481016a0 0x7f1048101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 --2- 172.21.15.22:0/910374458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481076b0 0x7f1048107a90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 -- 172.21.15.22:0/910374458 >> 172.21.15.22:0/910374458 conn(0x7f10480fd150 msgr2=0x7f10480ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 -- 172.21.15.22:0/910374458 shutdown_connections 2024-03-24T06:46:38.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.779+0000 7f104f000640 1 -- 172.21.15.22:0/910374458 wait complete. 2024-03-24T06:46:38.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.780+0000 7f104f000640 1 Processor -- start 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.780+0000 7f104f000640 1 -- start start 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f1048073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481076b0 0x7f1048074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f10481a6140 con 0x7f10481076b0 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f10481a62b0 con 0x7f1048102060 2024-03-24T06:46:38.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104f000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10481a6470 con 0x7f10481016a0 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f1047e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f1047e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49032/0 (socket says 172.21.15.22:49032) 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f1047e00640 1 -- 172.21.15.22:0/707501791 learned_addr learned my addr 172.21.15.22:0/707501791 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f1047400640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f1048073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.781+0000 7f104d000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481076b0 0x7f1048074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 -- 172.21.15.22:0/707501791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 msgr2=0x7f1048073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f1048073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 -- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481076b0 msgr2=0x7f1048074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481076b0 0x7f1048074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 -- 172.21.15.22:0/707501791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1038009560 con 0x7f10481016a0 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047400640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f1048073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1047e00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1038000c00 tx=0x7f103802cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1044c00640 1 -- 172.21.15.22:0/707501791 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038004350 con 0x7f10481016a0 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1044c00640 1 -- 172.21.15.22:0/707501791 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10380044b0 con 0x7f10481016a0 2024-03-24T06:46:38.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481a66f0 con 0x7f10481016a0 2024-03-24T06:46:38.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f1044c00640 1 -- 172.21.15.22:0/707501791 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038003d10 con 0x7f10481016a0 2024-03-24T06:46:38.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.782+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481a6c00 con 0x7f10481016a0 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.785+0000 7f1044c00640 1 -- 172.21.15.22:0/707501791 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1038004df0 con 0x7f10481016a0 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.785+0000 7f1051dd6740 1 Processor -- start 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 -- start start 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c8750 0x559e079d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 0x559e079d3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e079362d0 con 0x559e079d1330 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e07933800 con 0x559e079c7e30 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1051dd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e0792efe0 con 0x559e079c8750 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f104d000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1047e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c8750 0x559e079d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f104d000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35214/0 (socket says 172.21.15.22:35214) 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f104d000640 1 -- 172.21.15.22:0/2777640032 learned_addr learned my addr 172.21.15.22:0/2777640032 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.786+0000 7f1047400640 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 0x559e079d3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 -- 172.21.15.22:0/2777640032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c8750 msgr2=0x559e079d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c8750 0x559e079d0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 -- 172.21.15.22:0/2777640032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 msgr2=0x559e079d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 0x559e079d3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 -- 172.21.15.22:0/2777640032 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e078f0e90 con 0x559e079c7e30 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f1047400640 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 0x559e079d3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f104d000640 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1040009f50 tx=0x7f10400118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71542744526d23cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f1044c00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 0x7f102c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.787+0000 7f103d600640 1 -- 172.21.15.22:0/2777640032 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040004d20 con 0x559e079c7e30 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f103d600640 1 -- 172.21.15.22:0/2777640032 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1040004e80 con 0x559e079c7e30 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f103d600640 1 -- 172.21.15.22:0/2777640032 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040005220 con 0x559e079c7e30 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1044c00640 1 -- 172.21.15.22:0/707501791 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f10380f09b0 con 0x7f10481016a0 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1047e00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 0x7f102c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 -- 172.21.15.22:0/2777640032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 msgr2=0x559e079c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1040009f50 tx=0x7f10400118d0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1047e00640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 0x7f102c0a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10380f3fd0 tx=0x7f10380f6530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 -- 172.21.15.22:0/2777640032 shutdown_connections 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079d1330 0x559e079d3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c8750 0x559e079d0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 --2- 172.21.15.22:0/2777640032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079c8210 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 -- 172.21.15.22:0/2777640032 >> 172.21.15.22:0/2777640032 conn(0x559e079c2560 msgr2=0x559e079c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.788+0000 7f1051dd6740 1 -- 172.21.15.22:0/2777640032 shutdown_connections 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.789+0000 7f1051dd6740 1 -- 172.21.15.22:0/2777640032 wait complete. 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.789+0000 7f1051dd6740 1 -- start start 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.789+0000 7f1051dd6740 1 Processor -- start 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 0x559e079cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e078d3fc0 con 0x559e079c8750 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e078d3c60 con 0x559e079c7e30 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e076770c0 con 0x559e079d1330 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1051dd6740 10 client.0 _ll_register_callbacks cb 0x559e078ebe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f104d000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53152/0 (socket says 172.21.15.22:53152) 2024-03-24T06:46:38.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 -- 172.21.15.22:0/631221812 learned_addr learned my addr 172.21.15.22:0/631221812 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f104d000640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 0x559e079cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 -- 172.21.15.22:0/631221812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 msgr2=0x559e079cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 0x559e079cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 -- 172.21.15.22:0/631221812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 msgr2=0x559e079ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.790+0000 7f1047400640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1047400640 1 -- 172.21.15.22:0/631221812 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1040009c00 con 0x559e079c8750 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f104d000640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 0x559e079cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:38.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1047400640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10340127b0 tx=0x7f1034012c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60813af1068607e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1032000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034010e80 con 0x559e079c8750 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f103401c3f0 con 0x559e079c8750 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.791+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559e07925210 con 0x559e079c8750 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1034004020 con 0x559e079c8750 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 10 client.16551 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:38.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559e0792d660 con 0x559e079c8750 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559e07925210 con 0x559e079c8750 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f103401c5d0 con 0x559e079c8750 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 10 client.16551 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 20 client.16551 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079e18a0 0x559e079e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 4 client.16551 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:38.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.792+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e0762e860 con 0x559e079e18a0 2024-03-24T06:46:38.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.793+0000 7f104d000640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079e18a0 0x559e079e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.793+0000 7f104d000640 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079e18a0 0x559e079e3cc0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bab9fb212564833 server_cookie=744ac1b4debd5cda in_seq=0 out_seq=0 2024-03-24T06:46:38.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.793+0000 7f1032000640 10 client.16551 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:38.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1032000640 1 -- 172.21.15.22:0/631221812 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e0762e860 con 0x559e079e18a0 2024-03-24T06:46:38.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1032000640 10 client.16551 handle_command_reply: tid=0 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 client.16551 shutdown 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 msgr2=0x559e079ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f10340127b0 tx=0x7f1034012c70 comp rx=0 tx=0).stop 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 shutdown_connections 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079d1330 0x559e079cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079ce8b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079e18a0 0x559e079e3cc0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 --2- 172.21.15.22:0/631221812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 >> 172.21.15.22:0/631221812 conn(0x559e079c2560 msgr2=0x559e079c42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.794+0000 7f1032000640 0 client.16551 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.795+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 shutdown_connections 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.795+0000 7f1051dd6740 1 -- 172.21.15.22:0/631221812 wait complete. 2024-03-24T06:46:38.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.795+0000 7f1051dd6740 20 client.16551 trim_cache size 0 max 16384 2024-03-24T06:46:38.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 Processor -- start 2024-03-24T06:46:38.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 -- start start 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c2560 0x559e079d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079e95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e079cf2d0 con 0x559e079c8750 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e079bd770 con 0x559e079c7e30 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.814+0000 7f1051dd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e079c3560 con 0x559e079c2560 2024-03-24T06:46:38.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f104d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c2560 0x559e079d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35228/0 (socket says 172.21.15.22:35228) 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 -- 172.21.15.22:0/212946115 learned_addr learned my addr 172.21.15.22:0/212946115 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f104d000640 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079e95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 -- 172.21.15.22:0/212946115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c2560 msgr2=0x559e079d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c2560 0x559e079d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 -- 172.21.15.22:0/212946115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 msgr2=0x559e079e95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079e95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 -- 172.21.15.22:0/212946115 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e078c9ae0 con 0x559e079c7e30 2024-03-24T06:46:38.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f104d000640 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079e95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:38.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.815+0000 7f1047400640 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1034005c20 tx=0x7f103400bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ffd922d011b761 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f103cc00640 1 -- 172.21.15.22:0/212946115 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103401ce40 con 0x559e079c7e30 2024-03-24T06:46:38.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f103cc00640 1 -- 172.21.15.22:0/212946115 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1034005850 con 0x559e079c7e30 2024-03-24T06:46:38.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f103cc00640 1 -- 172.21.15.22:0/212946115 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103402da50 con 0x559e079c7e30 2024-03-24T06:46:38.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 -- 172.21.15.22:0/212946115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 msgr2=0x559e079dcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1034005c20 tx=0x7f103400bdd0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 -- 172.21.15.22:0/212946115 shutdown_connections 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c8750 0x559e079e95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c7e30 0x559e079dcbc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 --2- 172.21.15.22:0/212946115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c2560 0x559e079d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.816+0000 7f1051dd6740 1 -- 172.21.15.22:0/212946115 >> 172.21.15.22:0/212946115 conn(0x559e079c8ea0 msgr2=0x559e079cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.817+0000 7f1051dd6740 1 -- 172.21.15.22:0/212946115 shutdown_connections 2024-03-24T06:46:38.823 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.817+0000 7f1051dd6740 1 -- 172.21.15.22:0/212946115 wait complete. 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.817+0000 7f1051dd6740 1 -- start start 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.817+0000 7f1051dd6740 1 Processor -- start 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c2560 0x559e079d61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 0x559e07a47bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559e079d6ee0 con 0x559e079c2560 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559e079d7050 con 0x559e079c8750 2024-03-24T06:46:38.824 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559e07a48130 con 0x559e079c7e30 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1051dd6740 10 client.0 _ll_register_callbacks cb 0x7f1034005c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49064/0 (socket says 172.21.15.22:49064) 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 -- 172.21.15.22:0/73675231 learned_addr learned my addr 172.21.15.22:0/73675231 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f104d000640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c2560 0x559e079d61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f104d000640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 0x559e07a47bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 -- 172.21.15.22:0/73675231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 msgr2=0x559e07a47bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.825 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 0x559e07a47bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 -- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c2560 msgr2=0x559e079d61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c2560 0x559e079d61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f1047400640 1 -- 172.21.15.22:0/73675231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e0792d660 con 0x559e079c7e30 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.818+0000 7f104d000640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 0x559e07a47bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1047400640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f103401c9d0 tx=0x7f103402d470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bcbf1ea65c0520b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1032000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:38.826 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103403e070 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7f1034000e40 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559e07953e90 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f103401c7d0 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1051dd6740 10 client.26944 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.819+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559e078ef6f0 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559e07953e90 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1034041020 con 0x559e079c7e30 2024-03-24T06:46:38.827 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 10 client.26944 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 20 client.26944 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079d1330 0x559e07a4b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 4 client.26944 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559e0767f410 con 0x559e079d1330 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.820+0000 7f104d000640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079d1330 0x559e07a4b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:38.828 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.821+0000 7f104d000640 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079d1330 0x559e07a4b640 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96711bfb5e2dacf server_cookie=af90efbaea94be55 in_seq=0 out_seq=0 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.821+0000 7f1032000640 10 client.26944 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1032000640 1 -- 172.21.15.22:0/73675231 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559e0767f410 con 0x559e079d1330 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1032000640 10 client.26944 handle_command_reply: tid=0 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 client.26944 shutdown 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 msgr2=0x559e079d6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f103401c9d0 tx=0x7f103402d470 comp rx=0 tx=0).stop 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 shutdown_connections 2024-03-24T06:46:38.829 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559e079d1330 0x559e07a4b640 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559e079c8750 0x559e07a47bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e079c7e30 0x559e079d6700 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 --2- 172.21.15.22:0/73675231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559e079c2560 0x559e079d61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 >> 172.21.15.22:0/73675231 conn(0x559e079c8ea0 msgr2=0x559e079c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1032000640 0 client.26944 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 shutdown_connections 2024-03-24T06:46:38.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 1 -- 172.21.15.22:0/73675231 wait complete. 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.822+0000 7f1051dd6740 20 client.26944 trim_cache size 0 max 16384 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "ddd326c8-d37b-4d14-a82a-ce98702c81ea", 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:46:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:38.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.823+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 msgr2=0x7f102c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.823+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 0x7f102c0a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10380f3fd0 tx=0x7f10380f6530 comp rx=0 tx=0).stop 2024-03-24T06:46:38.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.823+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 msgr2=0x7f1048073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.823+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1038000c00 tx=0x7f103802cd80 comp rx=0 tx=0).stop 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 shutdown_connections 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f102c0a2230 0x7f102c0a46f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10481076b0 0x7f1048074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1048102060 0x7f1048073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 --2- 172.21.15.22:0/707501791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10481016a0 0x7f1048073a10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 >> 172.21.15.22:0/707501791 conn(0x7f10480fd150 msgr2=0x7f10480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 shutdown_connections 2024-03-24T06:46:38.833 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:38.824+0000 7f104f000640 1 -- 172.21.15.22:0/707501791 wait complete. 2024-03-24T06:46:38.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:46:38.848 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:46:39.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 -- 172.21.15.22:0/3331413752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 msgr2=0x7fbe2c100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.109 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 --2- 172.21.15.22:0/3331413752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c100700 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009900 tx=0x7fbe1c02bc40 comp rx=0 tx=0).stop 2024-03-24T06:46:39.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 -- 172.21.15.22:0/3331413752 shutdown_connections 2024-03-24T06:46:39.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 --2- 172.21.15.22:0/3331413752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 0x7fbe2c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 --2- 172.21.15.22:0/3331413752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c10d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.110 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 --2- 172.21.15.22:0/3331413752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c100700 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 -- 172.21.15.22:0/3331413752 >> 172.21.15.22:0/3331413752 conn(0x7fbe2c0fbf60 msgr2=0x7fbe2c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.109+0000 7fbe31a00640 1 -- 172.21.15.22:0/3331413752 shutdown_connections 2024-03-24T06:46:39.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.110+0000 7fbe31a00640 1 -- 172.21.15.22:0/3331413752 wait complete. 2024-03-24T06:46:39.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.110+0000 7fbe31a00640 1 Processor -- start 2024-03-24T06:46:39.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 -- start start 2024-03-24T06:46:39.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 0x7fbe2c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c19ecc0 con 0x7fbe2c100cd0 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c19ee30 con 0x7fbe2c10ded0 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe31a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c19eff0 con 0x7fbe2c100320 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49084/0 (socket says 172.21.15.22:49084) 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.111+0000 7fbe2be00640 1 -- 172.21.15.22:0/2385520804 learned_addr learned my addr 172.21.15.22:0/2385520804 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:39.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe31000640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 0x7fbe2c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 -- 172.21.15.22:0/2385520804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 msgr2=0x7fbe2c19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 0x7fbe2c19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 -- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 msgr2=0x7fbe2c19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 -- 172.21.15.22:0/2385520804 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1c009560 con 0x7fbe2c100320 2024-03-24T06:46:39.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe23e00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe2be00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c000c00 tx=0x7fbe1c02c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.112+0000 7fbe29600640 1 -- 172.21.15.22:0/2385520804 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c004390 con 0x7fbe2c100320 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.113+0000 7fbe29600640 1 -- 172.21.15.22:0/2385520804 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe1c0044f0 con 0x7fbe2c100320 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.113+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c19f270 con 0x7fbe2c100320 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.113+0000 7fbe29600640 1 -- 172.21.15.22:0/2385520804 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c003c90 con 0x7fbe2c100320 2024-03-24T06:46:39.117 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.113+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c1a6b50 con 0x7fbe2c100320 2024-03-24T06:46:39.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.116+0000 7fbe29600640 1 -- 172.21.15.22:0/2385520804 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fbe1c004df0 con 0x7fbe2c100320 2024-03-24T06:46:39.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.117+0000 7fbe29600640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 0x7fbe100a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.117+0000 7fbe35e1a740 1 Processor -- start 2024-03-24T06:46:39.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.117+0000 7fbe29600640 1 -- 172.21.15.22:0/2385520804 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe1c0f09a0 con 0x7fbe2c100320 2024-03-24T06:46:39.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.117+0000 7fbe35e1a740 1 -- start start 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.117+0000 7fbe23e00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 0x7fbe100a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e01990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e023e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e0d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9d70790 con 0x55e1a9e01f60 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9d6dcc0 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.125 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9d694a0 con 0x55e1a9e02c40 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e01990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe31000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e0d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e01990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49086/0 (socket says 172.21.15.22:49086) 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe2be00640 1 -- 172.21.15.22:0/2923379776 learned_addr learned my addr 172.21.15.22:0/2923379776 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.118+0000 7fbe23e00640 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 -- 172.21.15.22:0/2923379776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 msgr2=0x55e1a9e01990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e01990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 -- 172.21.15.22:0/2923379776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 msgr2=0x55e1a9e023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe23e00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 0x7fbe100a4770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0ff530 tx=0x7fbe14011860 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 -- 172.21.15.22:0/2923379776 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1a9d2ac20 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe31000640 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e0d6c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005fd0 tx=0x7fbe24011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195ba4cbb9d262a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.119+0000 7fbe21600640 1 -- 172.21.15.22:0/2923379776 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24004d20 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe21600640 1 -- 172.21.15.22:0/2923379776 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe24004e80 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe21600640 1 -- 172.21.15.22:0/2923379776 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24005280 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 -- 172.21.15.22:0/2923379776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 msgr2=0x55e1a9e0d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e0d6c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbe24005fd0 tx=0x7fbe24011810 comp rx=0 tx=0).stop 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 -- 172.21.15.22:0/2923379776 shutdown_connections 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e0d6c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/2923379776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e01990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 -- 172.21.15.22:0/2923379776 >> 172.21.15.22:0/2923379776 conn(0x55e1a9dc8b30 msgr2=0x55e1a9dfc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.120+0000 7fbe35e1a740 1 -- 172.21.15.22:0/2923379776 shutdown_connections 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.121+0000 7fbe35e1a740 1 -- 172.21.15.22:0/2923379776 wait complete. 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.121+0000 7fbe35e1a740 1 -- start start 2024-03-24T06:46:39.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.121+0000 7fbe35e1a740 1 Processor -- start 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e08d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9d0df20 con 0x55e1a9e01f60 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9d0dbc0 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9ad71c0 con 0x55e1a9e02c40 2024-03-24T06:46:39.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe35e1a740 10 client.0 _ll_register_callbacks cb 0x55e1a9a79be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe31000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e08d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49102/0 (socket says 172.21.15.22:49102) 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 -- 172.21.15.22:0/592937626 learned_addr learned my addr 172.21.15.22:0/592937626 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe2be00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 -- 172.21.15.22:0/592937626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 msgr2=0x55e1a9e08d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e08d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 -- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 msgr2=0x55e1a9e082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e082c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe23e00640 1 -- 172.21.15.22:0/592937626 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe24009be0 con 0x55e1a9e02c40 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.122+0000 7fbe31000640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e08d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe23e00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400fc40 tx=0x7fbe14004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb620b4cf6230c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe1a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:39.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14022070 con 0x55e1a9e02c40 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e1a9a687e0 con 0x55e1a9e02c40 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe1401db50 con 0x55e1a9e02c40 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbe1401dd10 con 0x55e1a9e02c40 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe35e1a740 10 client.26956 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e1a9d67b20 con 0x55e1a9e02c40 2024-03-24T06:46:39.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.123+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e1a9a687e0 con 0x55e1a9e02c40 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fbe14003bc0 con 0x55e1a9e02c40 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 10 client.26956 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 10 client.26956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 20 client.26956 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e1b880 0x55e1a9e1dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 4 client.26956 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:46:39.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1a9aa5720 con 0x55e1a9e1b880 2024-03-24T06:46:39.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.124+0000 7fbe2be00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e1b880 0x55e1a9e1dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.125+0000 7fbe2be00640 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e1b880 0x55e1a9e1dce0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ebef81f1c2c38cc server_cookie=79212efefea5d734 in_seq=0 out_seq=0 2024-03-24T06:46:39.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.125+0000 7fbe1a000640 10 client.26956 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:39.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.125+0000 7fbe1a000640 1 -- 172.21.15.22:0/592937626 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e1a9aa5720 con 0x55e1a9e1b880 2024-03-24T06:46:39.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.125+0000 7fbe1a000640 10 client.26956 handle_command_reply: tid=0 2024-03-24T06:46:39.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.125+0000 7fbe35e1a740 1 client.26956 shutdown 2024-03-24T06:46:39.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 msgr2=0x55e1a9e08800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400fc40 tx=0x7fbe14004730 comp rx=0 tx=0).stop 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 shutdown_connections 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e1b880 0x55e1a9e1dce0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e08d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e08800 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe1a000640 0 client.26956 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:39.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/592937626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 >> 172.21.15.22:0/592937626 conn(0x55e1a9dc8b30 msgr2=0x55e1a9dfe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 shutdown_connections 2024-03-24T06:46:39.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 1 -- 172.21.15.22:0/592937626 wait complete. 2024-03-24T06:46:39.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.126+0000 7fbe35e1a740 20 client.26956 trim_cache size 0 max 16384 2024-03-24T06:46:39.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.147+0000 7fbe35e1a740 1 Processor -- start 2024-03-24T06:46:39.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.147+0000 7fbe35e1a740 1 -- start start 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 0x55e1a9e07220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9e09610 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9e1a0b0 con 0x55e1a9e02c40 2024-03-24T06:46:39.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9df85f0 con 0x55e1a9e01f60 2024-03-24T06:46:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 0x55e1a9e07220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35270/0 (socket says 172.21.15.22:35270) 2024-03-24T06:46:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe23e00640 1 -- 172.21.15.22:0/910699434 learned_addr learned my addr 172.21.15.22:0/910699434 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:39.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe31000640 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 -- 172.21.15.22:0/910699434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 msgr2=0x55e1a9e16c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 -- 172.21.15.22:0/910699434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e0b2d0 msgr2=0x55e1a9e235f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e235f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe2be00640 1 -- 172.21.15.22:0/910699434 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1a9d039a0 con 0x55e1a9e01f60 2024-03-24T06:46:39.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.148+0000 7fbe23e00640 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.149+0000 7fbe2be00640 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 0x55e1a9e07220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0b95d0 tx=0x7fbe1c003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf884c081c8b5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.149+0000 7fbe20c00640 1 -- 172.21.15.22:0/910699434 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0d2030 con 0x55e1a9e01f60 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.149+0000 7fbe20c00640 1 -- 172.21.15.22:0/910699434 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe1c0c12c0 con 0x55e1a9e01f60 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.149+0000 7fbe20c00640 1 -- 172.21.15.22:0/910699434 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0bb500 con 0x55e1a9e01f60 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- 172.21.15.22:0/910699434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 msgr2=0x55e1a9e07220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 0x55e1a9e07220 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c0b95d0 tx=0x7fbe1c003760 comp rx=0 tx=0).stop 2024-03-24T06:46:39.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- 172.21.15.22:0/910699434 shutdown_connections 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e235f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e02c40 0x55e1a9e16c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/910699434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e01f60 0x55e1a9e07220 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- 172.21.15.22:0/910699434 >> 172.21.15.22:0/910699434 conn(0x55e1a9dc8b30 msgr2=0x55e1a9dfbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- 172.21.15.22:0/910699434 shutdown_connections 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- 172.21.15.22:0/910699434 wait complete. 2024-03-24T06:46:39.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 -- start start 2024-03-24T06:46:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.150+0000 7fbe35e1a740 1 Processor -- start 2024-03-24T06:46:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e05560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e05fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9e0bd30 con 0x55e1a9e01f60 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9e0bea0 con 0x55e1a9e0b2d0 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e1a9e0c060 con 0x55e1a9e02c40 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe35e1a740 10 client.0 _ll_register_callbacks cb 0x7fbe1400f660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e05560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:49122/0 (socket says 172.21.15.22:49122) 2024-03-24T06:46:39.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 -- 172.21.15.22:0/3638591457 learned_addr learned my addr 172.21.15.22:0/3638591457 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe31000640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e05fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 -- 172.21.15.22:0/3638591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 msgr2=0x55e1a9e05fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e05fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 -- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 msgr2=0x55e1a9e05560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.151+0000 7fbe23e00640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e05560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe23e00640 1 -- 172.21.15.22:0/3638591457 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1a9d67b20 con 0x55e1a9e02c40 2024-03-24T06:46:39.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe31000640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e05fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe23e00640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbe140275a0 tx=0x7fbe140278d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71fad8524f367c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe1a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14022040 con 0x55e1a9e02c40 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e1a9d61160 con 0x55e1a9e02c40 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9379+0+0 (secure 0 0 0) 0x7fbe14012e40 con 0x55e1a9e02c40 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbe1401dd60 con 0x55e1a9e02c40 2024-03-24T06:46:39.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe35e1a740 10 client.26962 fetch_fsmap learned FSMap version 37 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.152+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x7fbe2c1a6b50 con 0x55e1a9e02c40 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e1a9d61160 con 0x55e1a9e02c40 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fbe140035f0 con 0x55e1a9e02c40 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 10 client.26962 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 10 client.26962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:46:39.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 20 client.26962 populate_metadata read hostname 'smithi022' 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e825a0 0x55e1a9e849c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 4 client.26962 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e1a9af4890 con 0x55e1a9e825a0 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.153+0000 7fbe2be00640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e825a0 0x55e1a9e849c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe2be00640 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e825a0 0x55e1a9e849c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b708320999fc1c2e server_cookie=3e4f0e88f74b11d0 in_seq=0 out_seq=0 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe1a000640 10 client.26962 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:39.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe1a000640 1 -- 172.21.15.22:0/3638591457 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e1a9af4890 con 0x55e1a9e825a0 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe1a000640 10 client.26962 handle_command_reply: tid=0 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 client.26962 shutdown 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 msgr2=0x55e1a9e05aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbe140275a0 tx=0x7fbe140278d0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 shutdown_connections 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55e1a9e825a0 0x55e1a9e849c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e1a9e0b2d0 0x55e1a9e05fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e1a9e02c40 0x55e1a9e05aa0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 --2- 172.21.15.22:0/3638591457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e1a9e01f60 0x55e1a9e05560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe1a000640 0 client.26962 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.154+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 >> 172.21.15.22:0/3638591457 conn(0x55e1a9dc8b30 msgr2=0x55e1a9e09750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 shutdown_connections 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe35e1a740 1 -- 172.21.15.22:0/3638591457 wait complete. 2024-03-24T06:46:39.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe35e1a740 20 client.26962 trim_cache size 0 max 16384 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "ddd326c8-d37b-4d14-a82a-ce98702c81ea": { 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:46:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "ddd326c8-d37b-4d14-a82a-ce98702c81ea", 2024-03-24T06:46:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:46:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:46:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:46:39.167 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 msgr2=0x7fbe100a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 0x7fbe100a4770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0ff530 tx=0x7fbe14011860 comp rx=0 tx=0).stop 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 msgr2=0x7fbe2c199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c000c00 tx=0x7fbe1c02c640 comp rx=0 tx=0).stop 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 shutdown_connections 2024-03-24T06:46:39.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fbe100a22b0 0x7fbe100a4770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe2c10ded0 0x7fbe2c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbe2c100cd0 0x7fbe2c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 --2- 172.21.15.22:0/2385520804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe2c100320 0x7fbe2c199e50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.155+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 >> 172.21.15.22:0/2385520804 conn(0x7fbe2c0fbf60 msgr2=0x7fbe2c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.156+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 shutdown_connections 2024-03-24T06:46:39.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:46:39.156+0000 7fbe31a00640 1 -- 172.21.15.22:0/2385520804 wait complete. 2024-03-24T06:46:39.173 INFO:tasks.cephfs.filesystem:scrub status for tag:ddd326c8-d37b-4d14-a82a-ce98702c81ea - {'path': '/', 'tag': 'ddd326c8-d37b-4d14-a82a-ce98702c81ea', 'options': 'recursive,force'} 2024-03-24T06:46:39.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:39.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:39.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:39.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:39 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:39.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:39.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:39.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:39 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:39.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:39.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:39.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:39.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:39 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:40 smithi155 ceph-mon[36262]: pgmap v204: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-24T06:46:40.553 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:46:40.553 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.553 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:40.553 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:46:40.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.554 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:40 smithi022 ceph-mon[32003]: pgmap v204: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-24T06:46:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:46:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:40.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:46:40.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:46:40.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:40 smithi143 ceph-mon[36520]: pgmap v204: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-03-24T06:46:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:42 smithi155 ceph-mon[36262]: pgmap v205: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-24T06:46:42.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:42 smithi022 ceph-mon[32003]: pgmap v205: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-24T06:46:42.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:42 smithi143 ceph-mon[36520]: pgmap v205: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-24T06:46:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:44 smithi155 ceph-mon[36262]: pgmap v206: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-24T06:46:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:44 smithi022 ceph-mon[32003]: pgmap v206: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-24T06:46:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:44 smithi143 ceph-mon[36520]: pgmap v206: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-24T06:46:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:46 smithi155 ceph-mon[36262]: pgmap v207: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-03-24T06:46:46.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:46 smithi022 ceph-mon[32003]: pgmap v207: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-03-24T06:46:46.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:46 smithi143 ceph-mon[36520]: pgmap v207: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 10 op/s 2024-03-24T06:46:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:48 smithi155 ceph-mon[36262]: pgmap v208: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-03-24T06:46:48.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:48 smithi022 ceph-mon[32003]: pgmap v208: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-03-24T06:46:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:48 smithi143 ceph-mon[36520]: pgmap v208: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-03-24T06:46:50.481 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:50 smithi143 ceph-mon[36520]: pgmap v209: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:50 smithi155 ceph-mon[36262]: pgmap v209: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:50.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:50 smithi022 ceph-mon[32003]: pgmap v209: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.7 KiB/s wr, 9 op/s 2024-03-24T06:46:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:52 smithi155 ceph-mon[36262]: pgmap v210: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:52.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:52 smithi022 ceph-mon[32003]: pgmap v210: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:52.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:52 smithi143 ceph-mon[36520]: pgmap v210: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-03-24T06:46:53.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:53 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:53.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:53 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:53.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:53 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:53.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:53 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:53 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:53 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:53 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:53.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:53 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:53.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:53 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:53.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:53 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:53.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:53 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:53.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:53 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:54 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:46:54.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:54 smithi155 ceph-mon[36262]: pgmap v211: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T06:46:54.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:54 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:46:54.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:54 smithi022 ceph-mon[32003]: pgmap v211: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T06:46:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:54 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:46:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:54 smithi143 ceph-mon[36520]: pgmap v211: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T06:46:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:56 smithi155 ceph-mon[36262]: pgmap v212: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T06:46:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:56 smithi022 ceph-mon[32003]: pgmap v212: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T06:46:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:56 smithi143 ceph-mon[36520]: pgmap v212: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-24T06:46:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:46:58 smithi155 ceph-mon[36262]: pgmap v213: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T06:46:58.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:46:58 smithi022 ceph-mon[32003]: pgmap v213: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T06:46:58.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:46:58 smithi143 ceph-mon[36520]: pgmap v213: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T06:47:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:00 smithi155 ceph-mon[36262]: pgmap v214: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-24T06:47:00.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:00 smithi022 ceph-mon[32003]: pgmap v214: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-24T06:47:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:00 smithi143 ceph-mon[36520]: pgmap v214: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2024-03-24T06:47:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:02 smithi155 ceph-mon[36262]: pgmap v215: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-24T06:47:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:02 smithi022 ceph-mon[32003]: pgmap v215: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-24T06:47:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:02 smithi143 ceph-mon[36520]: pgmap v215: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-24T06:47:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:04 smithi155 ceph-mon[36262]: pgmap v216: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-24T06:47:04.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:04 smithi022 ceph-mon[32003]: pgmap v216: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-24T06:47:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:04 smithi143 ceph-mon[36520]: pgmap v216: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-24T06:47:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:06 smithi155 ceph-mon[36262]: pgmap v217: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:47:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:06 smithi022 ceph-mon[32003]: pgmap v217: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:47:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:06 smithi143 ceph-mon[36520]: pgmap v217: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:47:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:08 smithi155 ceph-mon[36262]: pgmap v218: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:47:08.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:08 smithi022 ceph-mon[32003]: pgmap v218: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:47:08.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:08 smithi143 ceph-mon[36520]: pgmap v218: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:47:09.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:47:09.175 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:47:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 -- 172.21.15.22:0/347785877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb254107ec0 msgr2=0x7fb2541082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 --2- 172.21.15.22:0/347785877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb254107ec0 0x7fb2541082a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb248009900 tx=0x7fb24802bc40 comp rx=0 tx=0).stop 2024-03-24T06:47:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 -- 172.21.15.22:0/347785877 shutdown_connections 2024-03-24T06:47:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 --2- 172.21.15.22:0/347785877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb25410cf10 0x7fb254114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 --2- 172.21.15.22:0/347785877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254108870 0x7fb25410c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 --2- 172.21.15.22:0/347785877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb254107ec0 0x7fb2541082a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 -- 172.21.15.22:0/347785877 >> 172.21.15.22:0/347785877 conn(0x7fb254077530 msgr2=0x7fb254106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 -- 172.21.15.22:0/347785877 shutdown_connections 2024-03-24T06:47:09.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.410+0000 7fb25ac00640 1 -- 172.21.15.22:0/347785877 wait complete. 2024-03-24T06:47:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.411+0000 7fb25ac00640 1 Processor -- start 2024-03-24T06:47:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 -- start start 2024-03-24T06:47:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb25410cf10 0x7fb25419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 0x7fb2541a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a34d0 con 0x7fb2541a2ae0 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a3640 con 0x7fb254107ec0 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25ac00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a37b0 con 0x7fb25410cf10 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb259800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb259800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60540/0 (socket says 172.21.15.22:60540) 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb258e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb25410cf10 0x7fb25419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb259800640 1 -- 172.21.15.22:0/1229619311 learned_addr learned my addr 172.21.15.22:0/1229619311 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.412+0000 7fb25a200640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 0x7fb2541a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 -- 172.21.15.22:0/1229619311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb25410cf10 msgr2=0x7fb25419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb25410cf10 0x7fb25419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 -- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 msgr2=0x7fb2541a2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 0x7fb2541a2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 -- 172.21.15.22:0/1229619311 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb248009560 con 0x7fb254107ec0 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb25a200640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 0x7fb2541a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb259800640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb248005940 tx=0x7fb24802c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb252000640 1 -- 172.21.15.22:0/1229619311 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480043d0 con 0x7fb254107ec0 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb252000640 1 -- 172.21.15.22:0/1229619311 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2480052d0 con 0x7fb254107ec0 2024-03-24T06:47:09.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2541aad90 con 0x7fb254107ec0 2024-03-24T06:47:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.413+0000 7fb252000640 1 -- 172.21.15.22:0/1229619311 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248003cc0 con 0x7fb254107ec0 2024-03-24T06:47:09.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.414+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2541ab1e0 con 0x7fb254107ec0 2024-03-24T06:47:09.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.418+0000 7fb25ef7c740 1 Processor -- start 2024-03-24T06:47:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.418+0000 7fb252000640 1 -- 172.21.15.22:0/1229619311 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb248005440 con 0x7fb254107ec0 2024-03-24T06:47:09.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.418+0000 7fb25ef7c740 1 -- start start 2024-03-24T06:47:09.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 0x555987d70990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d7a2d0 0x555987d7c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555987cdf790 con 0x555987d71c40 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555987cdccc0 con 0x555987d7a2d0 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555987cd84a0 con 0x555987d70f60 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb259800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb25a200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 0x555987d70990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb259800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48282/0 (socket says 172.21.15.22:48282) 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb259800640 1 -- 172.21.15.22:0/1462901869 learned_addr learned my addr 172.21.15.22:0/1462901869 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.419+0000 7fb258e00640 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d7a2d0 0x555987d7c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 -- 172.21.15.22:0/1462901869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d7a2d0 msgr2=0x555987d7c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d7a2d0 0x555987d7c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 -- 172.21.15.22:0/1462901869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 msgr2=0x555987d70990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 0x555987d70990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 -- 172.21.15.22:0/1462901869 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555987c99c20 con 0x555987d70f60 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb25a200640 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 0x555987d70990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb252000640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 0x7fb23c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb259800640 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb2480ed850 tx=0x7fb2480f6550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3e9ed02960b593 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb252000640 1 -- 172.21.15.22:0/1229619311 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb2480f0520 con 0x7fb254107ec0 2024-03-24T06:47:09.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb25a200640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 0x7fb23c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.420+0000 7fb242a00640 1 -- 172.21.15.22:0/1462901869 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480b8e50 con 0x555987d70f60 2024-03-24T06:47:09.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb242a00640 1 -- 172.21.15.22:0/1462901869 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2480bc980 con 0x555987d70f60 2024-03-24T06:47:09.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb242a00640 1 -- 172.21.15.22:0/1462901869 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480bc510 con 0x555987d70f60 2024-03-24T06:47:09.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 -- 172.21.15.22:0/1462901869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 msgr2=0x555987d713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb2480ed850 tx=0x7fb2480f6550 comp rx=0 tx=0).stop 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25a200640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 0x7fb23c0a4770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb24c005fd0 tx=0x7fb24c00b4d0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 -- 172.21.15.22:0/1462901869 shutdown_connections 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d7a2d0 0x555987d7c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d70f60 0x555987d713e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/1462901869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d71c40 0x555987d70990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 -- 172.21.15.22:0/1462901869 >> 172.21.15.22:0/1462901869 conn(0x555987d37b30 msgr2=0x555987d6b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 -- 172.21.15.22:0/1462901869 shutdown_connections 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.421+0000 7fb25ef7c740 1 -- 172.21.15.22:0/1462901869 wait complete. 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.422+0000 7fb25ef7c740 1 -- start start 2024-03-24T06:47:09.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.422+0000 7fb25ef7c740 1 Processor -- start 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.422+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d77240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d77cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555987c7cf20 con 0x555987d70f60 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555987c7cbc0 con 0x555987d71c40 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555987a461c0 con 0x555987d7a2d0 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25ef7c740 10 client.0 _ll_register_callbacks cb 0x5559879e8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb259800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d77240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60570/0 (socket says 172.21.15.22:60570) 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 -- 172.21.15.22:0/3193279799 learned_addr learned my addr 172.21.15.22:0/3193279799 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb25a200640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d77cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 -- 172.21.15.22:0/3193279799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 msgr2=0x555987d77cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d77cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 -- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 msgr2=0x555987d77240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d77240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb258e00640 1 -- 172.21.15.22:0/3193279799 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2541ab1e0 con 0x555987d71c40 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.423+0000 7fb259800640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d77240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:09.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb258e00640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb244011d10 tx=0x7fb24400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=250f4d238ae62013 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb237400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244016070 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb244004b20 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555987cd0160 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb244004ce0 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb25ef7c740 10 client.26951 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.424+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555987cd6b20 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555987cd0160 con 0x555987d71c40 2024-03-24T06:47:09.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb24401a3f0 con 0x555987d71c40 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 10 client.26951 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 10 client.26951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 20 client.26951 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987d8a610 0x555987d8ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 4 client.26951 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559879d77e0 con 0x555987d8a610 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.425+0000 7fb259800640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987d8a610 0x555987d8ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.426+0000 7fb259800640 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987d8a610 0x555987d8ca50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4681057469dc0b server_cookie=8282bf08218ea1ba in_seq=0 out_seq=0 2024-03-24T06:47:09.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.426+0000 7fb237400640 10 client.26951 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:09.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb237400640 1 -- 172.21.15.22:0/3193279799 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559879d77e0 con 0x555987d8a610 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb237400640 10 client.26951 handle_command_reply: tid=0 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 client.26951 shutdown 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 msgr2=0x555987d77780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb244011d10 tx=0x7fb24400f790 comp rx=0 tx=0).stop 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 shutdown_connections 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d77cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987d8a610 0x555987d8ca50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d77780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb237400640 0 client.26951 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:09.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3193279799 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d77240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.427+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 >> 172.21.15.22:0/3193279799 conn(0x555987d37b30 msgr2=0x555987d6d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.428+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 shutdown_connections 2024-03-24T06:47:09.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.428+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3193279799 wait complete. 2024-03-24T06:47:09.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.428+0000 7fb25ef7c740 20 client.26951 trim_cache size 0 max 16384 2024-03-24T06:47:09.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 Processor -- start 2024-03-24T06:47:09.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 -- start start 2024-03-24T06:47:09.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d72560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d859e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555987d707d0 con 0x555987d70f60 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555987d78590 con 0x555987d71c40 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555987d70210 con 0x555987d7a2d0 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb259800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.447+0000 7fb259800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60842/0 (socket says 172.21.15.22:60842) 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 -- 172.21.15.22:0/3969954269 learned_addr learned my addr 172.21.15.22:0/3969954269 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb258e00640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d72560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb25a200640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d859e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 -- 172.21.15.22:0/3969954269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 msgr2=0x555987d859e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d859e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 -- 172.21.15.22:0/3969954269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 msgr2=0x555987d72560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d72560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 -- 172.21.15.22:0/3969954269 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555987c729a0 con 0x555987d70f60 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb258e00640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d72560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:09.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb259800640 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb2480b9580 tx=0x7fb2480f3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94c13567de6ac5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.448+0000 7fb242000640 1 -- 172.21.15.22:0/3969954269 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2480d4070 con 0x555987d70f60 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb242000640 1 -- 172.21.15.22:0/3969954269 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2480b8e50 con 0x555987d70f60 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb242000640 1 -- 172.21.15.22:0/3969954269 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248003e20 con 0x555987d70f60 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3969954269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 msgr2=0x555987d761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb2480b9580 tx=0x7fb2480f3f90 comp rx=0 tx=0).stop 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3969954269 shutdown_connections 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d7a2d0 0x555987d859e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d71c40 0x555987d72560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/3969954269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d70f60 0x555987d761a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.449+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3969954269 >> 172.21.15.22:0/3969954269 conn(0x555987d37b30 msgr2=0x555987d6d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.450+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3969954269 shutdown_connections 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.450+0000 7fb25ef7c740 1 -- 172.21.15.22:0/3969954269 wait complete. 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.450+0000 7fb25ef7c740 1 -- start start 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.450+0000 7fb25ef7c740 1 Processor -- start 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 0x555987d73d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d7a2d0 0x555987d747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x555987d82870 con 0x555987d7a2d0 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555987def990 con 0x555987d70f60 2024-03-24T06:47:09.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555987defb20 con 0x555987d71c40 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25ef7c740 10 client.0 _ll_register_callbacks cb 0x7fb244009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb258e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb25a200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d7a2d0 0x555987d747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb258e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48308/0 (socket says 172.21.15.22:48308) 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb258e00640 1 -- 172.21.15.22:0/2562404944 learned_addr learned my addr 172.21.15.22:0/2562404944 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.451+0000 7fb259800640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 0x555987d73d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 -- 172.21.15.22:0/2562404944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 msgr2=0x555987d73d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 0x555987d73d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 -- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d7a2d0 msgr2=0x555987d747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d7a2d0 0x555987d747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 -- 172.21.15.22:0/2562404944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555987cd6b20 con 0x555987d71c40 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb259800640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 0x555987d73d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb258e00640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb244011d10 tx=0x7fb24401a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7fd1496f0b4a6b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb237400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244016040 con 0x555987d71c40 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.452+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb244003b80 con 0x555987d71c40 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555987d72bc0 con 0x555987d71c40 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb244003d40 con 0x555987d71c40 2024-03-24T06:47:09.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 10 client.26980 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555987c99c20 con 0x555987d71c40 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555987d72bc0 con 0x555987d71c40 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb2440314d0 con 0x555987d71c40 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 10 client.26980 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 10 client.26980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 20 client.26980 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987df13d0 0x555987df37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 4 client.26980 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:47:09.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.453+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fb2480bc980 con 0x555987df13d0 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.454+0000 7fb259800640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987df13d0 0x555987df37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.454+0000 7fb259800640 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987df13d0 0x555987df37f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8b2a3d6ad7cd9bc server_cookie=aaa6de64380c3124 in_seq=0 out_seq=0 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.454+0000 7fb237400640 10 client.26980 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.454+0000 7fb237400640 1 -- 172.21.15.22:0/2562404944 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fb2480bc980 con 0x555987df13d0 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.454+0000 7fb237400640 10 client.26980 handle_command_reply: tid=0 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 client.26980 shutdown 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 msgr2=0x555987d74270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb244011d10 tx=0x7fb24401a6a0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 shutdown_connections 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x555987d7a2d0 0x555987d747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555987d71c40 0x555987d74270 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x555987df13d0 0x555987df37f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 --2- 172.21.15.22:0/2562404944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555987d70f60 0x555987d73d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb237400640 0 client.26980 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 >> 172.21.15.22:0/2562404944 conn(0x555987d37b30 msgr2=0x555987d6cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 shutdown_connections 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 1 -- 172.21.15.22:0/2562404944 wait complete. 2024-03-24T06:47:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.455+0000 7fb25ef7c740 20 client.26980 trim_cache size 0 max 16384 2024-03-24T06:47:09.462 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:09.462 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:47:09.462 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:47:09.462 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 msgr2=0x7fb23c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 0x7fb23c0a4770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb24c005fd0 tx=0x7fb24c00b4d0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 msgr2=0x7fb25419e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb248005940 tx=0x7fb24802c710 comp rx=0 tx=0).stop 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 shutdown_connections 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb23c0a22b0 0x7fb23c0a4770 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2541a2ae0 0x7fb2541a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb25410cf10 0x7fb25419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 --2- 172.21.15.22:0/1229619311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb254107ec0 0x7fb25419e270 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 >> 172.21.15.22:0/1229619311 conn(0x7fb254077530 msgr2=0x7fb25410e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 shutdown_connections 2024-03-24T06:47:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.456+0000 7fb25ac00640 1 -- 172.21.15.22:0/1229619311 wait complete. 2024-03-24T06:47:09.478 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:47:09.479 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:47:09.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:09 smithi022 ceph-mon[32003]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T06:47:09.555 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:09 smithi022 ceph-mon[32003]: pgmap v220: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:09 smithi143 ceph-mon[36520]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T06:47:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:09 smithi143 ceph-mon[36520]: pgmap v220: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:09.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 -- 172.21.15.22:0/2721716897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc10d000 msgr2=0x7f3bcc114bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/2721716897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc10d000 0x7f3bcc114bd0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4009c10 tx=0x7f3bc402bee0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 -- 172.21.15.22:0/2721716897 shutdown_connections 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/2721716897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc10d000 0x7f3bcc114bd0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/2721716897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc1088e0 0x7f3bcc10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/2721716897 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 0x7f3bcc108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 -- 172.21.15.22:0/2721716897 >> 172.21.15.22:0/2721716897 conn(0x7f3bcc075c60 msgr2=0x7f3bcc078080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 -- 172.21.15.22:0/2721716897 shutdown_connections 2024-03-24T06:47:09.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.696+0000 7f3bd1a00640 1 -- 172.21.15.22:0/2721716897 wait complete. 2024-03-24T06:47:09.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.697+0000 7f3bd1a00640 1 Processor -- start 2024-03-24T06:47:09.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 -- start start 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 0x7f3bcc19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc1a2c40 0x7f3bcc1a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc1a3630 con 0x7f3bcc107f30 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc1a37a0 con 0x7f3bcc1a2c40 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bd1a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc1a3910 con 0x7f3bcc10d000 2024-03-24T06:47:09.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.698+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48320/0 (socket says 172.21.15.22:48320) 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 -- 172.21.15.22:0/3809572005 learned_addr learned my addr 172.21.15.22:0/3809572005 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bcaa00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 0x7f3bcc19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bcb400640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc1a2c40 0x7f3bcc1a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 -- 172.21.15.22:0/3809572005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc1a2c40 msgr2=0x7f3bcc1a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc1a2c40 0x7f3bcc1a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 -- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 msgr2=0x7f3bcc19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 0x7f3bcc19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 -- 172.21.15.22:0/3809572005 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb4009560 con 0x7f3bcc10d000 2024-03-24T06:47:09.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.699+0000 7f3bc2000640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400e9a0 tx=0x7f3bb400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.700+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400cd80 con 0x7f3bcc10d000 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.700+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb40035f0 con 0x7f3bcc10d000 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.700+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4010690 con 0x7f3bcc10d000 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.700+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc4009840 con 0x7f3bcc10d000 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.700+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1ab1c0 con 0x7f3bcc10d000 2024-03-24T06:47:09.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.701+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc068950 con 0x7f3bcc10d000 2024-03-24T06:47:09.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.703+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3bb4003ad0 con 0x7f3bcc10d000 2024-03-24T06:47:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.704+0000 7f3bc3e00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 0x7f3bb00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.704+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f3bb4014070 con 0x7f3bcc10d000 2024-03-24T06:47:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.705+0000 7f3bcaa00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 0x7f3bb00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:09.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.705+0000 7f3bcaa00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 0x7f3bb00a4770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009bd0 tx=0x7f3bbc005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:09.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.708+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3bb40970b0 con 0x7f3bcc10d000 2024-03-24T06:47:09.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:09 smithi155 ceph-mon[36262]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T06:47:09.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:09 smithi155 ceph-mon[36262]: pgmap v220: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:09.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.950+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bcc110320 con 0x7f3bcc10d000 2024-03-24T06:47:09.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.952+0000 7f3bc3e00640 1 -- 172.21.15.22:0/3809572005 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f3bb40aee60 con 0x7f3bcc10d000 2024-03-24T06:47:09.952 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:47:09.952 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:47:09.953 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:47:09.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.954+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 msgr2=0x7f3bb00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.954+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 0x7f3bb00a4770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009bd0 tx=0x7f3bbc005f90 comp rx=0 tx=0).stop 2024-03-24T06:47:09.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.954+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 msgr2=0x7f3bcc19e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.954+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400e9a0 tx=0x7f3bb400ee60 comp rx=0 tx=0).stop 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 shutdown_connections 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3bb00a22b0 0x7f3bb00a4770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bcc1a2c40 0x7f3bcc1a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bcc10d000 0x7f3bcc19e910 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 --2- 172.21.15.22:0/3809572005 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3bcc107f30 0x7f3bcc19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 >> 172.21.15.22:0/3809572005 conn(0x7f3bcc075c60 msgr2=0x7f3bcc076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 shutdown_connections 2024-03-24T06:47:09.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:09.955+0000 7f3bd1a00640 1 -- 172.21.15.22:0/3809572005 wait complete. 2024-03-24T06:47:09.976 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 -- 172.21.15.22:0/821613385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24103060 msgr2=0x7f0c2410b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 --2- 172.21.15.22:0/821613385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24103060 0x7f0c2410b540 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009900 tx=0x7f0c0c02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 -- 172.21.15.22:0/821613385 shutdown_connections 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 --2- 172.21.15.22:0/821613385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 0x7f0c2410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 --2- 172.21.15.22:0/821613385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24103060 0x7f0c2410b540 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 --2- 172.21.15.22:0/821613385 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24102740 0x7f0c24102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 -- 172.21.15.22:0/821613385 >> 172.21.15.22:0/821613385 conn(0x7f0c240fbf60 msgr2=0x7f0c240fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.226+0000 7f0c29400640 1 -- 172.21.15.22:0/821613385 shutdown_connections 2024-03-24T06:47:10.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.227+0000 7f0c29400640 1 -- 172.21.15.22:0/821613385 wait complete. 2024-03-24T06:47:10.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 Processor -- start 2024-03-24T06:47:10.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 -- start start 2024-03-24T06:47:10.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 0x7f0c241008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 0x7f0c24100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0c241a6040 con 0x7f0c24103060 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0c241a61b0 con 0x7f0c2410bb60 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.228+0000 7f0c29400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0c241a6370 con 0x7f0c24102740 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48340/0 (socket says 172.21.15.22:48340) 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 -- 172.21.15.22:0/2011383996 learned_addr learned my addr 172.21.15.22:0/2011383996 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 0x7f0c24100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 0x7f0c241008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 -- 172.21.15.22:0/2011383996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 msgr2=0x7f0c24100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 0x7f0c24100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 -- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 msgr2=0x7f0c241008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 0x7f0c241008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c23e00640 1 -- 172.21.15.22:0/2011383996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c009560 con 0x7f0c24102740 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.229+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 0x7f0c241008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:10.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c23e00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c1400e9f0 tx=0x7f0c1400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c21600640 1 -- 172.21.15.22:0/2011383996 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14009700 con 0x7f0c24102740 2024-03-24T06:47:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c21600640 1 -- 172.21.15.22:0/2011383996 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c1400cd80 con 0x7f0c24102740 2024-03-24T06:47:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c241a6650 con 0x7f0c24102740 2024-03-24T06:47:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c21600640 1 -- 172.21.15.22:0/2011383996 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14018920 con 0x7f0c24102740 2024-03-24T06:47:10.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.230+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c241a6b80 con 0x7f0c24102740 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.233+0000 7f0c2d78b740 1 Processor -- start 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.233+0000 7f0c21600640 1 -- 172.21.15.22:0/2011383996 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0c14018a80 con 0x7f0c24102740 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.233+0000 7f0c2d78b740 1 -- start start 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49726410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49728d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558e4968b6e0 con 0x558e49726950 2024-03-24T06:47:10.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558e49688c10 con 0x558e4971dd70 2024-03-24T06:47:10.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558e496843f0 con 0x558e4971d450 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c23e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49726410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48352/0 (socket says 172.21.15.22:48352) 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 -- 172.21.15.22:0/2991034631 learned_addr learned my addr 172.21.15.22:0/2991034631 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49728d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 -- 172.21.15.22:0/2991034631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 msgr2=0x558e49726410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49726410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 -- 172.21.15.22:0/2991034631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 msgr2=0x558e49728d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49728d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c28a00640 1 -- 172.21.15.22:0/2991034631 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e49645e40 con 0x558e4971d450 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.234+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49728d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009f50 tx=0x7f0c1c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5844caf3b7bee7ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c21600640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 0x7f0c080a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c19200640 1 -- 172.21.15.22:0/2991034631 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c004d20 con 0x558e4971d450 2024-03-24T06:47:10.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c19200640 1 -- 172.21.15.22:0/2991034631 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c1c004e80 con 0x558e4971d450 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c19200640 1 -- 172.21.15.22:0/2991034631 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c005280 con 0x558e4971d450 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c21600640 1 -- 172.21.15.22:0/2011383996 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f0c14014070 con 0x7f0c24102740 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.235+0000 7f0c23e00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 0x7f0c080a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2991034631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 msgr2=0x558e4971d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009f50 tx=0x7f0c1c0118d0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c23e00640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 0x7f0c080a4770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x558e4971cf20 tx=0x7f0c140d4530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2991034631 shutdown_connections 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49728d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49726410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2991034631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e4971d830 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2991034631 >> 172.21.15.22:0/2991034631 conn(0x558e49717b80 msgr2=0x558e49717f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2991034631 shutdown_connections 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.236+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2991034631 wait complete. 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 -- start start 2024-03-24T06:47:10.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 Processor -- start 2024-03-24T06:47:10.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49723a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49724490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558e49629060 con 0x558e49726950 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558e49628d00 con 0x558e4971dd70 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.237+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558e493f21c0 con 0x558e4971d450 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c2d78b740 10 client.0 _ll_register_callbacks cb 0x558e49394be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60654/0 (socket says 172.21.15.22:60654) 2024-03-24T06:47:10.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 -- 172.21.15.22:0/2784131988 learned_addr learned my addr 172.21.15.22:0/2784131988 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49723a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 -- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 msgr2=0x558e49723a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49723a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 -- 172.21.15.22:0/2784131988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 msgr2=0x558e49724490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49724490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49724490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 -- 172.21.15.22:0/2784131988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c009c00 con 0x558e4971dd70 2024-03-24T06:47:10.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49724490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009900 tx=0x7f0c0c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f80add2e42568b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.238+0000 7f0c11600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c02c880 con 0x558e4971dd70 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f0c0c02c9e0 con 0x558e4971dd70 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558e493c0720 con 0x558e4971dd70 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0c0c02cba0 con 0x558e4971dd70 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c2d78b740 10 client.26975 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558e49682a70 con 0x558e4971dd70 2024-03-24T06:47:10.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.239+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558e493c0720 con 0x558e4971dd70 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0c0c03c050 con 0x558e4971dd70 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 10 client.26975 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 10 client.26975 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 20 client.26975 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e49736de0 0x558e49739200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 4 client.26975 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e4940f890 con 0x558e49736de0 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.240+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e49736de0 0x558e49739200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.241+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e49736de0 0x558e49739200 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2339944a258762d2 server_cookie=6299ca72b30ec9fb in_seq=0 out_seq=0 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.241+0000 7f0c11600640 10 client.26975 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.241+0000 7f0c11600640 1 -- 172.21.15.22:0/2784131988 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e4940f890 con 0x558e49736de0 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.241+0000 7f0c11600640 10 client.26975 handle_command_reply: tid=0 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.241+0000 7f0c2d78b740 1 client.26975 shutdown 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 msgr2=0x558e49723f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009900 tx=0x7f0c0c004270 comp rx=0 tx=0).stop 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 shutdown_connections 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e49736de0 0x558e49739200 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e49726950 0x558e49724490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e49723f50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2784131988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49723a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c11600640 0 client.26975 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 >> 172.21.15.22:0/2784131988 conn(0x558e49717b80 msgr2=0x558e497198d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 shutdown_connections 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2784131988 wait complete. 2024-03-24T06:47:10.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.242+0000 7f0c2d78b740 20 client.26975 trim_cache size 0 max 16384 2024-03-24T06:47:10.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 Processor -- start 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 -- start start 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 0x558e49726140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e49726950 0x558e4971e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558e49732720 con 0x558e4971d450 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558e4971e690 con 0x558e4971dd70 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558e4971d0b0 con 0x558e49726950 2024-03-24T06:47:10.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c1be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.257+0000 7f0c28a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 0x558e49726140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60668/0 (socket says 172.21.15.22:60668) 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 -- 172.21.15.22:0/3348834145 learned_addr learned my addr 172.21.15.22:0/3348834145 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c28a00640 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e49726950 0x558e4971e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 -- 172.21.15.22:0/3348834145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e49726950 msgr2=0x558e4971e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e49726950 0x558e4971e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 -- 172.21.15.22:0/3348834145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 msgr2=0x558e49726140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 0x558e49726140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 -- 172.21.15.22:0/3348834145 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e4961eb00 con 0x558e4971dd70 2024-03-24T06:47:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c28a00640 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 0x558e49726140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c1be00640 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009850 tx=0x7f0c0c02c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e62c177518e810d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c13e00640 1 -- 172.21.15.22:0/3348834145 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0042e0 con 0x558e4971dd70 2024-03-24T06:47:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.258+0000 7f0c13e00640 1 -- 172.21.15.22:0/3348834145 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c0c005d50 con 0x558e4971dd70 2024-03-24T06:47:10.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c13e00640 1 -- 172.21.15.22:0/3348834145 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c005550 con 0x558e4971dd70 2024-03-24T06:47:10.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 -- 172.21.15.22:0/3348834145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 msgr2=0x558e497321e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009850 tx=0x7f0c0c02c730 comp rx=0 tx=0).stop 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 -- 172.21.15.22:0/3348834145 shutdown_connections 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e49726950 0x558e4971e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e4971dd70 0x558e497321e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/3348834145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971d450 0x558e49726140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 -- 172.21.15.22:0/3348834145 >> 172.21.15.22:0/3348834145 conn(0x558e49717b80 msgr2=0x558e49716430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 -- 172.21.15.22:0/3348834145 shutdown_connections 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.259+0000 7f0c2d78b740 1 -- 172.21.15.22:0/3348834145 wait complete. 2024-03-24T06:47:10.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 -- start start 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 Processor -- start 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49720880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 0x558e49720dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e49726950 0x558e4979c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558e497215a0 con 0x558e4971dd70 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558e49721710 con 0x558e49726950 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558e4979c640 con 0x558e4971d450 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.260+0000 7f0c2d78b740 10 client.0 _ll_register_callbacks cb 0x7f0c240698d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c28a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49720880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 0x558e49720dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c28a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49720880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48374/0 (socket says 172.21.15.22:48374) 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c28a00640 1 -- 172.21.15.22:0/2288382304 learned_addr learned my addr 172.21.15.22:0/2288382304 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e49726950 0x558e4979c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 -- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 msgr2=0x558e49720880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49720880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 -- 172.21.15.22:0/2288382304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e49726950 msgr2=0x558e4979c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e49726950 0x558e4979c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.261+0000 7f0c1be00640 1 -- 172.21.15.22:0/2288382304 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e49682a70 con 0x558e4971dd70 2024-03-24T06:47:10.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c1be00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 0x558e49720dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x558e49724db0 tx=0x7f0c0c02cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af560e0fb814c005 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c11600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c057070 con 0x558e4971dd70 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558e493837e0 con 0x558e4971dd70 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f0c0c0054f0 con 0x558e4971dd70 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0c0c003af0 con 0x558e4971dd70 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c2d78b740 10 client.16614 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.262+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558e496446a0 con 0x558e4971dd70 2024-03-24T06:47:10.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558e493837e0 con 0x558e4971dd70 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0c0c05a030 con 0x558e4971dd70 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 10 client.16614 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 10 client.16614 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 20 client.16614 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e4979d8f0 0x558e4979fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 4 client.16614 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558e4932f700 con 0x558e4979d8f0 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.263+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e4979d8f0 0x558e4979fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c11600640 10 client.16614 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c28a00640 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e4979d8f0 0x558e4979fce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d76a308e1b8e207f server_cookie=c870e0a3b397b64 in_seq=0 out_seq=0 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c11600640 1 -- 172.21.15.22:0/2288382304 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558e4932f700 con 0x558e4979d8f0 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c11600640 10 client.16614 handle_command_reply: tid=0 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 client.16614 shutdown 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 msgr2=0x558e49720dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 0x558e49720dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x558e49724db0 tx=0x7f0c0c02cbc0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 shutdown_connections 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x558e4979d8f0 0x558e4979fce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558e49726950 0x558e4979c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558e4971dd70 0x558e49720dc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 --2- 172.21.15.22:0/2288382304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558e4971d450 0x558e49720880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c11600640 0 client.16614 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 >> 172.21.15.22:0/2288382304 conn(0x558e49717b80 msgr2=0x558e49716cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 shutdown_connections 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 1 -- 172.21.15.22:0/2288382304 wait complete. 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.264+0000 7f0c2d78b740 20 client.16614 trim_cache size 0 max 16384 2024-03-24T06:47:10.271 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:10.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 msgr2=0x7f0c080a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 0x7f0c080a4770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x558e4971cf20 tx=0x7f0c140d4530 comp rx=0 tx=0).stop 2024-03-24T06:47:10.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 msgr2=0x7f0c24100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0c1400e9f0 tx=0x7f0c1400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 shutdown_connections 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c080a22b0 0x7f0c080a4770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2410bb60 0x7f0c24100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c24103060 0x7f0c241008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 --2- 172.21.15.22:0/2011383996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c24102740 0x7f0c24100390 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 >> 172.21.15.22:0/2011383996 conn(0x7f0c240fbf60 msgr2=0x7f0c240fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 shutdown_connections 2024-03-24T06:47:10.273 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.265+0000 7f0c29400640 1 -- 172.21.15.22:0/2011383996 wait complete. 2024-03-24T06:47:10.286 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:47:10.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.538+0000 7f71c6c00640 1 -- 172.21.15.22:0/1325297170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c01097a0 msgr2=0x7f71c0109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.538+0000 7f71c6c00640 1 --2- 172.21.15.22:0/1325297170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c01097a0 0x7f71c0109b80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009930 tx=0x7f71b402bc30 comp rx=0 tx=0).stop 2024-03-24T06:47:10.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 -- 172.21.15.22:0/1325297170 shutdown_connections 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 --2- 172.21.15.22:0/1325297170 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c0104150 0x7f71c0110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 --2- 172.21.15.22:0/1325297170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0103790 0x7f71c0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 --2- 172.21.15.22:0/1325297170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c01097a0 0x7f71c0109b80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 -- 172.21.15.22:0/1325297170 >> 172.21.15.22:0/1325297170 conn(0x7f71c00ff220 msgr2=0x7f71c0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 -- 172.21.15.22:0/1325297170 shutdown_connections 2024-03-24T06:47:10.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.539+0000 7f71c6c00640 1 -- 172.21.15.22:0/1325297170 wait complete. 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.540+0000 7f71c6c00640 1 Processor -- start 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.540+0000 7f71c6c00640 1 -- start start 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 0x7f71c019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 0x7f71c019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f71c019ed20 con 0x7f71c01097a0 2024-03-24T06:47:10.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f71c019ee90 con 0x7f71c0104150 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f71c019f050 con 0x7f71c0103790 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 0x7f71c019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48392/0 (socket says 172.21.15.22:48392) 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c5800640 1 -- 172.21.15.22:0/2880994979 learned_addr learned my addr 172.21.15.22:0/2880994979 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.541+0000 7f71c6200640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 0x7f71c019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 -- 172.21.15.22:0/2880994979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 msgr2=0x7f71c019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 0x7f71c019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 -- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 msgr2=0x7f71c019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 0x7f71c019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 -- 172.21.15.22:0/2880994979 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b4009560 con 0x7f71c0103790 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c6200640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 0x7f71c019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c4e00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 0x7f71c019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71c5800640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f71b4000c00 tx=0x7f71b4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71be000640 1 -- 172.21.15.22:0/2880994979 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4004440 con 0x7f71c0103790 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71be000640 1 -- 172.21.15.22:0/2880994979 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b40052d0 con 0x7f71c0103790 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.542+0000 7f71be000640 1 -- 172.21.15.22:0/2880994979 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4003da0 con 0x7f71c0103790 2024-03-24T06:47:10.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.543+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c019f2d0 con 0x7f71c0103790 2024-03-24T06:47:10.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.543+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01a6bb0 con 0x7f71c0103790 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 Processor -- start 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 -- start start 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8e1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558a1d84f6e0 con 0x558a1d8e1450 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558a1d84cc10 con 0x558a1d8ea950 2024-03-24T06:47:10.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.546+0000 7f71cae3c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8483f0 con 0x558a1d8e1d70 2024-03-24T06:47:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c6200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8e1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60696/0 (socket says 172.21.15.22:60696) 2024-03-24T06:47:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c6200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8e1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60950/0 (socket says 172.21.15.22:60950) 2024-03-24T06:47:10.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 -- 172.21.15.22:0/755869929 learned_addr learned my addr 172.21.15.22:0/755869929 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 -- 172.21.15.22:0/755869929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 msgr2=0x558a1d8ea410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8ea410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 -- 172.21.15.22:0/755869929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 msgr2=0x558a1d8e1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8e1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.547+0000 7f71c4e00640 1 -- 172.21.15.22:0/755869929 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1d809e40 con 0x558a1d8ea950 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71be000640 1 -- 172.21.15.22:0/2880994979 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f71b4005440 con 0x7f71c0103790 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71c4e00640 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f71ac010480 tx=0x7f71ac010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26548e545946cae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71b2a00640 1 -- 172.21.15.22:0/755869929 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac011760 con 0x558a1d8ea950 2024-03-24T06:47:10.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71b2a00640 1 -- 172.21.15.22:0/755869929 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac015070 con 0x558a1d8ea950 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71b2a00640 1 -- 172.21.15.22:0/755869929 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac01cde0 con 0x558a1d8ea950 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71cae3c740 1 -- 172.21.15.22:0/755869929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 msgr2=0x558a1d8ecd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.548+0000 7f71cae3c740 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f71ac010480 tx=0x7f71ac010940 comp rx=0 tx=0).stop 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 -- 172.21.15.22:0/755869929 shutdown_connections 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8ecd40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 --2- 172.21.15.22:0/755869929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8e1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 -- 172.21.15.22:0/755869929 >> 172.21.15.22:0/755869929 conn(0x558a1d8dbb80 msgr2=0x558a1d8dbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 -- 172.21.15.22:0/755869929 shutdown_connections 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 -- 172.21.15.22:0/755869929 wait complete. 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71be000640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 0x7f71a80a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.549+0000 7f71cae3c740 1 -- start start 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.550+0000 7f71be000640 1 -- 172.21.15.22:0/2880994979 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f71b40f0b80 con 0x7f71c0103790 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.550+0000 7f71c6200640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 0x7f71a80a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.550+0000 7f71cae3c740 1 Processor -- start 2024-03-24T06:47:10.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.550+0000 7f71c6200640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 0x7f71a80a46a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005fd0 tx=0x7f71b8009820 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1d70 0x558a1d8e7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8ea950 0x558a1d8e8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558a1d7ed060 con 0x558a1d8e1d70 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558a1d7ecd00 con 0x558a1d8e1450 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558a1d5b61c0 con 0x558a1d8ea950 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71cae3c740 10 client.0 _ll_register_callbacks cb 0x558a1d558be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1d70 0x558a1d8e7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60708/0 (socket says 172.21.15.22:60708) 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 -- 172.21.15.22:0/991182416 learned_addr learned my addr 172.21.15.22:0/991182416 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c6200640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8ea950 0x558a1d8e8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 -- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8ea950 msgr2=0x558a1d8e8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8ea950 0x558a1d8e8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 -- 172.21.15.22:0/991182416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1d70 msgr2=0x558a1d8e7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1d70 0x558a1d8e7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.551+0000 7f71c5800640 1 -- 172.21.15.22:0/991182416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac00bcd0 con 0x558a1d8e1450 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71c5800640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009900 tx=0x7f71b40bcbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff63e4b5f0ed16b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b40c13c0 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f71b40c5a10 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558a1d5477e0 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f71b40c1170 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71cae3c740 10 client.26993 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558a1d846a70 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558a1d5477e0 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.552+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f71b403c070 con 0x558a1d8e1450 2024-03-24T06:47:10.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 10 client.26993 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 10 client.26993 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 20 client.26993 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d8fa910 0x558a1d8fcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 4 client.26993 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a1d584720 con 0x558a1d8fa910 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71c4e00640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d8fa910 0x558a1d8fcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71c4e00640 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d8fa910 0x558a1d8fcd30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5da6210214eb38e2 server_cookie=48a90fd883fbaf36 in_seq=0 out_seq=0 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.553+0000 7f71a7400640 10 client.26993 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:10.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.554+0000 7f71a7400640 1 -- 172.21.15.22:0/991182416 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a1d584720 con 0x558a1d8fa910 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.554+0000 7f71a7400640 10 client.26993 handle_command_reply: tid=0 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.554+0000 7f71cae3c740 1 client.26993 shutdown 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 msgr2=0x558a1d8e79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009900 tx=0x7f71b40bcbd0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 shutdown_connections 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d8fa910 0x558a1d8fcd30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8ea950 0x558a1d8e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1d70 0x558a1d8e7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71a7400640 0 client.26993 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:10.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 --2- 172.21.15.22:0/991182416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1450 0x558a1d8e79c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 >> 172.21.15.22:0/991182416 conn(0x558a1d8dbb80 msgr2=0x558a1d8dd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 shutdown_connections 2024-03-24T06:47:10.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 1 -- 172.21.15.22:0/991182416 wait complete. 2024-03-24T06:47:10.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.555+0000 7f71cae3c740 20 client.26993 trim_cache size 0 max 16384 2024-03-24T06:47:10.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 Processor -- start 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 -- start start 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 0x558a1d8e8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8ea950 0x558a1d902600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8d74b0 con 0x558a1d8ea950 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8f7910 con 0x558a1d8e1d70 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71cae3c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8dcbb0 con 0x558a1d8e1450 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71c5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 0x558a1d8e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71c5800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48410/0 (socket says 172.21.15.22:48410) 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71c5800640 1 -- 172.21.15.22:0/3283321017 learned_addr learned my addr 172.21.15.22:0/3283321017 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.570+0000 7f71c6200640 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8ea950 0x558a1d902600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 -- 172.21.15.22:0/3283321017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 msgr2=0x558a1d8e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 0x558a1d8e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 -- 172.21.15.22:0/3283321017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8ea950 msgr2=0x558a1d902600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8ea950 0x558a1d902600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 -- 172.21.15.22:0/3283321017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1d7e2b00 con 0x558a1d8e1450 2024-03-24T06:47:10.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c4e00640 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 0x558a1d8e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71c5800640 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f71b40ee4c0 tx=0x7f71b40bd3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e4123dc342faaee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71b2000640 1 -- 172.21.15.22:0/3283321017 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b40bd970 con 0x558a1d8e1450 2024-03-24T06:47:10.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71b2000640 1 -- 172.21.15.22:0/3283321017 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b40c5a10 con 0x558a1d8e1450 2024-03-24T06:47:10.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.571+0000 7f71b2000640 1 -- 172.21.15.22:0/3283321017 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b40c5580 con 0x558a1d8e1450 2024-03-24T06:47:10.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- 172.21.15.22:0/3283321017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 msgr2=0x558a1d8f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f71b40ee4c0 tx=0x7f71b40bd3f0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- 172.21.15.22:0/3283321017 shutdown_connections 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8ea950 0x558a1d902600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8e1d70 0x558a1d8e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3283321017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1450 0x558a1d8f6240 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- 172.21.15.22:0/3283321017 >> 172.21.15.22:0/3283321017 conn(0x558a1d8dbb80 msgr2=0x558a1d8dacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- 172.21.15.22:0/3283321017 shutdown_connections 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- 172.21.15.22:0/3283321017 wait complete. 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.572+0000 7f71cae3c740 1 -- start start 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 Processor -- start 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8f2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8edde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8ee4b0 con 0x558a1d8e1450 2024-03-24T06:47:10.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8ee620 con 0x558a1d8ea950 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558a1d8ee7e0 con 0x558a1d8e1d70 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71cae3c740 10 client.0 _ll_register_callbacks cb 0x7f71b40bd040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71c5800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71c6200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8edde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71c5800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8f2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60966/0 (socket says 172.21.15.22:60966) 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71c4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.573+0000 7f71c5800640 1 -- 172.21.15.22:0/3550205367 learned_addr learned my addr 172.21.15.22:0/3550205367 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 -- 172.21.15.22:0/3550205367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 msgr2=0x558a1d8edde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8edde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 -- 172.21.15.22:0/3550205367 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 msgr2=0x558a1d8f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 -- 172.21.15.22:0/3550205367 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a1d809e40 con 0x558a1d8e1d70 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71c4e00640 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8f2cc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f71ac010450 tx=0x7f71ac0058f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=550b677f6b438730 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.574+0000 7f71a7400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac01cd70 con 0x558a1d8e1d70 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f71ac017070 con 0x558a1d8e1d70 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac015070 con 0x558a1d8e1d70 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f71ac005040 con 0x558a1d8e1d70 2024-03-24T06:47:10.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71cae3c740 10 client.27019 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x7f71b40da070 con 0x558a1d8e1d70 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac015070 con 0x558a1d8e1d70 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.575+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f71ac029630 con 0x558a1d8e1d70 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 10 client.27019 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 10 client.27019 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 20 client.27019 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d960fc0 0x558a1d9633e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 4 client.27019 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a1d5d3890 con 0x558a1d960fc0 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.576+0000 7f71c5800640 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d960fc0 0x558a1d9633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71a7400640 10 client.27019 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71c5800640 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d960fc0 0x558a1d9633e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ec5f680125e1866 server_cookie=ea0dcde6b85e3a58 in_seq=0 out_seq=0 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71a7400640 1 -- 172.21.15.22:0/3550205367 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a1d5d3890 con 0x558a1d960fc0 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71a7400640 10 client.27019 handle_command_reply: tid=0 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 client.27019 shutdown 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 msgr2=0x558a1d8f2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8f2cc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f71ac010450 tx=0x7f71ac0058f0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 shutdown_connections 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x558a1d960fc0 0x558a1d9633e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558a1d8ea950 0x558a1d8edde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558a1d8e1d70 0x558a1d8f2cc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71a7400640 0 client.27019 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 --2- 172.21.15.22:0/3550205367 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558a1d8e1450 0x558a1d8f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 >> 172.21.15.22:0/3550205367 conn(0x558a1d8dbb80 msgr2=0x558a1d8dcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 shutdown_connections 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 1 -- 172.21.15.22:0/3550205367 wait complete. 2024-03-24T06:47:10.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.577+0000 7f71cae3c740 20 client.27019 trim_cache size 0 max 16384 2024-03-24T06:47:10.585 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 msgr2=0x7f71a80a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 0x7f71a80a46a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005fd0 tx=0x7f71b8009820 comp rx=0 tx=0).stop 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 msgr2=0x7f71c0199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f71b4000c00 tx=0x7f71b4004040 comp rx=0 tx=0).stop 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 shutdown_connections 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f71a80a21e0 0x7f71a80a46a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71c01097a0 0x7f71c019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71c0104150 0x7f71c019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 --2- 172.21.15.22:0/2880994979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71c0103790 0x7f71c0199e60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 >> 172.21.15.22:0/2880994979 conn(0x7f71c00ff220 msgr2=0x7f71c00ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 shutdown_connections 2024-03-24T06:47:10.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.578+0000 7f71c6c00640 1 -- 172.21.15.22:0/2880994979 wait complete. 2024-03-24T06:47:10.594 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:10 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3809572005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:10.597 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:47:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:10 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3809572005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:10.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:10 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3809572005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:10.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 -- 172.21.15.22:0/2047706995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 msgr2=0x7fb88810e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 --2- 172.21.15.22:0/2047706995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb88810e420 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb880009cd0 tx=0x7fb88002c0f0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 -- 172.21.15.22:0/2047706995 shutdown_connections 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 --2- 172.21.15.22:0/2047706995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb88810e420 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 --2- 172.21.15.22:0/2047706995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8881015a0 0x7fb888101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 --2- 172.21.15.22:0/2047706995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb888107660 0x7fb888107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 -- 172.21.15.22:0/2047706995 >> 172.21.15.22:0/2047706995 conn(0x7fb8880fd0b0 msgr2=0x7fb8880ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 -- 172.21.15.22:0/2047706995 shutdown_connections 2024-03-24T06:47:10.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.796+0000 7fb88e600640 1 -- 172.21.15.22:0/2047706995 wait complete. 2024-03-24T06:47:10.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.798+0000 7fb88e600640 1 Processor -- start 2024-03-24T06:47:10.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.798+0000 7fb88e600640 1 -- start start 2024-03-24T06:47:10.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8881015a0 0x7fb888073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 0x7fb888074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb8881a5ed0 con 0x7fb8881015a0 2024-03-24T06:47:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb8881a6040 con 0x7fb888107660 2024-03-24T06:47:10.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb8881a6200 con 0x7fb888101f60 2024-03-24T06:47:10.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8881015a0 0x7fb888073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48434/0 (socket says 172.21.15.22:48434) 2024-03-24T06:47:10.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb88dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 0x7fb888074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 -- 172.21.15.22:0/2785250410 learned_addr learned my addr 172.21.15.22:0/2785250410 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 -- 172.21.15.22:0/2785250410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 msgr2=0x7fb888074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 0x7fb888074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 -- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8881015a0 msgr2=0x7fb888073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.799+0000 7fb887e00640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8881015a0 0x7fb888073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb887e00640 1 -- 172.21.15.22:0/2785250410 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb880009900 con 0x7fb888101f60 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb88dc00640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 0x7fb888074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb887e00640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb87400e9f0 tx=0x7fb87400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb885600640 1 -- 172.21.15.22:0/2785250410 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874009700 con 0x7fb888101f60 2024-03-24T06:47:10.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb885600640 1 -- 172.21.15.22:0/2785250410 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87400cd80 con 0x7fb888101f60 2024-03-24T06:47:10.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8881a64e0 con 0x7fb888101f60 2024-03-24T06:47:10.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb885600640 1 -- 172.21.15.22:0/2785250410 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874018920 con 0x7fb888101f60 2024-03-24T06:47:10.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.800+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8881a6a40 con 0x7fb888101f60 2024-03-24T06:47:10.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.804+0000 7fb885600640 1 -- 172.21.15.22:0/2785250410 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb874018a80 con 0x7fb888101f60 2024-03-24T06:47:10.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.804+0000 7fb892964740 1 Processor -- start 2024-03-24T06:47:10.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 -- start start 2024-03-24T06:47:10.807 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02bd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628f02226e0 con 0x5628f02b4d70 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628f021fc10 con 0x5628f02b4450 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb892964740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628f021b3f0 con 0x5628f02bd950 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb88dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb887e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb88d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02bd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.808 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb88dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60762/0 (socket says 172.21.15.22:60762) 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.805+0000 7fb88dc00640 1 -- 172.21.15.22:0/3426158116 learned_addr learned my addr 172.21.15.22:0/3426158116 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 -- 172.21.15.22:0/3426158116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 msgr2=0x5628f02bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02bfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 -- 172.21.15.22:0/3426158116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 msgr2=0x5628f02bd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02bd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 -- 172.21.15.22:0/3426158116 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628f01dce40 con 0x5628f02b4450 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb887e00640 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb885600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 0x7fb8700a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb880009ca0 tx=0x7fb880034880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a059967e31ff300 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb885600640 1 -- 172.21.15.22:0/2785250410 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb874014070 con 0x7fb888101f60 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.806+0000 7fb88d200640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 0x7fb8700a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb87a000640 1 -- 172.21.15.22:0/3426158116 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880032340 con 0x5628f02b4450 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb87a000640 1 -- 172.21.15.22:0/3426158116 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8800324a0 con 0x5628f02b4450 2024-03-24T06:47:10.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb87a000640 1 -- 172.21.15.22:0/3426158116 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880003d90 con 0x5628f02b4450 2024-03-24T06:47:10.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb892964740 1 -- 172.21.15.22:0/3426158116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 msgr2=0x5628f02b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb88d200640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 0x7fb8700a4770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5628f02b3f20 tx=0x7fb87c00b410 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.814 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.807+0000 7fb892964740 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb880009ca0 tx=0x7fb880034880 comp rx=0 tx=0).stop 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 -- 172.21.15.22:0/3426158116 shutdown_connections 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02bd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 --2- 172.21.15.22:0/3426158116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02b4830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 -- 172.21.15.22:0/3426158116 >> 172.21.15.22:0/3426158116 conn(0x5628f02aeb80 msgr2=0x5628f02aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 -- 172.21.15.22:0/3426158116 shutdown_connections 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 -- 172.21.15.22:0/3426158116 wait complete. 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 -- start start 2024-03-24T06:47:10.815 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.808+0000 7fb892964740 1 Processor -- start 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 0x5628f02baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 0x5628f02bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628f01c0060 con 0x5628f02bd950 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628f01bfd00 con 0x5628f02b4d70 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628eff891c0 con 0x5628f02b4450 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb892964740 10 client.0 _ll_register_callbacks cb 0x5628eff2bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 0x5628f02baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 0x5628f02bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb887e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 0x5628f02baa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48450/0 (socket says 172.21.15.22:48450) 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88d200640 1 -- 172.21.15.22:0/3479132642 learned_addr learned my addr 172.21.15.22:0/3479132642 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb887e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60764/0 (socket says 172.21.15.22:60764) 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88dc00640 1 -- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 msgr2=0x5628f02baa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 0x5628f02baa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88dc00640 1 -- 172.21.15.22:0/3479132642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 msgr2=0x5628f02baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.809+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb88dc00640 1 -- 172.21.15.22:0/3479132642 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88810ed20 con 0x5628f02bd950 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb887e00640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 0x5628f02bb490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb880037040 tx=0x7fb8800328e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd362a79a96828aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb86ea00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880032340 con 0x5628f02bd950 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5628eff1a7e0 con 0x5628f02bd950 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb880050070 con 0x5628f02bd950 2024-03-24T06:47:10.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.810+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb880055480 con 0x5628f02bd950 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 10 client.16644 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5628f0219a70 con 0x5628f02bd950 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5628eff1a7e0 con 0x5628f02bd950 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb88004a680 con 0x5628f02bd950 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 10 client.16644 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 10 client.16644 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 20 client.16644 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f02cda90 0x5628f02cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 4 client.16644 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.811+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628eff57720 con 0x5628f02cda90 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.812+0000 7fb88d200640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f02cda90 0x5628f02cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.812+0000 7fb86ea00640 10 client.16644 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.812+0000 7fb88d200640 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f02cda90 0x5628f02cfe80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f12a7822f1c46ca1 server_cookie=e772ff7c3ffe0d63 in_seq=0 out_seq=0 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb86ea00640 1 -- 172.21.15.22:0/3479132642 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628eff57720 con 0x5628f02cda90 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb86ea00640 10 client.16644 handle_command_reply: tid=0 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 client.16644 shutdown 2024-03-24T06:47:10.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 msgr2=0x5628f02bb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 0x5628f02bb490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb880037040 tx=0x7fb8800328e0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 shutdown_connections 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f02cda90 0x5628f02cfe80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02bd950 0x5628f02bb490 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4d70 0x5628f02baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 --2- 172.21.15.22:0/3479132642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4450 0x5628f02baa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb86ea00640 0 client.16644 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:10.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.813+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 >> 172.21.15.22:0/3479132642 conn(0x5628f02aeb80 msgr2=0x5628f02b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.814+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 shutdown_connections 2024-03-24T06:47:10.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.814+0000 7fb892964740 1 -- 172.21.15.22:0/3479132642 wait complete. 2024-03-24T06:47:10.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.814+0000 7fb892964740 20 client.16644 trim_cache size 0 max 16384 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.828+0000 7fb892964740 1 Processor -- start 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.828+0000 7fb892964740 1 -- start start 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02bd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02d5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.830 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628f02bb970 con 0x5628f02b4d70 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628f02c96c0 con 0x5628f02b4450 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb892964740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628f02cd890 con 0x5628f02bd950 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb88dc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02d5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32798/0 (socket says 172.21.15.22:32798) 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 -- 172.21.15.22:0/1874376591 learned_addr learned my addr 172.21.15.22:0/1874376591 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb88d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02bd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 -- 172.21.15.22:0/1874376591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 msgr2=0x5628f02d5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.831 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02d5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 -- 172.21.15.22:0/1874376591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 msgr2=0x5628f02bd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.829+0000 7fb887e00640 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02bd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb887e00640 1 -- 172.21.15.22:0/1874376591 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628f01b5b00 con 0x5628f02b4d70 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb88dc00640 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02d5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb88d200640 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02bd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.832 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb887e00640 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb874097810 tx=0x7fb874009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae631a989f4479a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb879600640 1 -- 172.21.15.22:0/1874376591 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87409a980 con 0x5628f02b4d70 2024-03-24T06:47:10.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb879600640 1 -- 172.21.15.22:0/1874376591 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87409a7a0 con 0x5628f02b4d70 2024-03-24T06:47:10.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.830+0000 7fb879600640 1 -- 172.21.15.22:0/1874376591 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8740ce380 con 0x5628f02b4d70 2024-03-24T06:47:10.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- 172.21.15.22:0/1874376591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 msgr2=0x5628f02c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb874097810 tx=0x7fb874009310 comp rx=0 tx=0).stop 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- 172.21.15.22:0/1874376591 shutdown_connections 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02bd950 0x5628f02d5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4d70 0x5628f02c9180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 --2- 172.21.15.22:0/1874376591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02b4450 0x5628f02bd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- 172.21.15.22:0/1874376591 >> 172.21.15.22:0/1874376591 conn(0x5628f02aeb80 msgr2=0x5628f02bb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- 172.21.15.22:0/1874376591 shutdown_connections 2024-03-24T06:47:10.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- 172.21.15.22:0/1874376591 wait complete. 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 -- start start 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.831+0000 7fb892964740 1 Processor -- start 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 0x5628f02b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 0x5628f02b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628f0332eb0 con 0x5628f02b4450 2024-03-24T06:47:10.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628f0333020 con 0x5628f02bd950 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628f03331e0 con 0x5628f02b4d70 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb892964740 10 client.0 _ll_register_callbacks cb 0x7fb87c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb88d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb887e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 0x5628f02b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb88d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32804/0 (socket says 172.21.15.22:32804) 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb88d200640 1 -- 172.21.15.22:0/3582315760 learned_addr learned my addr 172.21.15.22:0/3582315760 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.832+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 0x5628f02b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 -- 172.21.15.22:0/3582315760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 msgr2=0x5628f02b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 0x5628f02b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 -- 172.21.15.22:0/3582315760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 msgr2=0x5628f02b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 -- 172.21.15.22:0/3582315760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c01e680 con 0x5628f02b4d70 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb88d200640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb88dc00640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 0x5628f02b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb887e00640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 0x5628f02b7e90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb87409f0a0 tx=0x7fb874010db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59a50890dd180f77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:10.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb86ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87409ae40 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7fb8740a3530 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8800324a0 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb8740b90d0 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb892964740 10 client.27037 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.833+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5628f0219a70 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8800324a0 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb8740d6040 con 0x5628f02b4d70 2024-03-24T06:47:10.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 10 client.27037 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 10 client.27037 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 20 client.27037 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f03345c0 0x5628f03369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 4 client.27037 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628effa6890 con 0x5628f03345c0 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.834+0000 7fb88d200640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f03345c0 0x5628f03369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb88d200640 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f03345c0 0x5628f03369b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37711a80674226d0 server_cookie=623cfd799b5a2be in_seq=0 out_seq=0 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb86ea00640 10 client.27037 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:10.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb86ea00640 1 -- 172.21.15.22:0/3582315760 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628effa6890 con 0x5628f03345c0 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb86ea00640 10 client.27037 handle_command_reply: tid=0 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 client.27037 shutdown 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 msgr2=0x5628f02b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 0x5628f02b7e90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb87409f0a0 tx=0x7fb874010db0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 shutdown_connections 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5628f03345c0 0x5628f03369b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628f02bd950 0x5628f02b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb86ea00640 0 client.27037 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:10.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628f02b4d70 0x5628f02b7e90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 --2- 172.21.15.22:0/3582315760 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628f02b4450 0x5628f02b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.835+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 >> 172.21.15.22:0/3582315760 conn(0x5628f02aeb80 msgr2=0x5628f02ad430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 shutdown_connections 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb892964740 1 -- 172.21.15.22:0/3582315760 wait complete. 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb892964740 20 client.27037 trim_cache size 0 max 16384 2024-03-24T06:47:10.844 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 msgr2=0x7fb8700a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 0x7fb8700a4770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5628f02b3f20 tx=0x7fb87c00b410 comp rx=0 tx=0).stop 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 msgr2=0x7fb888073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.836+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb87400e9f0 tx=0x7fb87400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 shutdown_connections 2024-03-24T06:47:10.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb8700a22b0 0x7fb8700a4770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb888107660 0x7fb888074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb888101f60 0x7fb888073f50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 --2- 172.21.15.22:0/2785250410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb8881015a0 0x7fb888073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 >> 172.21.15.22:0/2785250410 conn(0x7fb8880fd0b0 msgr2=0x7fb8880fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 shutdown_connections 2024-03-24T06:47:10.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:10.837+0000 7fb88e600640 1 -- 172.21.15.22:0/2785250410 wait complete. 2024-03-24T06:47:10.862 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:47:11.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 -- 172.21.15.22:0/3129264126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100e40 msgr2=0x7f321010db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 --2- 172.21.15.22:0/3129264126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100e40 0x7f321010db90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009900 tx=0x7f31fc02bc40 comp rx=0 tx=0).stop 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 -- 172.21.15.22:0/3129264126 shutdown_connections 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 --2- 172.21.15.22:0/3129264126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321010e0d0 0x7f32101104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 --2- 172.21.15.22:0/3129264126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100e40 0x7f321010db90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 --2- 172.21.15.22:0/3129264126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3210100520 0x7f3210100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 -- 172.21.15.22:0/3129264126 >> 172.21.15.22:0/3129264126 conn(0x7f3210077770 msgr2=0x7f3210077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 -- 172.21.15.22:0/3129264126 shutdown_connections 2024-03-24T06:47:11.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.081+0000 7f3215400640 1 -- 172.21.15.22:0/3129264126 wait complete. 2024-03-24T06:47:11.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.082+0000 7f3215400640 1 Processor -- start 2024-03-24T06:47:11.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 -- start start 2024-03-24T06:47:11.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 0x7f3210102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f321010e0d0 0x7f3210103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f32101aa530 con 0x7f3210100520 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f32101aa670 con 0x7f3210100e40 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f3215400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f32101aa7e0 con 0x7f321010e0d0 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f320de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f320e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 0x7f3210102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f320de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60820/0 (socket says 172.21.15.22:60820) 2024-03-24T06:47:11.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f320de00640 1 -- 172.21.15.22:0/2393899838 learned_addr learned my addr 172.21.15.22:0/2393899838 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.083+0000 7f320f200640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f321010e0d0 0x7f3210103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 -- 172.21.15.22:0/2393899838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f321010e0d0 msgr2=0x7f3210103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f321010e0d0 0x7f3210103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 -- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 msgr2=0x7f3210102800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 0x7f3210102800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 -- 172.21.15.22:0/2393899838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31fc009560 con 0x7f3210100e40 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320e800640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 0x7f3210102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f320de00640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006010 tx=0x7f31fc02c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.084+0000 7f3207400640 1 -- 172.21.15.22:0/2393899838 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc004350 con 0x7f3210100e40 2024-03-24T06:47:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.085+0000 7f3207400640 1 -- 172.21.15.22:0/2393899838 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31fc0044b0 con 0x7f3210100e40 2024-03-24T06:47:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.085+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32101aaa60 con 0x7f3210100e40 2024-03-24T06:47:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.085+0000 7f3207400640 1 -- 172.21.15.22:0/2393899838 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc003d30 con 0x7f3210100e40 2024-03-24T06:47:11.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.085+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32101aafc0 con 0x7f3210100e40 2024-03-24T06:47:11.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.089+0000 7f32181b9740 1 Processor -- start 2024-03-24T06:47:11.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.089+0000 7f32181b9740 1 -- start start 2024-03-24T06:47:11.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.089+0000 7f3207400640 1 -- 172.21.15.22:0/2393899838 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f31fc004df0 con 0x7f3210100e40 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8c1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563a7f8266e0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563a7f823c10 con 0x563a7f8c1780 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f32181b9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563a7f81f3f0 con 0x563a7f8b8280 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48496/0 (socket says 172.21.15.22:48496) 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 -- 172.21.15.22:0/2325252944 learned_addr learned my addr 172.21.15.22:0/2325252944 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 -- 172.21.15.22:0/2325252944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 msgr2=0x563a7f8c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8c3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 -- 172.21.15.22:0/2325252944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 msgr2=0x563a7f8c1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8c1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320f200640 1 -- 172.21.15.22:0/2325252944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7f7e0e40 con 0x563a7f8b8280 2024-03-24T06:47:11.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.090+0000 7f320e800640 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:11.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f320f200640 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3208009f20 tx=0x7f32080118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c50783014c38bbe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f31fb400640 1 -- 172.21.15.22:0/2325252944 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208004d20 con 0x563a7f8b8280 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f31fb400640 1 -- 172.21.15.22:0/2325252944 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3208004e80 con 0x563a7f8b8280 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f31fb400640 1 -- 172.21.15.22:0/2325252944 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208005280 con 0x563a7f8b8280 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f3207400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 0x7f31f40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.091+0000 7f3207400640 1 -- 172.21.15.22:0/2393899838 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f31fc0f03a0 con 0x7f3210100e40 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f320e800640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 0x7f31f40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 -- 172.21.15.22:0/2325252944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 msgr2=0x563a7f8b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3208009f20 tx=0x7f32080118d0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 -- 172.21.15.22:0/2325252944 shutdown_connections 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 --2- 172.21.15.22:0/2325252944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8b8660 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 -- 172.21.15.22:0/2325252944 >> 172.21.15.22:0/2325252944 conn(0x563a7f8b29b0 msgr2=0x563a7f8b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 -- 172.21.15.22:0/2325252944 shutdown_connections 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f32181b9740 1 -- 172.21.15.22:0/2325252944 wait complete. 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.092+0000 7f320e800640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 0x7f31f40a4770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3200009fd0 tx=0x7f3200005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 -- start start 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 Processor -- start 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8c1780 0x563a7f8bf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563a7f7c4060 con 0x563a7f8b8ba0 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563a7f7c3d00 con 0x563a7f8b8280 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.093+0000 7f32181b9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563a7f58d1c0 con 0x563a7f8c1780 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f32181b9740 10 client.0 _ll_register_callbacks cb 0x563a7f52fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:11.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8c1780 0x563a7f8bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60834/0 (socket says 172.21.15.22:60834) 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 -- 172.21.15.22:0/1340912447 learned_addr learned my addr 172.21.15.22:0/1340912447 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 -- 172.21.15.22:0/1340912447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8c1780 msgr2=0x563a7f8bf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8c1780 0x563a7f8bf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 -- 172.21.15.22:0/1340912447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 msgr2=0x563a7f8bedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 -- 172.21.15.22:0/1340912447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3208009bd0 con 0x563a7f8b8280 2024-03-24T06:47:11.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f320e800640 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32000042d0 tx=0x7f3200004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff9c4c32de671a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.094+0000 7f31efe00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200017070 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a7f55b720 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f3200004480 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3200003d80 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f32181b9740 10 client.27029 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563a7f81da70 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563a7f55b720 con 0x563a7f8b8280 2024-03-24T06:47:11.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.095+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f320002a720 con 0x563a7f8b8280 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 10 client.27029 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 10 client.27029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 20 client.27029 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f8d1dd0 0x563a7f8d4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 4 client.27029 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a7f5aa890 con 0x563a7f8d1dd0 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f320de00640 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f8d1dd0 0x563a7f8d4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f320de00640 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f8d1dd0 0x563a7f8d4230 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2dd9c7f7b248265f server_cookie=3c9baa8dad46b884 in_seq=0 out_seq=0 2024-03-24T06:47:11.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.096+0000 7f31efe00640 10 client.27029 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:11.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f31efe00640 1 -- 172.21.15.22:0/1340912447 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a7f5aa890 con 0x563a7f8d1dd0 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f31efe00640 10 client.27029 handle_command_reply: tid=0 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 client.27029 shutdown 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 msgr2=0x563a7f8be880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32000042d0 tx=0x7f3200004730 comp rx=0 tx=0).stop 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 shutdown_connections 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8c1780 0x563a7f8bf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f8d1dd0 0x563a7f8d4230 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 --2- 172.21.15.22:0/1340912447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8b8280 0x563a7f8be880 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 >> 172.21.15.22:0/1340912447 conn(0x563a7f8b29b0 msgr2=0x563a7f8b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.097+0000 7f31efe00640 0 client.27029 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.098+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 shutdown_connections 2024-03-24T06:47:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.098+0000 7f32181b9740 1 -- 172.21.15.22:0/1340912447 wait complete. 2024-03-24T06:47:11.105 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.098+0000 7f32181b9740 20 client.27029 trim_cache size 0 max 16384 2024-03-24T06:47:11.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 Processor -- start 2024-03-24T06:47:11.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 -- start start 2024-03-24T06:47:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8280 0x563a7f8cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8ba0 0x563a7f8b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563a7f8b7cc0 con 0x563a7f8b8280 2024-03-24T06:47:11.114 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563a7f8cd550 con 0x563a7f8c1780 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f32181b9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563a7f8b88a0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f320f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f320f200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60836/0 (socket says 172.21.15.22:60836) 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f320f200640 1 -- 172.21.15.22:0/2612812399 learned_addr learned my addr 172.21.15.22:0/2612812399 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f320e800640 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8280 0x563a7f8cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.113+0000 7f320de00640 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8ba0 0x563a7f8b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 -- 172.21.15.22:0/2612812399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8ba0 msgr2=0x563a7f8b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8ba0 0x563a7f8b18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.115 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 -- 172.21.15.22:0/2612812399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8280 msgr2=0x563a7f8cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8280 0x563a7f8cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 -- 172.21.15.22:0/2612812399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7f7b9b00 con 0x563a7f8c1780 2024-03-24T06:47:11.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f320f200640 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3208009230 tx=0x7f32080046b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665503e31591f9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f31faa00640 1 -- 172.21.15.22:0/2612812399 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32080042d0 con 0x563a7f8c1780 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f31faa00640 1 -- 172.21.15.22:0/2612812399 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3208027070 con 0x563a7f8c1780 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.114+0000 7f31faa00640 1 -- 172.21.15.22:0/2612812399 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208005000 con 0x563a7f8c1780 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 -- 172.21.15.22:0/2612812399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 msgr2=0x563a7f8d9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3208009230 tx=0x7f32080046b0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 -- 172.21.15.22:0/2612812399 shutdown_connections 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8d9b40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8ba0 0x563a7f8b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 --2- 172.21.15.22:0/2612812399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8280 0x563a7f8cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 -- 172.21.15.22:0/2612812399 >> 172.21.15.22:0/2612812399 conn(0x563a7f8b29b0 msgr2=0x563a7f8b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 -- 172.21.15.22:0/2612812399 shutdown_connections 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.115+0000 7f32181b9740 1 -- 172.21.15.22:0/2612812399 wait complete. 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 -- start start 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 Processor -- start 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563a7f8bc140 con 0x563a7f8b8ba0 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563a7f937170 con 0x563a7f8c1780 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563a7f9372b0 con 0x563a7f8b8280 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.116+0000 7f32181b9740 10 client.0 _ll_register_callbacks cb 0x7f3200012110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48512/0 (socket says 172.21.15.22:48512) 2024-03-24T06:47:11.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32848/0 (socket says 172.21.15.22:32848) 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 -- 172.21.15.22:0/2250435701 learned_addr learned my addr 172.21.15.22:0/2250435701 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320f200640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 -- 172.21.15.22:0/2250435701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 msgr2=0x563a7f8c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 -- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 msgr2=0x563a7f8bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8bba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 -- 172.21.15.22:0/2250435701 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a7f7e0e40 con 0x563a7f8b8ba0 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320e800640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:11.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320f200640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f320de00640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0ed8a0 tx=0x7f31fc0f2870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1389055f435bc1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.117+0000 7f31efe00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0bc3f0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f31fc0eee40 con 0x563a7f8b8ba0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a7f51e7e0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f31fc003e90 con 0x563a7f8b8ba0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 10 client.16668 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563a7f7df6a0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.123 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563a7f51e7e0 con 0x563a7f8b8ba0 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f31fc03c070 con 0x563a7f8b8ba0 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 10 client.16668 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 10 client.16668 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 20 client.16668 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.118+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f938930 0x563a7f93ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.119+0000 7f32181b9740 4 client.16668 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.119+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a7f4ca700 con 0x563a7f938930 2024-03-24T06:47:11.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.119+0000 7f320e800640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f938930 0x563a7f93ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.119+0000 7f320e800640 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f938930 0x563a7f93ad50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1d48cf12b93dadb server_cookie=b0905f82ea9e672b in_seq=0 out_seq=0 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.119+0000 7f31efe00640 10 client.16668 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f31efe00640 1 -- 172.21.15.22:0/2250435701 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a7f4ca700 con 0x563a7f938930 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f31efe00640 10 client.16668 handle_command_reply: tid=0 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 client.16668 shutdown 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 msgr2=0x563a7f8bb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f31fc0ed8a0 tx=0x7f31fc0f2870 comp rx=0 tx=0).stop 2024-03-24T06:47:11.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 shutdown_connections 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x563a7f938930 0x563a7f93ad50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563a7f8c1780 0x563a7f8bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563a7f8b8ba0 0x563a7f8bb300 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 --2- 172.21.15.22:0/2250435701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a7f8b8280 0x563a7f8c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 >> 172.21.15.22:0/2250435701 conn(0x563a7f8b29b0 msgr2=0x563a7f8b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f31efe00640 0 client.16668 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 shutdown_connections 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 1 -- 172.21.15.22:0/2250435701 wait complete. 2024-03-24T06:47:11.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.120+0000 7f32181b9740 20 client.16668 trim_cache size 0 max 16384 2024-03-24T06:47:11.128 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:11.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 msgr2=0x7f31f40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.128 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 0x7f31f40a4770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3200009fd0 tx=0x7f3200005f90 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 msgr2=0x7f3210102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f31fc006010 tx=0x7f31fc02c680 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 shutdown_connections 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f31f40a22b0 0x7f31f40a4770 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f321010e0d0 0x7f3210103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3210100e40 0x7f3210102d40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 --2- 172.21.15.22:0/2393899838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3210100520 0x7f3210102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 >> 172.21.15.22:0/2393899838 conn(0x7f3210077770 msgr2=0x7f321010f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 shutdown_connections 2024-03-24T06:47:11.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.121+0000 7f3215400640 1 -- 172.21.15.22:0/2393899838 wait complete. 2024-03-24T06:47:11.140 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 -- 172.21.15.22:0/3727573548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 msgr2=0x7f5a0410b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3727573548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0410b540 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009900 tx=0x7f59ec02bcc0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 -- 172.21.15.22:0/3727573548 shutdown_connections 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3727573548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a0410bb60 0x7f5a0410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3727573548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0410b540 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3727573548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04102740 0x7f5a04102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 -- 172.21.15.22:0/3727573548 >> 172.21.15.22:0/3727573548 conn(0x7f5a040fbf60 msgr2=0x7f5a040fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 -- 172.21.15.22:0/3727573548 shutdown_connections 2024-03-24T06:47:11.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.373+0000 7f5a09e00640 1 -- 172.21.15.22:0/3727573548 wait complete. 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.374+0000 7f5a09e00640 1 Processor -- start 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 -- start start 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a04102740 0x7f5a04199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0410bb60 0x7f5a0419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0419ecb0 con 0x7f5a04102740 2024-03-24T06:47:11.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0419ee20 con 0x7f5a0410bb60 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0419efe0 con 0x7f5a04103060 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a04102740 0x7f5a04199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f5a09400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0410bb60 0x7f5a0419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48518/0 (socket says 172.21.15.22:48518) 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.375+0000 7f59fbe00640 1 -- 172.21.15.22:0/3072114117 learned_addr learned my addr 172.21.15.22:0/3072114117 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 -- 172.21.15.22:0/3072114117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0410bb60 msgr2=0x7f5a0419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0410bb60 0x7f5a0419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 -- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a04102740 msgr2=0x7f5a04199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a04102740 0x7f5a04199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 -- 172.21.15.22:0/3072114117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec009560 con 0x7f5a04103060 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f59ec000c00 tx=0x7f59ec02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f5a02000640 1 -- 172.21.15.22:0/3072114117 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec02dbd0 con 0x7f5a04103060 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f5a02000640 1 -- 172.21.15.22:0/3072114117 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec02dd30 con 0x7f5a04103060 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0419f260 con 0x7f5a04103060 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.376+0000 7f5a02000640 1 -- 172.21.15.22:0/3072114117 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec003b20 con 0x7f5a04103060 2024-03-24T06:47:11.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.377+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041a6b40 con 0x7f5a04103060 2024-03-24T06:47:11.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.379+0000 7f5a02000640 1 -- 172.21.15.22:0/3072114117 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f59ec003c80 con 0x7f5a04103060 2024-03-24T06:47:11.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a02000640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 0x7f59e80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 Processor -- start 2024-03-24T06:47:11.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a02000640 1 -- 172.21.15.22:0/3072114117 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f59ec0f0980 con 0x7f5a04103060 2024-03-24T06:47:11.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 -- start start 2024-03-24T06:47:11.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a08a00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 0x7f59e80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39450 0x559755d39830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d42950 0x559755d44d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559755ca76e0 con 0x559755d42950 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559755ca4c10 con 0x559755d39450 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.381+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559755ca03f0 con 0x559755d39d70 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48532/0 (socket says 172.21.15.22:48532) 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 -- 172.21.15.22:0/3428763203 learned_addr learned my addr 172.21.15.22:0/3428763203 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f5a08a00640 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39450 0x559755d39830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f5a09400640 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d42950 0x559755d44d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 -- 172.21.15.22:0/3428763203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39450 msgr2=0x559755d39830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39450 0x559755d39830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 -- 172.21.15.22:0/3428763203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d42950 msgr2=0x559755d44d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d42950 0x559755d44d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f5a08a00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 0x7f59e80a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009c20 tx=0x7f59f4005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 -- 172.21.15.22:0/3428763203 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559755c61e40 con 0x559755d39d70 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.382+0000 7f59fbe00640 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0b9a70 tx=0x7f59ec009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6f867cf612aa27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f59f9c00640 1 -- 172.21.15.22:0/3428763203 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0c11f0 con 0x559755d39d70 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f59f9c00640 1 -- 172.21.15.22:0/3428763203 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec0c57b0 con 0x559755d39d70 2024-03-24T06:47:11.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f59f9c00640 1 -- 172.21.15.22:0/3428763203 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec005660 con 0x559755d39d70 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/3428763203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 msgr2=0x559755d42410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0b9a70 tx=0x7f59ec009260 comp rx=0 tx=0).stop 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/3428763203 shutdown_connections 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d42950 0x559755d44d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d42410 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/3428763203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39450 0x559755d39830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.383+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/3428763203 >> 172.21.15.22:0/3428763203 conn(0x559755d33b80 msgr2=0x559755d33f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.384+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/3428763203 shutdown_connections 2024-03-24T06:47:11.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.384+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/3428763203 wait complete. 2024-03-24T06:47:11.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.384+0000 7f5a0e1fe740 1 -- start start 2024-03-24T06:47:11.388 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.384+0000 7f5a0e1fe740 1 Processor -- start 2024-03-24T06:47:11.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d42950 0x559755d40470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559755c45060 con 0x559755d39450 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559755c44d00 con 0x559755d39d70 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559755a0e1c0 con 0x559755d42950 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a0e1fe740 10 client.0 _ll_register_callbacks cb 0x5597559b0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60898/0 (socket says 172.21.15.22:60898) 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32878/0 (socket says 172.21.15.22:32878) 2024-03-24T06:47:11.392 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a08a00640 1 -- 172.21.15.22:0/1478427695 learned_addr learned my addr 172.21.15.22:0/1478427695 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f5a09400640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d42950 0x559755d40470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f59fbe00640 1 -- 172.21.15.22:0/1478427695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d42950 msgr2=0x559755d40470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.385+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d42950 0x559755d40470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59fbe00640 1 -- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 msgr2=0x559755d3f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59fbe00640 1 -- 172.21.15.22:0/1478427695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041a6b40 con 0x559755d39d70 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x559755d38f20 tx=0x7f59ec02bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=439105a63dd0dfe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59f2000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0db070 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f59ec0b9860 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55975599f7e0 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f59ec0c11f0 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f5a0e1fe740 10 client.27047 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559755c9ea70 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.386+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec0a3080 con 0x559755d39d70 2024-03-24T06:47:11.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f59ec03c070 con 0x559755d39d70 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 10 client.27047 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 10 client.27047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 20 client.27047 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755d526b0 0x559755d54ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 4 client.27047 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597559dc720 con 0x559755d526b0 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.387+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755d526b0 0x559755d54ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.388+0000 7f59f2000640 10 client.27047 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:11.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.388+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755d526b0 0x559755d54ad0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c56182997407a7d3 server_cookie=fe8c377954ef0cdc in_seq=0 out_seq=0 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.388+0000 7f59f2000640 1 -- 172.21.15.22:0/1478427695 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597559dc720 con 0x559755d526b0 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.388+0000 7f59f2000640 10 client.27047 handle_command_reply: tid=0 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.388+0000 7f5a0e1fe740 1 client.27047 shutdown 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 msgr2=0x559755d3ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x559755d38f20 tx=0x7f59ec02bcc0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 shutdown_connections 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d42950 0x559755d40470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755d526b0 0x559755d54ad0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d39d70 0x559755d3ff30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1478427695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d3f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f59f2000640 0 client.27047 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 >> 172.21.15.22:0/1478427695 conn(0x559755d33b80 msgr2=0x559755d358d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 shutdown_connections 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1478427695 wait complete. 2024-03-24T06:47:11.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.389+0000 7f5a0e1fe740 20 client.27047 trim_cache size 0 max 16384 2024-03-24T06:47:11.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:11 smithi022 ceph-mon[32003]: pgmap v221: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.416+0000 7f5a0e1fe740 1 Processor -- start 2024-03-24T06:47:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.416+0000 7f5a0e1fe740 1 -- start start 2024-03-24T06:47:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d40290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559755d2f4b0 con 0x559755d39450 2024-03-24T06:47:11.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559755d3e9d0 con 0x559755d42950 2024-03-24T06:47:11.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559755d4e3f0 con 0x559755d39d70 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d42120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32880/0 (socket says 172.21.15.22:32880) 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a08a00640 1 -- 172.21.15.22:0/2767898944 learned_addr learned my addr 172.21.15.22:0/2767898944 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f59fbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48538/0 (socket says 172.21.15.22:48538) 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.417+0000 7f5a09400640 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d40290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 -- 172.21.15.22:0/2767898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 msgr2=0x559755d40290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d40290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 -- 172.21.15.22:0/2767898944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 msgr2=0x559755d42120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d42120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 -- 172.21.15.22:0/2767898944 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559755c3ab00 con 0x559755d39d70 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59fbe00640 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0bcda0 tx=0x7f59ec009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=103cdb36d5c4d535 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59f9200640 1 -- 172.21.15.22:0/2767898944 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0db040 con 0x559755d39d70 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59f9200640 1 -- 172.21.15.22:0/2767898944 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55975599f7e0 con 0x559755d39d70 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.418+0000 7f59f9200640 1 -- 172.21.15.22:0/2767898944 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0bd0a0 con 0x559755d39d70 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/2767898944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 msgr2=0x559755d4deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0bcda0 tx=0x7f59ec009260 comp rx=0 tx=0).stop 2024-03-24T06:47:11.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/2767898944 shutdown_connections 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d40290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39d70 0x559755d4deb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/2767898944 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39450 0x559755d42120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/2767898944 >> 172.21.15.22:0/2767898944 conn(0x559755d33b80 msgr2=0x559755d3e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/2767898944 shutdown_connections 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/2767898944 wait complete. 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.419+0000 7f5a0e1fe740 1 -- start start 2024-03-24T06:47:11.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 Processor -- start 2024-03-24T06:47:11.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39d70 0x559755d45870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d45db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559755d462f0 con 0x559755d39d70 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559755d46460 con 0x559755d42950 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559755d46620 con 0x559755d39450 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a0e1fe740 10 client.0 _ll_register_callbacks cb 0x7f59fc009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a08a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48554/0 (socket says 172.21.15.22:48554) 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a08a00640 1 -- 172.21.15.22:0/1778887437 learned_addr learned my addr 172.21.15.22:0/1778887437 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:11.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f5a09400640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d45db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.420+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39d70 0x559755d45870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 -- 172.21.15.22:0/1778887437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 msgr2=0x559755d45db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d45db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 -- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39d70 msgr2=0x559755d45870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39d70 0x559755d45870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 -- 172.21.15.22:0/1778887437 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559755c9ea70 con 0x559755d39450 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a08a00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f59f401ca80 tx=0x7f59f4013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c51b5946789a3c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f59f2000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f402d040 con 0x559755d39450 2024-03-24T06:47:11.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ec0c57b0 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.421+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f59f40275f0 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f59f40277b0 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 10 client.27073 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559755c606a0 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec0c57b0 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f59f4036b70 con 0x559755d39450 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 10 client.27073 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 10 client.27073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:11.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 20 client.27073 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755dc1720 0x559755dc3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 4 client.27073 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.422+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559755a2b890 con 0x559755dc1720 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.423+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755dc1720 0x559755dc3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.423+0000 7f59fbe00640 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755dc1720 0x559755dc3b40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c1e5fec0be01b3 server_cookie=8643cc76813c1305 in_seq=0 out_seq=0 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.423+0000 7f59f2000640 10 client.27073 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f59f2000640 1 -- 172.21.15.22:0/1778887437 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559755a2b890 con 0x559755dc1720 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f59f2000640 10 client.27073 handle_command_reply: tid=0 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 client.27073 shutdown 2024-03-24T06:47:11.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 msgr2=0x559755d4a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f59f401ca80 tx=0x7f59f4013040 comp rx=0 tx=0).stop 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 shutdown_connections 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559755dc1720 0x559755dc3b40 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559755d42950 0x559755d45db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559755d39d70 0x559755d45870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 --2- 172.21.15.22:0/1778887437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559755d39450 0x559755d4a500 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 >> 172.21.15.22:0/1778887437 conn(0x559755d33b80 msgr2=0x559755d32430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f59f2000640 0 client.27073 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 shutdown_connections 2024-03-24T06:47:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 1 -- 172.21.15.22:0/1778887437 wait complete. 2024-03-24T06:47:11.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.424+0000 7f5a0e1fe740 20 client.27073 trim_cache size 0 max 16384 2024-03-24T06:47:11.431 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 msgr2=0x7f59e80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 0x7f59e80a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009c20 tx=0x7f59f4005f90 comp rx=0 tx=0).stop 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 msgr2=0x7f5a0419a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f59ec000c00 tx=0x7f59ec02c6c0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 shutdown_connections 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59e80a22b0 0x7f59e80a4770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0410bb60 0x7f5a0419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04103060 0x7f5a0419a330 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 --2- 172.21.15.22:0/3072114117 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5a04102740 0x7f5a04199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 >> 172.21.15.22:0/3072114117 conn(0x7f5a040fbf60 msgr2=0x7f5a040fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 shutdown_connections 2024-03-24T06:47:11.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:11.425+0000 7f5a09e00640 1 -- 172.21.15.22:0/3072114117 wait complete. 2024-03-24T06:47:11.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:11 smithi143 ceph-mon[36520]: pgmap v221: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:11.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:11 smithi155 ceph-mon[36262]: pgmap v221: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:47:12.444 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:47:12.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:47:12.445 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:47:12.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 -- 172.21.15.22:0/4026682871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748073ae0 msgr2=0x7f7748073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 --2- 172.21.15.22:0/4026682871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748073ae0 0x7f7748073ec0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f773c009900 tx=0x7f773c02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 -- 172.21.15.22:0/4026682871 shutdown_connections 2024-03-24T06:47:12.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 --2- 172.21.15.22:0/4026682871 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f7748110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 --2- 172.21.15.22:0/4026682871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748074400 0x7f774810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 --2- 172.21.15.22:0/4026682871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748073ae0 0x7f7748073ec0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 -- 172.21.15.22:0/4026682871 >> 172.21.15.22:0/4026682871 conn(0x7f77480fd0b0 msgr2=0x7f77480ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 -- 172.21.15.22:0/4026682871 shutdown_connections 2024-03-24T06:47:12.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.682+0000 7f774e800640 1 -- 172.21.15.22:0/4026682871 wait complete. 2024-03-24T06:47:12.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.683+0000 7f774e800640 1 Processor -- start 2024-03-24T06:47:12.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.683+0000 7f774e800640 1 -- start start 2024-03-24T06:47:12.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748074400 0x7f774819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f774819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f774819ed10 con 0x7f774810de20 2024-03-24T06:47:12.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f774819ee80 con 0x7f7748074400 2024-03-24T06:47:12.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774e800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f774819f040 con 0x7f7748073ae0 2024-03-24T06:47:12.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48558/0 (socket says 172.21.15.22:48558) 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774d400640 1 -- 172.21.15.22:0/1782280784 learned_addr learned my addr 172.21.15.22:0/1782280784 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774ca00640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748074400 0x7f774819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.684+0000 7f774de00640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f774819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 -- 172.21.15.22:0/1782280784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748074400 msgr2=0x7f774819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748074400 0x7f774819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 -- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 msgr2=0x7f774819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f774819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 -- 172.21.15.22:0/1782280784 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c009560 con 0x7f7748073ae0 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774de00640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f774819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774d400640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f773c000c00 tx=0x7f773c02c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f7746000640 1 -- 172.21.15.22:0/1782280784 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c02dc60 con 0x7f7748073ae0 2024-03-24T06:47:12.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774819f2c0 con 0x7f7748073ae0 2024-03-24T06:47:12.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f7746000640 1 -- 172.21.15.22:0/1782280784 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f773c02ddc0 con 0x7f7748073ae0 2024-03-24T06:47:12.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.685+0000 7f7746000640 1 -- 172.21.15.22:0/1782280784 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c003bf0 con 0x7f7748073ae0 2024-03-24T06:47:12.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.686+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77481a6ba0 con 0x7f7748073ae0 2024-03-24T06:47:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.689+0000 7f7752bc3740 1 Processor -- start 2024-03-24T06:47:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.689+0000 7f7752bc3740 1 -- start start 2024-03-24T06:47:12.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.689+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0baff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7746000640 1 -- 172.21.15.22:0/1782280784 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f773c003d50 con 0x7f7748073ae0 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7752bc3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56476b0202d0 con 0x56476b0bb530 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7752bc3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56476b01d800 con 0x56476b0b2030 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f7752bc3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56476b018fe0 con 0x56476b0b2950 2024-03-24T06:47:12.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60956/0 (socket says 172.21.15.22:60956) 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 -- 172.21.15.22:0/3970335202 learned_addr learned my addr 172.21.15.22:0/3970335202 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774ca00640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774d400640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0baff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 -- 172.21.15.22:0/3970335202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 msgr2=0x56476b0baff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0baff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 -- 172.21.15.22:0/3970335202 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 msgr2=0x56476b0bd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774de00640 1 -- 172.21.15.22:0/3970335202 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56476afdae90 con 0x56476b0b2030 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.690+0000 7f774ca00640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.691+0000 7f774de00640 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7740009ea0 tx=0x7f7740009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd7f066b4719baf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.691+0000 7f773a600640 1 -- 172.21.15.22:0/3970335202 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774001b660 con 0x56476b0b2030 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.691+0000 7f773a600640 1 -- 172.21.15.22:0/3970335202 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f774001bca0 con 0x56476b0b2030 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.691+0000 7f773a600640 1 -- 172.21.15.22:0/3970335202 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740012e40 con 0x56476b0b2030 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.691+0000 7f7746000640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 0x7f77300a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 -- 172.21.15.22:0/3970335202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 msgr2=0x56476b0b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7740009ea0 tx=0x7f7740009f90 comp rx=0 tx=0).stop 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7746000640 1 -- 172.21.15.22:0/1782280784 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f773c0f09f0 con 0x7f7748073ae0 2024-03-24T06:47:12.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 -- 172.21.15.22:0/3970335202 shutdown_connections 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0baff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3970335202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0b2410 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 -- 172.21.15.22:0/3970335202 >> 172.21.15.22:0/3970335202 conn(0x56476b0ac760 msgr2=0x56476b0acb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f774d400640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 0x7f77300a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 -- 172.21.15.22:0/3970335202 shutdown_connections 2024-03-24T06:47:12.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.692+0000 7f7752bc3740 1 -- 172.21.15.22:0/3970335202 wait complete. 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 -- start start 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f774d400640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 0x7f77300a4770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x56476b0b1b00 tx=0x7f773c0f3f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 Processor -- start 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2030 0x56476b0b8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0b8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56476afbdfc0 con 0x56476b0b2030 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56476afbdc60 con 0x56476b0b2950 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.693+0000 7f7752bc3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56476ad610c0 con 0x56476b0bb530 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f7752bc3740 10 client.0 _ll_register_callbacks cb 0x56476afd5e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774de00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2030 0x56476b0b8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60970/0 (socket says 172.21.15.22:60970) 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 -- 172.21.15.22:0/419645155 learned_addr learned my addr 172.21.15.22:0/419645155 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774de00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0b8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 -- 172.21.15.22:0/419645155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 msgr2=0x56476b0b8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0b8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 -- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2030 msgr2=0x56476b0b8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2030 0x56476b0b8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774ca00640 1 -- 172.21.15.22:0/419645155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7740009b00 con 0x56476b0b2950 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.694+0000 7f774de00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f774ca00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7734012700 tx=0x7f7734012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ffe399636f5c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f772ea00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734017070 con 0x56476b0b2950 2024-03-24T06:47:12.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56476b03de90 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f7734004050 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7734004210 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f7752bc3740 10 client.27065 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56476b017660 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.695+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56476b03de90 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7734003a60 con 0x56476b0b2950 2024-03-24T06:47:12.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 10 client.27065 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 10 client.27065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 20 client.27065 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b0cb9d0 0x56476b0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 4 client.27065 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56476b00f210 con 0x56476b0cb9d0 2024-03-24T06:47:12.710 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.696+0000 7f774de00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b0cb9d0 0x56476b0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.697+0000 7f774de00640 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b0cb9d0 0x56476b0cddf0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44b1ce82581384a3 server_cookie=ad83cefe1b088e98 in_seq=0 out_seq=0 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.697+0000 7f772ea00640 10 client.27065 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.697+0000 7f772ea00640 1 -- 172.21.15.22:0/419645155 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56476b00f210 con 0x56476b0cb9d0 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.697+0000 7f772ea00640 10 client.27065 handle_command_reply: tid=0 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 client.27065 shutdown 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 msgr2=0x56476b0b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.711 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7734012700 tx=0x7f7734012bc0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 shutdown_connections 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0b8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b0cb9d0 0x56476b0cddf0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2950 0x56476b0b8ab0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 --2- 172.21.15.22:0/419645155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2030 0x56476b0b8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f772ea00640 0 client.27065 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 >> 172.21.15.22:0/419645155 conn(0x56476b0ac760 msgr2=0x56476b0ae4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 shutdown_connections 2024-03-24T06:47:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 1 -- 172.21.15.22:0/419645155 wait complete. 2024-03-24T06:47:12.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.698+0000 7f7752bc3740 20 client.27065 trim_cache size 0 max 16384 2024-03-24T06:47:12.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.717+0000 7f7752bc3740 1 Processor -- start 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 -- start start 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2950 0x56476b0c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56476b0b94d0 con 0x56476b0b2950 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56476b0a8030 con 0x56476b0b2030 2024-03-24T06:47:12.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f7752bc3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56476b0b98c0 con 0x56476b0bb530 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f774de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f774de00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:60972/0 (socket says 172.21.15.22:60972) 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f774de00640 1 -- 172.21.15.22:0/2137404300 learned_addr learned my addr 172.21.15.22:0/2137404300 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f774ca00640 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2950 0x56476b0c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.718+0000 7f774de00640 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 -- 172.21.15.22:0/2137404300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 msgr2=0x56476b0d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0d3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 -- 172.21.15.22:0/2137404300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2950 msgr2=0x56476b0c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2950 0x56476b0c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 -- 172.21.15.22:0/2137404300 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56476afb3ae0 con 0x56476b0b2030 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f774de00640 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f77400127f0 tx=0x7f7740012980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8cb6a528238c4af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f7739c00640 1 -- 172.21.15.22:0/2137404300 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774001f040 con 0x56476b0b2030 2024-03-24T06:47:12.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f7739c00640 1 -- 172.21.15.22:0/2137404300 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7740005ac0 con 0x56476b0b2030 2024-03-24T06:47:12.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.719+0000 7f7739c00640 1 -- 172.21.15.22:0/2137404300 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774002e3e0 con 0x56476b0b2030 2024-03-24T06:47:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- 172.21.15.22:0/2137404300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 msgr2=0x56476b0baca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f77400127f0 tx=0x7f7740012980 comp rx=0 tx=0).stop 2024-03-24T06:47:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- 172.21.15.22:0/2137404300 shutdown_connections 2024-03-24T06:47:12.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0bb530 0x56476b0d3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0b2950 0x56476b0c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 --2- 172.21.15.22:0/2137404300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0baca0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- 172.21.15.22:0/2137404300 >> 172.21.15.22:0/2137404300 conn(0x56476b0ac760 msgr2=0x56476b0b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- 172.21.15.22:0/2137404300 shutdown_connections 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- 172.21.15.22:0/2137404300 wait complete. 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 -- start start 2024-03-24T06:47:12.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.720+0000 7f7752bc3740 1 Processor -- start 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56476b0bfa60 con 0x56476b0bb530 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56476b0bfbd0 con 0x56476b0b2030 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56476b0bfd90 con 0x56476b0b2950 2024-03-24T06:47:12.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f7752bc3740 10 client.0 _ll_register_callbacks cb 0x7f7734009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774ca00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48590/0 (socket says 172.21.15.22:48590) 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774ca00640 1 -- 172.21.15.22:0/3212063218 learned_addr learned my addr 172.21.15.22:0/3212063218 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774de00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774de00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.721+0000 7f774ca00640 1 -- 172.21.15.22:0/3212063218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 msgr2=0x56476b0c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f774ca00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f774ca00640 1 -- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 msgr2=0x56476b0bf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f774ca00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f774ca00640 1 -- 172.21.15.22:0/3212063218 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7740022040 con 0x56476b0b2950 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f774ca00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x56476b0b7dd0 tx=0x7f7734010660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8dbac3538a1a0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f772ea00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:12.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7734017070 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56476b0b83a0 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7f77340052c0 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7734005480 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f7752bc3740 10 client.27091 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.722+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56476afd96f0 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56476b0b83a0 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f773400fb60 con 0x56476b0b2950 2024-03-24T06:47:12.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 10 client.27091 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 10 client.27091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 20 client.27091 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b132f00 0x56476b135320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 4 client.27091 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f774001bca0 con 0x56476b132f00 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.723+0000 7f774de00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b132f00 0x56476b135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.724+0000 7f774de00640 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b132f00 0x56476b135320 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50c5d7221647ef47 server_cookie=19bcf834a3a33a60 in_seq=0 out_seq=0 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.724+0000 7f772ea00640 10 client.27091 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f772ea00640 1 -- 172.21.15.22:0/3212063218 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f774001bca0 con 0x56476b132f00 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f772ea00640 10 client.27091 handle_command_reply: tid=0 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 client.27091 shutdown 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 msgr2=0x56476b0c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x56476b0b7dd0 tx=0x7f7734010660 comp rx=0 tx=0).stop 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 shutdown_connections 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56476b0bb530 0x56476b0bf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56476b0b2950 0x56476b0c3470 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56476b132f00 0x56476b135320 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 --2- 172.21.15.22:0/3212063218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56476b0b2030 0x56476b0c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 >> 172.21.15.22:0/3212063218 conn(0x56476b0ac760 msgr2=0x56476b0aaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f772ea00640 0 client.27091 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.725+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 shutdown_connections 2024-03-24T06:47:12.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.726+0000 7f7752bc3740 1 -- 172.21.15.22:0/3212063218 wait complete. 2024-03-24T06:47:12.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.726+0000 7f7752bc3740 20 client.27091 trim_cache size 0 max 16384 2024-03-24T06:47:12.736 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:12.736 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:47:12.737 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "59bd0f52-8898-47b4-a4a2-4686a4103c00", 2024-03-24T06:47:12.737 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:47:12.737 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 msgr2=0x7f77300a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 0x7f77300a4770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x56476b0b1b00 tx=0x7f773c0f3f90 comp rx=0 tx=0).stop 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 msgr2=0x7f7748199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f773c000c00 tx=0x7f773c02c740 comp rx=0 tx=0).stop 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 shutdown_connections 2024-03-24T06:47:12.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f77300a22b0 0x7f77300a4770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f774810de20 0x7f774819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7748074400 0x7f774819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 --2- 172.21.15.22:0/1782280784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7748073ae0 0x7f7748199e50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 >> 172.21.15.22:0/1782280784 conn(0x7f77480fd0b0 msgr2=0x7f77480fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 shutdown_connections 2024-03-24T06:47:12.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.727+0000 7f774e800640 1 -- 172.21.15.22:0/1782280784 wait complete. 2024-03-24T06:47:12.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:47:12.751 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:47:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 -- 172.21.15.22:0/2680246352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc076560 msgr2=0x7faccc112010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 --2- 172.21.15.22:0/2680246352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc076560 0x7faccc112010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7facc0009930 tx=0x7facc002bc30 comp rx=0 tx=0).stop 2024-03-24T06:47:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 -- 172.21.15.22:0/2680246352 shutdown_connections 2024-03-24T06:47:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 --2- 172.21.15.22:0/2680246352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc112550 0x7faccc114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 --2- 172.21.15.22:0/2680246352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc076560 0x7faccc112010 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 --2- 172.21.15.22:0/2680246352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc075c40 0x7faccc076020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.958+0000 7facd4400640 1 -- 172.21.15.22:0/2680246352 >> 172.21.15.22:0/2680246352 conn(0x7faccc0ff360 msgr2=0x7faccc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.959+0000 7facd4400640 1 -- 172.21.15.22:0/2680246352 shutdown_connections 2024-03-24T06:47:12.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.959+0000 7facd4400640 1 -- 172.21.15.22:0/2680246352 wait complete. 2024-03-24T06:47:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 Processor -- start 2024-03-24T06:47:12.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 -- start start 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 0x7faccc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 0x7faccc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 0x7faccc1a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7faccc1a32e0 con 0x7faccc075c40 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7faccc1a3450 con 0x7faccc112550 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.960+0000 7facd4400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faccc1a3610 con 0x7faccc076560 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 0x7faccc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 0x7faccc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 0x7faccc19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32958/0 (socket says 172.21.15.22:32958) 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1a00640 1 -- 172.21.15.22:0/2358380445 learned_addr learned my addr 172.21.15.22:0/2358380445 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd2400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 0x7faccc1a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 -- 172.21.15.22:0/2358380445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 msgr2=0x7faccc1a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 0x7faccc1a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 -- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 msgr2=0x7faccc19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 0x7faccc19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd1000640 1 -- 172.21.15.22:0/2358380445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc0009560 con 0x7faccc076560 2024-03-24T06:47:12.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.961+0000 7facd2400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 0x7faccc1a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:12.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facd1000640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 0x7faccc19e960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7facc0036040 tx=0x7facc002c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facca000640 1 -- 172.21.15.22:0/2358380445 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0004350 con 0x7faccc076560 2024-03-24T06:47:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facca000640 1 -- 172.21.15.22:0/2358380445 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facc00044b0 con 0x7faccc076560 2024-03-24T06:47:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faccc1a3890 con 0x7faccc076560 2024-03-24T06:47:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facca000640 1 -- 172.21.15.22:0/2358380445 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0003cc0 con 0x7faccc076560 2024-03-24T06:47:12.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.962+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faccc1ab170 con 0x7faccc076560 2024-03-24T06:47:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.965+0000 7facca000640 1 -- 172.21.15.22:0/2358380445 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7facc0004df0 con 0x7faccc076560 2024-03-24T06:47:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.965+0000 7facd707f740 1 Processor -- start 2024-03-24T06:47:12.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 -- start start 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf22759200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf227644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cf226c7790 con 0x55cf22758d80 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cf226c4cc0 con 0x55cf227620f0 2024-03-24T06:47:12.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd707f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cf226c04a0 con 0x55cf22759a60 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd2400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd2400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48632/0 (socket says 172.21.15.22:48632) 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd2400640 1 -- 172.21.15.22:0/1068127441 learned_addr learned my addr 172.21.15.22:0/1068127441 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd1a00640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf22759200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.966+0000 7facd1000640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf227644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 -- 172.21.15.22:0/1068127441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 msgr2=0x55cf227644e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf227644e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facca000640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 0x7facb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 -- 172.21.15.22:0/1068127441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 msgr2=0x55cf22759200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf22759200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 -- 172.21.15.22:0/1068127441 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf22681c20 con 0x55cf22759a60 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd1a00640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf22759200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facca000640 1 -- 172.21.15.22:0/2358380445 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7facc00f0af0 con 0x7faccc076560 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.967+0000 7facd2400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 0x7facb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.968+0000 7facd2400640 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7facc40106c0 tx=0x7facc4010b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=459316f0e14300fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.968+0000 7facbaa00640 1 -- 172.21.15.22:0/1068127441 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc400ed30 con 0x55cf22759a60 2024-03-24T06:47:12.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.968+0000 7facbaa00640 1 -- 172.21.15.22:0/1068127441 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facc4012930 con 0x55cf22759a60 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.968+0000 7facbaa00640 1 -- 172.21.15.22:0/1068127441 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc401eb40 con 0x55cf22759a60 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- 172.21.15.22:0/1068127441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 msgr2=0x55cf227587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd2400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 0x7facb40a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7facc401f000 tx=0x7facc40117b0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7facc40106c0 tx=0x7facc4010b80 comp rx=0 tx=0).stop 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- 172.21.15.22:0/1068127441 shutdown_connections 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf227644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf22759200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 --2- 172.21.15.22:0/1068127441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf227587b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- 172.21.15.22:0/1068127441 >> 172.21.15.22:0/1068127441 conn(0x55cf2271fb30 msgr2=0x55cf227537d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- 172.21.15.22:0/1068127441 shutdown_connections 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- 172.21.15.22:0/1068127441 wait complete. 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.969+0000 7facd707f740 1 -- start start 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 Processor -- start 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cf22664f20 con 0x55cf22758d80 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cf22664bc0 con 0x55cf227620f0 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cf2242e1c0 con 0x55cf22759a60 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.970+0000 7facd707f740 10 client.0 _ll_register_callbacks cb 0x55cf223d0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd2400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48636/0 (socket says 172.21.15.22:48636) 2024-03-24T06:47:12.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 -- 172.21.15.22:0/814523711 learned_addr learned my addr 172.21.15.22:0/814523711 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 -- 172.21.15.22:0/814523711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 msgr2=0x55cf2275fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 -- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 msgr2=0x55cf2275f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 -- 172.21.15.22:0/814523711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc4010370 con 0x55cf22759a60 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd2400640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facd1000640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7facc0005ec0 tx=0x7facc00035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e74e55f472a4b593 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:12.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.971+0000 7facaf400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc00f0ea0 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7facc00f9070 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cf226b8160 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7facc00bccc0 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facd707f740 10 client.27103 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cf226beb20 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cf226b8160 con 0x55cf22759a60 2024-03-24T06:47:12.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.972+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7facc003c070 con 0x55cf22759a60 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 10 client.27103 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 20 client.27103 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf22772540 0x55cf22774960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 4 client.27103 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf223bf7e0 con 0x55cf22772540 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.973+0000 7facd1a00640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf22772540 0x55cf22774960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:12.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.974+0000 7facaf400640 10 client.27103 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.974+0000 7facd1a00640 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf22772540 0x55cf22774960 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b3195b99c0d262 server_cookie=99a3806993fb8422 in_seq=0 out_seq=0 2024-03-24T06:47:12.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.974+0000 7facaf400640 1 -- 172.21.15.22:0/814523711 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf223bf7e0 con 0x55cf22772540 2024-03-24T06:47:12.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.974+0000 7facaf400640 10 client.27103 handle_command_reply: tid=0 2024-03-24T06:47:12.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.974+0000 7facd707f740 1 client.27103 shutdown 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 msgr2=0x55cf2275f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7facc0005ec0 tx=0x7facc00035f0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 shutdown_connections 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf22772540 0x55cf22774960 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275f630 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 --2- 172.21.15.22:0/814523711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facaf400640 0 client.27103 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 >> 172.21.15.22:0/814523711 conn(0x55cf2271fb30 msgr2=0x55cf22755520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 shutdown_connections 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 1 -- 172.21.15.22:0/814523711 wait complete. 2024-03-24T06:47:12.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.975+0000 7facd707f740 20 client.27103 trim_cache size 0 max 16384 2024-03-24T06:47:12.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 Processor -- start 2024-03-24T06:47:12.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 -- start start 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2276da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cf22774fc0 con 0x55cf22758d80 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cf22759e40 con 0x55cf227620f0 2024-03-24T06:47:13.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd707f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cf22759700 con 0x55cf22759a60 2024-03-24T06:47:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48648/0 (socket says 172.21.15.22:48648) 2024-03-24T06:47:13.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd1000640 1 -- 172.21.15.22:0/3284782206 learned_addr learned my addr 172.21.15.22:0/3284782206 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:12.999+0000 7facd2400640 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2276da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1a00640 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 -- 172.21.15.22:0/3284782206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 msgr2=0x55cf2276da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2276da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 -- 172.21.15.22:0/3284782206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 msgr2=0x55cf2275e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 -- 172.21.15.22:0/3284782206 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2265a9a0 con 0x55cf22759a60 2024-03-24T06:47:13.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facd1000640 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7facc0006010 tx=0x7facc00c12f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78eb9522816b87c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facba000640 1 -- 172.21.15.22:0/3284782206 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0003e20 con 0x55cf22759a60 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facba000640 1 -- 172.21.15.22:0/3284782206 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facc00bccc0 con 0x55cf22759a60 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.000+0000 7facba000640 1 -- 172.21.15.22:0/3284782206 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc00c5490 con 0x55cf22759a60 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 -- 172.21.15.22:0/3284782206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 msgr2=0x55cf22753300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7facc0006010 tx=0x7facc00c12f0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 -- 172.21.15.22:0/3284782206 shutdown_connections 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2276da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf22753300 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 --2- 172.21.15.22:0/3284782206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2275e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 -- 172.21.15.22:0/3284782206 >> 172.21.15.22:0/3284782206 conn(0x55cf2271fb30 msgr2=0x55cf227554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 -- 172.21.15.22:0/3284782206 shutdown_connections 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.001+0000 7facd707f740 1 -- 172.21.15.22:0/3284782206 wait complete. 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 -- start start 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 Processor -- start 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2276a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55cf22762bd0 con 0x55cf22758d80 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cf22762d40 con 0x55cf227620f0 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cf22762f00 con 0x55cf22759a60 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.002+0000 7facd707f740 10 client.0 _ll_register_callbacks cb 0x7facc0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2276a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd2400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:48654/0 (socket says 172.21.15.22:48654) 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 -- 172.21.15.22:0/1752142116 learned_addr learned my addr 172.21.15.22:0/1752142116 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 -- 172.21.15.22:0/1752142116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 msgr2=0x55cf2275cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 -- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 msgr2=0x55cf2276a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2276a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 -- 172.21.15.22:0/1752142116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf226beb20 con 0x55cf22759a60 2024-03-24T06:47:13.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facd1000640 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7facc00b9b50 tx=0x7facc00035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa660939983b2bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facaf400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.003+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc00bd8d0 con 0x55cf22759a60 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9555+0+0 (secure 0 0 0) 0x7facc00f9070 con 0x55cf22759a60 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cf22753960 con 0x55cf22759a60 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7facc00bcae0 con 0x55cf22759a60 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 10 client.27109 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cf22680480 con 0x55cf22759a60 2024-03-24T06:47:13.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cf22753960 con 0x55cf22759a60 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7facc003c070 con 0x55cf22759a60 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 10 client.27109 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 10 client.27109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 20 client.27109 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf227d8cf0 0x55cf227db0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 4 client.27109 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:47:13.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7facc4012930 con 0x55cf227d8cf0 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.004+0000 7facd1a00640 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf227d8cf0 0x55cf227db0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facd1a00640 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf227d8cf0 0x55cf227db0e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d290818f50154a75 server_cookie=f7e32fe9e934680c in_seq=0 out_seq=0 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facaf400640 10 client.27109 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facaf400640 1 -- 172.21.15.22:0/1752142116 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7facc4012930 con 0x55cf227d8cf0 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facaf400640 10 client.27109 handle_command_reply: tid=0 2024-03-24T06:47:13.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facd707f740 1 client.27109 shutdown 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 msgr2=0x55cf2275c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.005+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7facc00b9b50 tx=0x7facc00035f0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 shutdown_connections 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55cf227d8cf0 0x55cf227db0e0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cf227620f0 0x55cf2275cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cf22759a60 0x55cf2275c450 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facaf400640 0 client.27109 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:13.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 --2- 172.21.15.22:0/1752142116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55cf22758d80 0x55cf2276a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 >> 172.21.15.22:0/1752142116 conn(0x55cf2271fb30 msgr2=0x55cf22751fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:13.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 shutdown_connections 2024-03-24T06:47:13.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 1 -- 172.21.15.22:0/1752142116 wait complete. 2024-03-24T06:47:13.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.006+0000 7facd707f740 20 client.27109 trim_cache size 0 max 16384 2024-03-24T06:47:13.015 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "59bd0f52-8898-47b4-a4a2-4686a4103c00": { 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "59bd0f52-8898-47b4-a4a2-4686a4103c00", 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:47:13.016 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:13.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 msgr2=0x7facb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:13.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 0x7facb40a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7facc401f000 tx=0x7facc40117b0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 msgr2=0x7faccc19e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 0x7faccc19e960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7facc0036040 tx=0x7facc002c6a0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 shutdown_connections 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7facb40a2230 0x7facb40a46f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faccc112550 0x7faccc1a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faccc076560 0x7faccc19e960 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 --2- 172.21.15.22:0/2358380445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faccc075c40 0x7faccc19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 >> 172.21.15.22:0/2358380445 conn(0x7faccc0ff360 msgr2=0x7faccc101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:13.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 shutdown_connections 2024-03-24T06:47:13.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:13.007+0000 7facd4400640 1 -- 172.21.15.22:0/2358380445 wait complete. 2024-03-24T06:47:13.032 INFO:tasks.cephfs.filesystem:scrub status for tag:59bd0f52-8898-47b4-a4a2-4686a4103c00 - {'path': '/', 'tag': '59bd0f52-8898-47b4-a4a2-4686a4103c00', 'options': 'recursive,force'} 2024-03-24T06:47:13.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:12 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:13.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:12 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:13.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:12 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:13.178 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:12 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:13.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:12 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:13.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:12 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:14.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:13 smithi155 ceph-mon[36262]: pgmap v222: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:47:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:13 smithi022 ceph-mon[32003]: pgmap v222: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:47:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:47:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:14.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:13 smithi143 ceph-mon[36520]: pgmap v222: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:47:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:16 smithi155 ceph-mon[36262]: pgmap v223: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:16 smithi155 ceph-mon[36262]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T06:47:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:16 smithi022 ceph-mon[32003]: pgmap v223: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:16 smithi022 ceph-mon[32003]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T06:47:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:16 smithi143 ceph-mon[36520]: pgmap v223: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:16.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:16 smithi143 ceph-mon[36520]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T06:47:18.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:18 smithi155 ceph-mon[36262]: pgmap v225: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 236 B/s wr, 12 op/s 2024-03-24T06:47:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:18 smithi022 ceph-mon[32003]: pgmap v225: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 236 B/s wr, 12 op/s 2024-03-24T06:47:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:18 smithi143 ceph-mon[36520]: pgmap v225: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 236 B/s wr, 12 op/s 2024-03-24T06:47:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:20 smithi022 ceph-mon[32003]: pgmap v226: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:20 smithi143 ceph-mon[36520]: pgmap v226: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:20 smithi155 ceph-mon[36262]: pgmap v226: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:22 smithi022 ceph-mon[32003]: pgmap v227: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:22 smithi143 ceph-mon[36520]: pgmap v227: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:22 smithi155 ceph-mon[36262]: pgmap v227: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:47:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:24 smithi022 ceph-mon[32003]: pgmap v228: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:24 smithi143 ceph-mon[36520]: pgmap v228: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:24 smithi155 ceph-mon[36262]: pgmap v228: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:26 smithi022 ceph-mon[32003]: pgmap v229: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:26 smithi143 ceph-mon[36520]: pgmap v229: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:26 smithi155 ceph-mon[36262]: pgmap v229: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:47:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:47:27.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:47:27.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:47:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:47:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:47:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:47:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:47:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:47:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:47:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: pgmap v230: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T06:47:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:47:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:28 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: pgmap v230: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T06:47:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:47:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:28 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: pgmap v230: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-24T06:47:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:47:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:28 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:29.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:29 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:29.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:29 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:29 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:30.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:30 smithi022 ceph-mon[32003]: pgmap v231: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:30 smithi143 ceph-mon[36520]: pgmap v231: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:30 smithi155 ceph-mon[36262]: pgmap v231: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:32.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:32 smithi022 ceph-mon[32003]: pgmap v232: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:32 smithi143 ceph-mon[36520]: pgmap v232: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:32 smithi155 ceph-mon[36262]: pgmap v232: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:34 smithi022 ceph-mon[32003]: pgmap v233: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:34.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:34 smithi143 ceph-mon[36520]: pgmap v233: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:34 smithi155 ceph-mon[36262]: pgmap v233: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:36.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:36 smithi022 ceph-mon[32003]: pgmap v234: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:36.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:36 smithi143 ceph-mon[36520]: pgmap v234: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:36 smithi155 ceph-mon[36262]: pgmap v234: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:38.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:38 smithi022 ceph-mon[32003]: pgmap v235: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:38.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:38 smithi143 ceph-mon[36520]: pgmap v235: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:38 smithi155 ceph-mon[36262]: pgmap v235: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:40.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:40 smithi022 ceph-mon[32003]: pgmap v236: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:40.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:40 smithi143 ceph-mon[36520]: pgmap v236: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:40 smithi155 ceph-mon[36262]: pgmap v236: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:42.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:42 smithi022 ceph-mon[32003]: pgmap v237: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:42 smithi143 ceph-mon[36520]: pgmap v237: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:42 smithi155 ceph-mon[36262]: pgmap v237: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:43.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:47:43.034 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:47:43.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.288+0000 7fd058e00640 1 -- 172.21.15.22:0/111760380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 msgr2=0x7fd054102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.288+0000 7fd058e00640 1 --2- 172.21.15.22:0/111760380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054102b20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd044009900 tx=0x7fd04402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 -- 172.21.15.22:0/111760380 shutdown_connections 2024-03-24T06:47:43.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 --2- 172.21.15.22:0/111760380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05410bb60 0x7fd05410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 --2- 172.21.15.22:0/111760380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd054103060 0x7fd05410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.290 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 --2- 172.21.15.22:0/111760380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054102b20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 -- 172.21.15.22:0/111760380 >> 172.21.15.22:0/111760380 conn(0x7fd0540fbf60 msgr2=0x7fd0540fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 -- 172.21.15.22:0/111760380 shutdown_connections 2024-03-24T06:47:43.291 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.289+0000 7fd058e00640 1 -- 172.21.15.22:0/111760380 wait complete. 2024-03-24T06:47:43.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 Processor -- start 2024-03-24T06:47:43.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 -- start start 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd05410bb60 0x7fd05419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd05419ed20 con 0x7fd05410bb60 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd05419ee90 con 0x7fd054103060 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd058e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd05419f050 con 0x7fd054102740 2024-03-24T06:47:43.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd053400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.290+0000 7fd052a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51408/0 (socket says 172.21.15.22:51408) 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 -- 172.21.15.22:0/3589998638 learned_addr learned my addr 172.21.15.22:0/3589998638 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd052a00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44450/0 (socket says 172.21.15.22:44450) 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd05410bb60 0x7fd05419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 -- 172.21.15.22:0/3589998638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 msgr2=0x7fd05419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 -- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd05410bb60 msgr2=0x7fd05419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd05410bb60 0x7fd05419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 -- 172.21.15.22:0/3589998638 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd044009560 con 0x7fd054102740 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd052a00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.291+0000 7fd053400640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd044000c00 tx=0x7fd04402c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.292+0000 7fd04be00640 1 -- 172.21.15.22:0/3589998638 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04402dbe0 con 0x7fd054102740 2024-03-24T06:47:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.292+0000 7fd04be00640 1 -- 172.21.15.22:0/3589998638 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd04402dd40 con 0x7fd054102740 2024-03-24T06:47:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.292+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05419f2d0 con 0x7fd054102740 2024-03-24T06:47:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.292+0000 7fd04be00640 1 -- 172.21.15.22:0/3589998638 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044003b20 con 0x7fd054102740 2024-03-24T06:47:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.292+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0541a6bb0 con 0x7fd054102740 2024-03-24T06:47:43.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.295+0000 7fd04be00640 1 -- 172.21.15.22:0/3589998638 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd044003c80 con 0x7fd054102740 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.295+0000 7fd05d16c740 1 Processor -- start 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.295+0000 7fd05d16c740 1 -- start start 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c1245463e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c1245516c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c1244b4790 con 0x55c124546c40 2024-03-24T06:47:43.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c1244b1cc0 con 0x55c12454f2d0 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd05d16c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c1244ad4a0 con 0x55c124545f60 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd053e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd053400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c1245463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd053e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60460/0 (socket says 172.21.15.22:60460) 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd053e00640 1 -- 172.21.15.22:0/332471671 learned_addr learned my addr 172.21.15.22:0/332471671 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.296+0000 7fd052a00640 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c1245516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 -- 172.21.15.22:0/332471671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 msgr2=0x55c1245463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c1245463e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 -- 172.21.15.22:0/332471671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 msgr2=0x55c1245516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c1245516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 -- 172.21.15.22:0/332471671 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c12446ec20 con 0x55c124546c40 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd04be00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 0x7fd0380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd04be00640 1 -- 172.21.15.22:0/3589998638 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd0440f09e0 con 0x7fd054102740 2024-03-24T06:47:43.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053400640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 0x7fd0380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd053e00640 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009f40 tx=0x7fd04c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e2cfc1e1e30e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd048c00640 1 -- 172.21.15.22:0/332471671 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c009a90 con 0x55c124546c40 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd048c00640 1 -- 172.21.15.22:0/332471671 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd04c015070 con 0x55c124546c40 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.297+0000 7fd048c00640 1 -- 172.21.15.22:0/332471671 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c011870 con 0x55c124546c40 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 -- 172.21.15.22:0/332471671 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 msgr2=0x55c124545990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd053400640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 0x7fd0380a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55c124547190 tx=0x7fd0440f64b0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009f40 tx=0x7fd04c010860 comp rx=0 tx=0).stop 2024-03-24T06:47:43.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 -- 172.21.15.22:0/332471671 shutdown_connections 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c1245516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c1245463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 --2- 172.21.15.22:0/332471671 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c124545990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 -- 172.21.15.22:0/332471671 >> 172.21.15.22:0/332471671 conn(0x55c12450cb30 msgr2=0x55c1245409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 -- 172.21.15.22:0/332471671 shutdown_connections 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.298+0000 7fd05d16c740 1 -- 172.21.15.22:0/332471671 wait complete. 2024-03-24T06:47:43.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.299+0000 7fd05d16c740 1 -- start start 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.299+0000 7fd05d16c740 1 Processor -- start 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c12454c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c12454cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c124451f20 con 0x55c124546c40 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c124451bc0 con 0x55c12454f2d0 2024-03-24T06:47:43.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c12421b1c0 con 0x55c124545f60 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd05d16c740 10 client.0 _ll_register_callbacks cb 0x55c1241bdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd053e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c12454c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60466/0 (socket says 172.21.15.22:60466) 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 -- 172.21.15.22:0/2810303351 learned_addr learned my addr 172.21.15.22:0/2810303351 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd053e00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c12454cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 -- 172.21.15.22:0/2810303351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 msgr2=0x55c12454c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c12454c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 -- 172.21.15.22:0/2810303351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 msgr2=0x55c12454cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c12454cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd052a00640 1 -- 172.21.15.22:0/2810303351 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04c009bf0 con 0x55c124546c40 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.300+0000 7fd053e00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c12454cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.301+0000 7fd052a00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd040012d20 tx=0x7fd04000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f640a38e426d363 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.301+0000 7fd03d600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:47:43.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.301+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040017070 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.301+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd040012110 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.301+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1244a5160 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd040004030 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 10 client.16734 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c1244abb20 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c1244a5160 con 0x55c124546c40 2024-03-24T06:47:43.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd040010430 con 0x55c124546c40 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 10 client.16734 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 20 client.16734 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c12455f980 0x55c124561da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 4 client.16734 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.302+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1241ac7e0 con 0x55c12455f980 2024-03-24T06:47:43.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.303+0000 7fd053e00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c12455f980 0x55c124561da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.303+0000 7fd053e00640 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c12455f980 0x55c124561da0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81bf976b0bdfd690 server_cookie=134b258407ebb999 in_seq=0 out_seq=0 2024-03-24T06:47:43.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.303+0000 7fd03d600640 10 client.16734 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:43.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd03d600640 1 -- 172.21.15.22:0/2810303351 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1241ac7e0 con 0x55c12455f980 2024-03-24T06:47:43.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd03d600640 10 client.16734 handle_command_reply: tid=0 2024-03-24T06:47:43.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 client.16734 shutdown 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 msgr2=0x55c12454c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd040012d20 tx=0x7fd04000f790 comp rx=0 tx=0).stop 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 shutdown_connections 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c12455f980 0x55c124561da0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c12454f2d0 0x55c12454cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124546c40 0x55c12454c7e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 --2- 172.21.15.22:0/2810303351 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c12454c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd03d600640 0 client.16734 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:43.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.304+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 >> 172.21.15.22:0/2810303351 conn(0x55c12450cb30 msgr2=0x55c124542700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.305+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 shutdown_connections 2024-03-24T06:47:43.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.305+0000 7fd05d16c740 1 -- 172.21.15.22:0/2810303351 wait complete. 2024-03-24T06:47:43.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.305+0000 7fd05d16c740 20 client.16734 trim_cache size 0 max 16384 2024-03-24T06:47:43.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.337+0000 7fd05d16c740 1 Processor -- start 2024-03-24T06:47:43.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.337+0000 7fd05d16c740 1 -- start start 2024-03-24T06:47:43.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.337+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124545f60 0x55c12455ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c12454f2d0 0x55c124567630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd05d16c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c12454c0c0 con 0x55c124545f60 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd05d16c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c1245419b0 con 0x55c124546c40 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd05d16c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c124562400 con 0x55c12454f2d0 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd053e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124545f60 0x55c12455ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd053e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c12454f2d0 0x55c124567630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44492/0 (socket says 172.21.15.22:44492) 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 -- 172.21.15.22:0/3144999731 learned_addr learned my addr 172.21.15.22:0/3144999731 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 -- 172.21.15.22:0/3144999731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c12454f2d0 msgr2=0x55c124567630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c12454f2d0 0x55c124567630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 -- 172.21.15.22:0/3144999731 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124545f60 msgr2=0x55c12455ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124545f60 0x55c12455ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.338+0000 7fd052a00640 1 -- 172.21.15.22:0/3144999731 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1244479a0 con 0x55c124546c40 2024-03-24T06:47:43.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.339+0000 7fd052a00640 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd040009c70 tx=0x7fd0400057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac3317fdec5a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.339+0000 7fd03fe00640 1 -- 172.21.15.22:0/3144999731 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040017040 con 0x55c124546c40 2024-03-24T06:47:43.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.339+0000 7fd03fe00640 1 -- 172.21.15.22:0/3144999731 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0400055b0 con 0x55c124546c40 2024-03-24T06:47:43.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.339+0000 7fd03fe00640 1 -- 172.21.15.22:0/3144999731 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040010a30 con 0x55c124546c40 2024-03-24T06:47:43.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- 172.21.15.22:0/3144999731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 msgr2=0x55c12453fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd040009c70 tx=0x7fd0400057f0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- 172.21.15.22:0/3144999731 shutdown_connections 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c12454f2d0 0x55c124567630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c12453fa50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 --2- 172.21.15.22:0/3144999731 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c124545f60 0x55c12455ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- 172.21.15.22:0/3144999731 >> 172.21.15.22:0/3144999731 conn(0x55c12450cb30 msgr2=0x55c12453f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- 172.21.15.22:0/3144999731 shutdown_connections 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- 172.21.15.22:0/3144999731 wait complete. 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.340+0000 7fd05d16c740 1 -- start start 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 Processor -- start 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c124549560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 0x55c1245c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55c1245c51a0 con 0x55c12454f2d0 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c1245c52e0 con 0x55c124546c40 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c1245c5420 con 0x55c124545f60 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd05d16c740 10 client.0 _ll_register_callbacks cb 0x7fd040009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.341+0000 7fd052a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd053e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c124549560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44500/0 (socket says 172.21.15.22:44500) 2024-03-24T06:47:43.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 -- 172.21.15.22:0/1624164599 learned_addr learned my addr 172.21.15.22:0/1624164599 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd053e00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 0x55c1245c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 -- 172.21.15.22:0/1624164599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 msgr2=0x55c124549560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c124549560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 -- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 msgr2=0x55c1245c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 0x55c1245c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 -- 172.21.15.22:0/1624164599 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd040021040 con 0x55c124546c40 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd053e00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 0x55c1245c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd052a00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd040012850 tx=0x7fd040009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652c13544ca3424d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.342+0000 7fd03d600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040017040 con 0x55c124546c40 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd04c015070 con 0x55c124546c40 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fd0400050d0 con 0x55c124546c40 2024-03-24T06:47:43.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd040005290 con 0x55c124546c40 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 10 client.27107 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c12446d480 con 0x55c124546c40 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd04c015070 con 0x55c124546c40 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd040003b80 con 0x55c124546c40 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 10 client.27107 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 10 client.27107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:43.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 20 client.27107 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c1245c67e0 0x55c1245c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 4 client.27107 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.343+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1241e9720 con 0x55c1245c67e0 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.344+0000 7fd053e00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c1245c67e0 0x55c1245c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.344+0000 7fd053e00640 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c1245c67e0 0x55c1245c8c00 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92c98c4e91111c4d server_cookie=68335eede86eeded in_seq=0 out_seq=0 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.344+0000 7fd03d600640 10 client.27107 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd03d600640 1 -- 172.21.15.22:0/1624164599 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1241e9720 con 0x55c1245c67e0 2024-03-24T06:47:43.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd03d600640 10 client.27107 handle_command_reply: tid=0 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 client.27107 shutdown 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 msgr2=0x55c124549aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd040012850 tx=0x7fd040009240 comp rx=0 tx=0).stop 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 shutdown_connections 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55c12454f2d0 0x55c1245c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55c1245c67e0 0x55c1245c8c00 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c124546c40 0x55c124549aa0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 --2- 172.21.15.22:0/1624164599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c124545f60 0x55c124549560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 >> 172.21.15.22:0/1624164599 conn(0x55c12450cb30 msgr2=0x55c124541cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd03d600640 0 client.27107 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 shutdown_connections 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 1 -- 172.21.15.22:0/1624164599 wait complete. 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.345+0000 7fd05d16c740 20 client.27107 trim_cache size 0 max 16384 2024-03-24T06:47:43.355 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:47:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:47:43.356 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 msgr2=0x7fd0380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 0x7fd0380a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55c124547190 tx=0x7fd0440f64b0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 msgr2=0x7fd054199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd044000c00 tx=0x7fd04402c6d0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 shutdown_connections 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0380a2230 0x7fd0380a46f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd05410bb60 0x7fd05419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd054103060 0x7fd05419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 --2- 172.21.15.22:0/3589998638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd054102740 0x7fd054199e60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 >> 172.21.15.22:0/3589998638 conn(0x7fd0540fbf60 msgr2=0x7fd0540fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 shutdown_connections 2024-03-24T06:47:43.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.346+0000 7fd058e00640 1 -- 172.21.15.22:0/3589998638 wait complete. 2024-03-24T06:47:43.367 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:47:43.367 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:47:43.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 -- 172.21.15.22:0/3700360961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 msgr2=0x7fa4b010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 --2- 172.21.15.22:0/3700360961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b010e570 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009930 tx=0x7fa4a802bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:43.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 -- 172.21.15.22:0/3700360961 shutdown_connections 2024-03-24T06:47:43.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 --2- 172.21.15.22:0/3700360961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b010e570 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 --2- 172.21.15.22:0/3700360961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 --2- 172.21.15.22:0/3700360961 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 0x7fa4b0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 -- 172.21.15.22:0/3700360961 >> 172.21.15.22:0/3700360961 conn(0x7fa4b00776e0 msgr2=0x7fa4b0077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 -- 172.21.15.22:0/3700360961 shutdown_connections 2024-03-24T06:47:43.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.558+0000 7fa4b8200640 1 -- 172.21.15.22:0/3700360961 wait complete. 2024-03-24T06:47:43.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.559+0000 7fa4b8200640 1 Processor -- start 2024-03-24T06:47:43.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 -- start start 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 0x7fa4b0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b01982c0 con 0x7fa4b01076b0 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b0198430 con 0x7fa4b0069dc0 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b8200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b01985f0 con 0x7fa4b006a780 2024-03-24T06:47:43.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44524/0 (socket says 172.21.15.22:44524) 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b5800640 1 -- 172.21.15.22:0/1327734513 learned_addr learned my addr 172.21.15.22:0/1327734513 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b4e00640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.560+0000 7fa4b6200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 0x7fa4b0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 -- 172.21.15.22:0/1327734513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 msgr2=0x7fa4b019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 -- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 msgr2=0x7fa4b0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 0x7fa4b0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 -- 172.21.15.22:0/1327734513 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a8009560 con 0x7fa4b0069dc0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b5800640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400db80 tx=0x7fa4a400b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4003b70 con 0x7fa4b0069dc0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4a400de80 con 0x7fa4b0069dc0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b01988d0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40106c0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.561+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b01a49f0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.563+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b00695b0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.564+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa4a4004020 con 0x7fa4b0069dc0 2024-03-24T06:47:43.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.566+0000 7fa4ae000640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:43.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.566+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fa4a40ce3f0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.566+0000 7fa4b4e00640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:43.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.567+0000 7fa4b4e00640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 0x7fa4980a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0199310 tx=0x7fa49c005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:43.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.570+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa4a4096c70 con 0x7fa4b0069dc0 2024-03-24T06:47:43.816 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.816+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4b0102de0 con 0x7fa4b0069dc0 2024-03-24T06:47:43.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.817+0000 7fa4ae000640 1 -- 172.21.15.22:0/1327734513 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7fa4a400be60 con 0x7fa4b0069dc0 2024-03-24T06:47:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:47:43.818 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:47:43.818 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:47:43.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 msgr2=0x7fa4980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.820 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 0x7fa4980a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0199310 tx=0x7fa49c005f90 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 msgr2=0x7fa4b019da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400db80 tx=0x7fa4a400b630 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 shutdown_connections 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4b01076b0 0x7fa4b0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4b006a780 0x7fa4b019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 --2- 172.21.15.22:0/1327734513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4b0069dc0 0x7fa4b019da70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 >> 172.21.15.22:0/1327734513 conn(0x7fa4b00776e0 msgr2=0x7fa4b00fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:43.821 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 shutdown_connections 2024-03-24T06:47:43.822 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:43.820+0000 7fa4b8200640 1 -- 172.21.15.22:0/1327734513 wait complete. 2024-03-24T06:47:43.839 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 -- 172.21.15.22:0/4106031411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4107690 msgr2=0x7f01e4107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4106031411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4107690 0x7f01e4107a70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009900 tx=0x7f01d802bc40 comp rx=0 tx=0).stop 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 -- 172.21.15.22:0/4106031411 shutdown_connections 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4106031411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4102040 0x7f01e410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4106031411 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4101680 0x7f01e4101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4106031411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4107690 0x7f01e4107a70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 -- 172.21.15.22:0/4106031411 >> 172.21.15.22:0/4106031411 conn(0x7f01e40fd170 msgr2=0x7f01e40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 -- 172.21.15.22:0/4106031411 shutdown_connections 2024-03-24T06:47:44.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.051+0000 7f01ebc00640 1 -- 172.21.15.22:0/4106031411 wait complete. 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.052+0000 7f01ebc00640 1 Processor -- start 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.052+0000 7f01ebc00640 1 -- start start 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 0x7f01e419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 0x7f01e419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f01e419ee30 con 0x7f01e4102040 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f01e419efa0 con 0x7f01e4101680 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01ebc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f01e419f160 con 0x7f01e4107690 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01e9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 0x7f01e419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01e9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44544/0 (socket says 172.21.15.22:44544) 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01e9200640 1 -- 172.21.15.22:0/4117804345 learned_addr learned my addr 172.21.15.22:0/4117804345 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.053+0000 7f01e9c00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 0x7f01e419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 -- 172.21.15.22:0/4117804345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 msgr2=0x7f01e419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 0x7f01e419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 -- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 msgr2=0x7f01e419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 0x7f01e419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 -- 172.21.15.22:0/4117804345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d8009560 con 0x7f01e4101680 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9c00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 0x7f01e419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e3e00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 0x7f01e419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e9200640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e1600640 1 -- 172.21.15.22:0/4117804345 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8004500 con 0x7f01e4101680 2024-03-24T06:47:44.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01e1600640 1 -- 172.21.15.22:0/4117804345 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01d80052d0 con 0x7f01e4101680 2024-03-24T06:47:44.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.054+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e419f3e0 con 0x7f01e4101680 2024-03-24T06:47:44.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.055+0000 7f01e1600640 1 -- 172.21.15.22:0/4117804345 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8003e20 con 0x7f01e4101680 2024-03-24T06:47:44.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.055+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e41a6cc0 con 0x7f01e4101680 2024-03-24T06:47:44.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.058+0000 7f01e1600640 1 -- 172.21.15.22:0/4117804345 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f01d80035f0 con 0x7f01e4101680 2024-03-24T06:47:44.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.059+0000 7f01e1600640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 0x7f01cc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.059+0000 7f01e1600640 1 -- 172.21.15.22:0/4117804345 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f01d80f0b60 con 0x7f01e4101680 2024-03-24T06:47:44.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.059+0000 7f01e3e00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 0x7f01cc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.059+0000 7f01ee9ae740 1 Processor -- start 2024-03-24T06:47:44.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.059+0000 7f01ee9ae740 1 -- start start 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 0x5601f8e00b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601f8d636e0 con 0x5601f8df5ba0 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601f8d60c10 con 0x5601f8df5280 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5601f8d5c3f0 con 0x5601f8dfe780 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e9c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 0x5601f8e00b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e9200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44560/0 (socket says 172.21.15.22:44560) 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e9200640 1 -- 172.21.15.22:0/3039036315 learned_addr learned my addr 172.21.15.22:0/3039036315 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfe240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60512/0 (socket says 172.21.15.22:60512) 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.060+0000 7f01e3e00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 0x7f01cc0a4770 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f01e410e9e0 tx=0x7f01d000b390 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 -- 172.21.15.22:0/3039036315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 msgr2=0x5601f8e00b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 0x5601f8e00b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 -- 172.21.15.22:0/3039036315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 msgr2=0x5601f8dfe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfe240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 -- 172.21.15.22:0/3039036315 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f8d1de40 con 0x5601f8df5280 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9c00640 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 0x5601f8e00b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01e9200640 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f01d80bd1e0 tx=0x7f01d8003920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c6d71df78ea10fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01d6000640 1 -- 172.21.15.22:0/3039036315 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80c1170 con 0x5601f8df5280 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01d6000640 1 -- 172.21.15.22:0/3039036315 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01d80bd250 con 0x5601f8df5280 2024-03-24T06:47:44.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.061+0000 7f01d6000640 1 -- 172.21.15.22:0/3039036315 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80c5560 con 0x5601f8df5280 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- 172.21.15.22:0/3039036315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 msgr2=0x5601f8df5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f01d80bd1e0 tx=0x7f01d8003920 comp rx=0 tx=0).stop 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- 172.21.15.22:0/3039036315 shutdown_connections 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8dfe780 0x5601f8e00b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/3039036315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8df5660 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- 172.21.15.22:0/3039036315 >> 172.21.15.22:0/3039036315 conn(0x5601f8def9b0 msgr2=0x5601f8defd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- 172.21.15.22:0/3039036315 shutdown_connections 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- 172.21.15.22:0/3039036315 wait complete. 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.062+0000 7f01ee9ae740 1 -- start start 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 Processor -- start 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 0x5601f8dfb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601f8d01060 con 0x5601f8df5ba0 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601f8d00d00 con 0x5601f8dfe780 2024-03-24T06:47:44.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5601f8aca1c0 con 0x5601f8df5280 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01ee9ae740 10 client.0 _ll_register_callbacks cb 0x5601f8a6cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01e9c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.063+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 0x5601f8dfb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44566/0 (socket says 172.21.15.22:44566) 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 -- 172.21.15.22:0/4012135842 learned_addr learned my addr 172.21.15.22:0/4012135842 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 -- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 msgr2=0x5601f8dfb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 0x5601f8dfb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 -- 172.21.15.22:0/4012135842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 msgr2=0x5601f8dfbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 -- 172.21.15.22:0/4012135842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e41a6cc0 con 0x5601f8dfe780 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9200640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 0x5601f8dfb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01e9c00640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f01dc013df0 tx=0x7f01dc0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=676e9c6c00062d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01caa00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.064+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc012450 con 0x5601f8dfe780 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f01dc0125b0 con 0x5601f8dfe780 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5601f8a5b7e0 con 0x5601f8dfe780 2024-03-24T06:47:44.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01dc0127b0 con 0x5601f8dfe780 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 10 client.27125 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5601f8d5aa70 con 0x5601f8dfe780 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5601f8a5b7e0 con 0x5601f8dfe780 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f01dc01ca80 con 0x5601f8dfe780 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 10 client.27125 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 10 client.27125 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 20 client.27125 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e0eb60 0x5601f8e10f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 4 client.27125 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:44.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.065+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601f8a98720 con 0x5601f8e0eb60 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.066+0000 7f01e9200640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e0eb60 0x5601f8e10f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.066+0000 7f01e9200640 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e0eb60 0x5601f8e10f80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27060746dd77d934 server_cookie=1ee2b96e726d86c6 in_seq=0 out_seq=0 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.066+0000 7f01caa00640 10 client.27125 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.067+0000 7f01caa00640 1 -- 172.21.15.22:0/4012135842 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601f8a98720 con 0x5601f8e0eb60 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.067+0000 7f01caa00640 10 client.27125 handle_command_reply: tid=0 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.067+0000 7f01ee9ae740 1 client.27125 shutdown 2024-03-24T06:47:44.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 msgr2=0x5601f8dfc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f01dc013df0 tx=0x7f01dc0117b0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 shutdown_connections 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e0eb60 0x5601f8e10f80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8dfc2c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01caa00640 0 client.27125 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5ba0 0x5601f8dfbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/4012135842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5280 0x5601f8dfb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 >> 172.21.15.22:0/4012135842 conn(0x5601f8def9b0 msgr2=0x5601f8df1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 shutdown_connections 2024-03-24T06:47:44.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 1 -- 172.21.15.22:0/4012135842 wait complete. 2024-03-24T06:47:44.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.068+0000 7f01ee9ae740 20 client.27125 trim_cache size 0 max 16384 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 Processor -- start 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 -- start start 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8e09dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8dfe780 0x5601f8e16850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601f8df0400 con 0x5601f8dfe780 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601f8a98750 con 0x5601f8df5280 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5601f8df4370 con 0x5601f8df5ba0 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51520/0 (socket says 172.21.15.22:51520) 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.083+0000 7f01e3e00640 1 -- 172.21.15.22:0/2685580976 learned_addr learned my addr 172.21.15.22:0/2685580976 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e9200640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8e09dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e9c00640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8dfe780 0x5601f8e16850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 -- 172.21.15.22:0/2685580976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 msgr2=0x5601f8e09dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8e09dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 -- 172.21.15.22:0/2685580976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8dfe780 msgr2=0x5601f8e16850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8dfe780 0x5601f8e16850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 -- 172.21.15.22:0/2685580976 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f8cf6b00 con 0x5601f8df5ba0 2024-03-24T06:47:44.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e9200640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8e09dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01e3e00640 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f01d000bfd0 tx=0x7f01d0013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=823c985f277ef998 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.084+0000 7f01d5600640 1 -- 172.21.15.22:0/2685580976 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0024070 con 0x5601f8df5ba0 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01d5600640 1 -- 172.21.15.22:0/2685580976 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01d0015070 con 0x5601f8df5ba0 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01d5600640 1 -- 172.21.15.22:0/2685580976 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00049c0 con 0x5601f8df5ba0 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 -- 172.21.15.22:0/2685580976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 msgr2=0x5601f8df14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f01d000bfd0 tx=0x7f01d0013000 comp rx=0 tx=0).stop 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 -- 172.21.15.22:0/2685580976 shutdown_connections 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8dfe780 0x5601f8e16850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df14f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/2685580976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8df5280 0x5601f8e09dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.085+0000 7f01ee9ae740 1 -- 172.21.15.22:0/2685580976 >> 172.21.15.22:0/2685580976 conn(0x5601f8def9b0 msgr2=0x5601f8df0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.086+0000 7f01ee9ae740 1 -- 172.21.15.22:0/2685580976 shutdown_connections 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.086+0000 7f01ee9ae740 1 -- 172.21.15.22:0/2685580976 wait complete. 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.086+0000 7f01ee9ae740 1 -- start start 2024-03-24T06:47:44.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.086+0000 7f01ee9ae740 1 Processor -- start 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 0x5601f8df8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8df8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601f8e73e80 con 0x5601f8df5280 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601f8e73fc0 con 0x5601f8dfe780 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5601f8e74100 con 0x5601f8df5ba0 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01ee9ae740 10 client.0 _ll_register_callbacks cb 0x7f01dc013a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01e3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51536/0 (socket says 172.21.15.22:51536) 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01e3e00640 1 -- 172.21.15.22:0/252758534 learned_addr learned my addr 172.21.15.22:0/252758534 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01e9200640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 0x5601f8df8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.087+0000 7f01e9c00640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8df8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 -- 172.21.15.22:0/252758534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 msgr2=0x5601f8df8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8df8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 -- 172.21.15.22:0/252758534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 msgr2=0x5601f8df8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 0x5601f8df8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 -- 172.21.15.22:0/252758534 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f8d1de40 con 0x5601f8df5ba0 2024-03-24T06:47:44.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e9c00640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8df8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e9200640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 0x5601f8df8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01e3e00640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f01d0011010 tx=0x7f01d001fb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dee28b8e5afc703 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.088+0000 7f01caa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0024030 con 0x5601f8df5ba0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01d80bd250 con 0x5601f8df5ba0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f01d0025070 con 0x5601f8df5ba0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01d002e070 con 0x5601f8df5ba0 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01ee9ae740 10 client.27151 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x7f01d00116d0 con 0x5601f8df5ba0 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01d80bd250 con 0x5601f8df5ba0 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.089+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f01d001a040 con 0x5601f8df5ba0 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 10 client.27151 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 10 client.27151 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 20 client.27151 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e75930 0x5601f8e77d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 4 client.27151 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601f8ae7890 con 0x5601f8e75930 2024-03-24T06:47:44.095 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01e9200640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e75930 0x5601f8e77d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01e9200640 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e75930 0x5601f8e77d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a4ab8f2563ef0d0 server_cookie=c8751ffde13d9788 in_seq=0 out_seq=0 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.090+0000 7f01caa00640 10 client.27151 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01caa00640 1 -- 172.21.15.22:0/252758534 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601f8ae7890 con 0x5601f8e75930 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01caa00640 10 client.27151 handle_command_reply: tid=0 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 client.27151 shutdown 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 msgr2=0x5601f8df8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f01d0011010 tx=0x7f01d001fb60 comp rx=0 tx=0).stop 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 shutdown_connections 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5601f8e75930 0x5601f8e77d50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601f8dfe780 0x5601f8df8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601f8df5ba0 0x5601f8df8970 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01caa00640 0 client.27151 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 --2- 172.21.15.22:0/252758534 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601f8df5280 0x5601f8df8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 >> 172.21.15.22:0/252758534 conn(0x5601f8def9b0 msgr2=0x5601f8deea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 shutdown_connections 2024-03-24T06:47:44.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 1 -- 172.21.15.22:0/252758534 wait complete. 2024-03-24T06:47:44.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.091+0000 7f01ee9ae740 20 client.27151 trim_cache size 0 max 16384 2024-03-24T06:47:44.099 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 msgr2=0x7f01cc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 0x7f01cc0a4770 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f01e410e9e0 tx=0x7f01d000b390 comp rx=0 tx=0).stop 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 msgr2=0x7f01e4199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f01d8000c00 tx=0x7f01d8004040 comp rx=0 tx=0).stop 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 shutdown_connections 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f01cc0a22b0 0x7f01cc0a4770 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e4107690 0x7f01e419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f01e4102040 0x7f01e419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 --2- 172.21.15.22:0/4117804345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01e4101680 0x7f01e4199f70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 >> 172.21.15.22:0/4117804345 conn(0x7f01e40fd170 msgr2=0x7f01e40fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 shutdown_connections 2024-03-24T06:47:44.101 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.092+0000 7f01ebc00640 1 -- 172.21.15.22:0/4117804345 wait complete. 2024-03-24T06:47:44.113 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:47:44.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 -- 172.21.15.22:0/3113561521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c107660 msgr2=0x7f149c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 --2- 172.21.15.22:0/3113561521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c107660 0x7f149c107a40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1490009930 tx=0x7f149002bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 -- 172.21.15.22:0/3113561521 shutdown_connections 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 --2- 172.21.15.22:0/3113561521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 0x7f149c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 --2- 172.21.15.22:0/3113561521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c1015a0 0x7f149c101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 --2- 172.21.15.22:0/3113561521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c107660 0x7f149c107a40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 -- 172.21.15.22:0/3113561521 >> 172.21.15.22:0/3113561521 conn(0x7f149c0fd0b0 msgr2=0x7f149c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 -- 172.21.15.22:0/3113561521 shutdown_connections 2024-03-24T06:47:44.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.311+0000 7f14a1e00640 1 -- 172.21.15.22:0/3113561521 wait complete. 2024-03-24T06:47:44.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.312+0000 7f14a1e00640 1 Processor -- start 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.312+0000 7f14a1e00640 1 -- start start 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.312+0000 7f14a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 0x7f149c073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c107660 0x7f149c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f149c1a5f40 con 0x7f149c101f60 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f149c1a60b0 con 0x7f149c1015a0 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f149c1a6270 con 0x7f149c107660 2024-03-24T06:47:44.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44622/0 (socket says 172.21.15.22:44622) 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 -- 172.21.15.22:0/2586472552 learned_addr learned my addr 172.21.15.22:0/2586472552 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f149be00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 0x7f149c073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a1400640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c107660 0x7f149c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 -- 172.21.15.22:0/2586472552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c107660 msgr2=0x7f149c074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c107660 0x7f149c074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 -- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 msgr2=0x7f149c073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 0x7f149c073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.313+0000 7f14a0a00640 1 -- 172.21.15.22:0/2586472552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1490009560 con 0x7f149c1015a0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f14a0a00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1490000c00 tx=0x7f149002cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f1499600640 1 -- 172.21.15.22:0/2586472552 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149002dea0 con 0x7f149c1015a0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f1499600640 1 -- 172.21.15.22:0/2586472552 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1490004350 con 0x7f149c1015a0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f1499600640 1 -- 172.21.15.22:0/2586472552 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1490003b20 con 0x7f149c1015a0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149c1a64f0 con 0x7f149c1015a0 2024-03-24T06:47:44.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.314+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f149c1a69b0 con 0x7f149c1015a0 2024-03-24T06:47:44.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.318+0000 7f14a61aa740 1 Processor -- start 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 -- start start 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f1499600640 1 -- 172.21.15.22:0/2586472552 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1490005460 con 0x7f149c1015a0 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41d8950 0x5628c41dad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628c413d6e0 con 0x5628c41d8950 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628c413ac10 con 0x5628c41cf450 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a61aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628c41363f0 con 0x5628c41cfd70 2024-03-24T06:47:44.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a0a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.319+0000 7f14a1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44638/0 (socket says 172.21.15.22:44638) 2024-03-24T06:47:44.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 -- 172.21.15.22:0/113751015 learned_addr learned my addr 172.21.15.22:0/113751015 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f149be00640 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41d8950 0x5628c41dad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 -- 172.21.15.22:0/113751015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 msgr2=0x5628c41d8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 -- 172.21.15.22:0/113751015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41d8950 msgr2=0x5628c41dad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41d8950 0x5628c41dad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 -- 172.21.15.22:0/113751015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628c40f7e40 con 0x5628c41cf450 2024-03-24T06:47:44.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.320+0000 7f14a1400640 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1494009f90 tx=0x7f1494010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67b8ae737ce09ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f148dc00640 1 -- 172.21.15.22:0/113751015 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494011700 con 0x5628c41cf450 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f148dc00640 1 -- 172.21.15.22:0/113751015 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1494015070 con 0x5628c41cf450 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f148dc00640 1 -- 172.21.15.22:0/113751015 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1494019cd0 con 0x5628c41cf450 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f14a61aa740 1 -- 172.21.15.22:0/113751015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 msgr2=0x5628c41cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f1499600640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 0x7f14840a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.321+0000 7f14a61aa740 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1494009f90 tx=0x7f1494010860 comp rx=0 tx=0).stop 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 -- 172.21.15.22:0/113751015 shutdown_connections 2024-03-24T06:47:44.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41d8950 0x5628c41dad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f1499600640 1 -- 172.21.15.22:0/2586472552 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f14900f08a0 con 0x7f149c1015a0 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 --2- 172.21.15.22:0/113751015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41cf450 0x5628c41cf830 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 -- 172.21.15.22:0/113751015 >> 172.21.15.22:0/113751015 conn(0x5628c41c9b80 msgr2=0x5628c41c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 -- 172.21.15.22:0/113751015 shutdown_connections 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a0a00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 0x7f14840a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 -- 172.21.15.22:0/113751015 wait complete. 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.322+0000 7f14a61aa740 1 -- start start 2024-03-24T06:47:44.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a0a00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 0x7f14840a47a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14900ede00 tx=0x7f14900f6550 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 Processor -- start 2024-03-24T06:47:44.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41d6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628c40db060 con 0x5628c41cf450 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628c40dad00 con 0x5628c41d8950 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628c3ea41c0 con 0x5628c41cfd70 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f14a61aa740 10 client.0 _ll_register_callbacks cb 0x5628c3e46be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.323+0000 7f149be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51584/0 (socket says 172.21.15.22:51584) 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 -- 172.21.15.22:0/1474693884 learned_addr learned my addr 172.21.15.22:0/1474693884 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f14a1400640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 -- 172.21.15.22:0/1474693884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 msgr2=0x5628c41d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41d6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 -- 172.21.15.22:0/1474693884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 msgr2=0x5628c41d59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 -- 172.21.15.22:0/1474693884 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1494009bf0 con 0x5628c41cfd70 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f149be00640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1488012700 tx=0x7f1488012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9eed75df7432dad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.324+0000 7f1482000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488017070 con 0x5628c41cfd70 2024-03-24T06:47:44.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5628c3e357e0 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f148800f8c0 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f148800fac0 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f14a61aa740 10 client.27163 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5628c4134a70 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.325+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5628c3e357e0 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1488005140 con 0x5628c41cfd70 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 10 client.27163 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 10 client.27163 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 20 client.27163 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c41e8e30 0x5628c41eb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 4 client.27163 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628c3e72720 con 0x5628c41e8e30 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.326+0000 7f14a1400640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c41e8e30 0x5628c41eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.327+0000 7f14a1400640 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c41e8e30 0x5628c41eb270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f12f96e611fd1a04 server_cookie=101484142584bbed in_seq=0 out_seq=0 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.327+0000 7f1482000640 10 client.27163 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.327+0000 7f1482000640 1 -- 172.21.15.22:0/1474693884 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628c3e72720 con 0x5628c41e8e30 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.327+0000 7f1482000640 10 client.27163 handle_command_reply: tid=0 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.327+0000 7f14a61aa740 1 client.27163 shutdown 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 msgr2=0x5628c41d5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1488012700 tx=0x7f1488012bc0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 shutdown_connections 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c41e8e30 0x5628c41eb270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41d6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f1482000640 0 client.27163 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41d5f20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1474693884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 >> 172.21.15.22:0/1474693884 conn(0x5628c41c9b80 msgr2=0x5628c41cb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 shutdown_connections 2024-03-24T06:47:44.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 1 -- 172.21.15.22:0/1474693884 wait complete. 2024-03-24T06:47:44.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.328+0000 7f14a61aa740 20 client.27163 trim_cache size 0 max 16384 2024-03-24T06:47:44.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.350+0000 7f14a61aa740 1 Processor -- start 2024-03-24T06:47:44.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.350+0000 7f14a61aa740 1 -- start start 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41e41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628c41c54b0 con 0x5628c41cf450 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628c41cebf0 con 0x5628c41d8950 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a61aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628c41cee90 con 0x5628c41cfd70 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60586/0 (socket says 172.21.15.22:60586) 2024-03-24T06:47:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a1400640 1 -- 172.21.15.22:0/3871508720 learned_addr learned my addr 172.21.15.22:0/3871508720 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f149be00640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41e41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.351+0000 7f14a1400640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 -- 172.21.15.22:0/3871508720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 msgr2=0x5628c41f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 -- 172.21.15.22:0/3871508720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 msgr2=0x5628c41d8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 -- 172.21.15.22:0/3871508720 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628c40d0b00 con 0x5628c41cfd70 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f14a1400640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f149be00640 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41e41e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1488005690 tx=0x7f1488009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b11a23b391e69a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f148d200640 1 -- 172.21.15.22:0/3871508720 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488017070 con 0x5628c41cfd70 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f148d200640 1 -- 172.21.15.22:0/3871508720 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1488018040 con 0x5628c41cfd70 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.352+0000 7f148d200640 1 -- 172.21.15.22:0/3871508720 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148801ca50 con 0x5628c41cfd70 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- 172.21.15.22:0/3871508720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 msgr2=0x5628c41e41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41e41e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1488005690 tx=0x7f1488009640 comp rx=0 tx=0).stop 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- 172.21.15.22:0/3871508720 shutdown_connections 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41e41e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 --2- 172.21.15.22:0/3871508720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41d8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- 172.21.15.22:0/3871508720 >> 172.21.15.22:0/3871508720 conn(0x5628c41c9b80 msgr2=0x5628c41d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- 172.21.15.22:0/3871508720 shutdown_connections 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- 172.21.15.22:0/3871508720 wait complete. 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.353+0000 7f14a61aa740 1 -- start start 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 Processor -- start 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41dc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5628c41dcdf0 con 0x5628c41cf450 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628c424e1b0 con 0x5628c41d8950 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628c424e2f0 con 0x5628c41cfd70 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a61aa740 10 client.0 _ll_register_callbacks cb 0x7f1488009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f149be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f149be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51588/0 (socket says 172.21.15.22:51588) 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f149be00640 1 -- 172.21.15.22:0/1576380548 learned_addr learned my addr 172.21.15.22:0/1576380548 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.354+0000 7f14a1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f14a1400640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41dc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 -- 172.21.15.22:0/1576380548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 msgr2=0x5628c41dc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41dc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 -- 172.21.15.22:0/1576380548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 msgr2=0x5628c41e0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41e0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 -- 172.21.15.22:0/1576380548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628c4134a70 con 0x5628c41cfd70 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f14a1400640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41dc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f149be00640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1488005690 tx=0x7f148800fa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3cc924327fd55fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f1482000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.355+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488017040 con 0x5628c41cfd70 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f148800fe40 con 0x5628c41cfd70 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1494015070 con 0x5628c41cfd70 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f148801c5b0 con 0x5628c41cfd70 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 10 client.27169 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5628c40f66a0 con 0x5628c41cfd70 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1494015070 con 0x5628c41cfd70 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f1488032030 con 0x5628c41cfd70 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 10 client.27169 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 10 client.27169 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 20 client.27169 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c424f680 0x5628c4251aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 4 client.27169 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.356+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628c3ec1890 con 0x5628c424f680 2024-03-24T06:47:44.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a1400640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c424f680 0x5628c4251aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a1400640 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c424f680 0x5628c4251aa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5e193b4c9ca4f261 server_cookie=d7535ef567b480d3 in_seq=0 out_seq=0 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f1482000640 10 client.27169 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f1482000640 1 -- 172.21.15.22:0/1576380548 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628c3ec1890 con 0x5628c424f680 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f1482000640 10 client.27169 handle_command_reply: tid=0 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 client.27169 shutdown 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 msgr2=0x5628c41dbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.365 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1488005690 tx=0x7f148800fa40 comp rx=0 tx=0).stop 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 shutdown_connections 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5628c424f680 0x5628c4251aa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628c41d8950 0x5628c41dc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628c41cfd70 0x5628c41dbfe0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 --2- 172.21.15.22:0/1576380548 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5628c41cf450 0x5628c41e0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 >> 172.21.15.22:0/1576380548 conn(0x5628c41c9b80 msgr2=0x5628c41d5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.357+0000 7f1482000640 0 client.27169 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:47:44.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 shutdown_connections 2024-03-24T06:47:44.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a61aa740 1 -- 172.21.15.22:0/1576380548 wait complete. 2024-03-24T06:47:44.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a61aa740 20 client.27169 trim_cache size 0 max 16384 2024-03-24T06:47:44.367 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 msgr2=0x7f14840a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 0x7f14840a47a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f14900ede00 tx=0x7f14900f6550 comp rx=0 tx=0).stop 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 msgr2=0x7f149c073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1490000c00 tx=0x7f149002cd70 comp rx=0 tx=0).stop 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 shutdown_connections 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.358+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f14840a22e0 0x7f14840a47a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f149c107660 0x7f149c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f149c101f60 0x7f149c073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 --2- 172.21.15.22:0/2586472552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f149c1015a0 0x7f149c073a10 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 >> 172.21.15.22:0/2586472552 conn(0x7f149c0fd0b0 msgr2=0x7f149c0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 shutdown_connections 2024-03-24T06:47:44.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.359+0000 7f14a1e00640 1 -- 172.21.15.22:0/2586472552 wait complete. 2024-03-24T06:47:44.374 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:44 smithi022 ceph-mon[32003]: pgmap v238: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:44.374 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:44 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1327734513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:44.378 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:47:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:44 smithi143 ceph-mon[36520]: pgmap v238: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:44 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1327734513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:44 smithi155 ceph-mon[36262]: pgmap v238: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:44 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1327734513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 -- 172.21.15.22:0/3210169225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db01097a0 msgr2=0x7f0db0109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 --2- 172.21.15.22:0/3210169225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db01097a0 0x7f0db0109b80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009930 tx=0x7f0da002bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 -- 172.21.15.22:0/3210169225 shutdown_connections 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 --2- 172.21.15.22:0/3210169225 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 0x7f0db0110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 --2- 172.21.15.22:0/3210169225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0103790 0x7f0db0103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 --2- 172.21.15.22:0/3210169225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db01097a0 0x7f0db0109b80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 -- 172.21.15.22:0/3210169225 >> 172.21.15.22:0/3210169225 conn(0x7f0db00ff220 msgr2=0x7f0db0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 -- 172.21.15.22:0/3210169225 shutdown_connections 2024-03-24T06:47:44.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.593+0000 7f0db5200640 1 -- 172.21.15.22:0/3210169225 wait complete. 2024-03-24T06:47:44.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.595+0000 7f0db5200640 1 Processor -- start 2024-03-24T06:47:44.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.595+0000 7f0db5200640 1 -- start start 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 0x7f0db019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db01097a0 0x7f0db019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0db019ed20 con 0x7f0db0104150 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0db019ee90 con 0x7f0db01097a0 2024-03-24T06:47:44.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0db5200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0db019f050 con 0x7f0db0103790 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 0x7f0db019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51596/0 (socket says 172.21.15.22:51596) 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 -- 172.21.15.22:0/622532214 learned_addr learned my addr 172.21.15.22:0/622532214 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0dafe00640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db01097a0 0x7f0db019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 -- 172.21.15.22:0/622532214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db01097a0 msgr2=0x7f0db019e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db01097a0 0x7f0db019e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 -- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 msgr2=0x7f0db019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 0x7f0db019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.596+0000 7f0daf400640 1 -- 172.21.15.22:0/622532214 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c009490 con 0x7f0db0103790 2024-03-24T06:47:44.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0daf400640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0da0000c00 tx=0x7f0da002cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0da7e00640 1 -- 172.21.15.22:0/622532214 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da002dea0 con 0x7f0db0103790 2024-03-24T06:47:44.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0da7e00640 1 -- 172.21.15.22:0/622532214 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da0004350 con 0x7f0db0103790 2024-03-24T06:47:44.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0da7e00640 1 -- 172.21.15.22:0/622532214 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0003b50 con 0x7f0db0103790 2024-03-24T06:47:44.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da0009560 con 0x7f0db0103790 2024-03-24T06:47:44.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.597+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db01a69d0 con 0x7f0db0103790 2024-03-24T06:47:44.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 Processor -- start 2024-03-24T06:47:44.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 -- start start 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b42b94b6e0 con 0x55b42b9dd450 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b42b948c10 con 0x55b42b9ddd70 2024-03-24T06:47:44.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0db9410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b42b9443f0 con 0x55b42b9e6950 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0dafe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0daea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0dafe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60610/0 (socket says 172.21.15.22:60610) 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.601+0000 7f0dafe00640 1 -- 172.21.15.22:0/1996432908 learned_addr learned my addr 172.21.15.22:0/1996432908 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 -- 172.21.15.22:0/1996432908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 msgr2=0x55b42b9e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 -- 172.21.15.22:0/1996432908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 msgr2=0x55b42b9e6410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e6410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 -- 172.21.15.22:0/1996432908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b42b905e40 con 0x55b42b9dd450 2024-03-24T06:47:44.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0da7e00640 1 -- 172.21.15.22:0/622532214 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0da0003cb0 con 0x7f0db0103790 2024-03-24T06:47:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0dafe00640 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009f90 tx=0x7f0da8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f28d8700890940c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.602+0000 7f0da4c00640 1 -- 172.21.15.22:0/1996432908 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8011700 con 0x55b42b9dd450 2024-03-24T06:47:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0da4c00640 1 -- 172.21.15.22:0/1996432908 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da8015070 con 0x55b42b9dd450 2024-03-24T06:47:44.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0da4c00640 1 -- 172.21.15.22:0/1996432908 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8019b70 con 0x55b42b9dd450 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 -- 172.21.15.22:0/1996432908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 msgr2=0x55b42b9dd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009f90 tx=0x7f0da8010860 comp rx=0 tx=0).stop 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 -- 172.21.15.22:0/1996432908 shutdown_connections 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0da7e00640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 0x7f0d940a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 --2- 172.21.15.22:0/1996432908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9dd830 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.603+0000 7f0db9410740 1 -- 172.21.15.22:0/1996432908 >> 172.21.15.22:0/1996432908 conn(0x55b42b9d7b80 msgr2=0x55b42b9d7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.604+0000 7f0da7e00640 1 -- 172.21.15.22:0/622532214 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f0da00f0820 con 0x7f0db0103790 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.604+0000 7f0dafe00640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 0x7f0d940a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.604+0000 7f0db9410740 1 -- 172.21.15.22:0/1996432908 shutdown_connections 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.604+0000 7f0db9410740 1 -- 172.21.15.22:0/1996432908 wait complete. 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.604+0000 7f0db9410740 1 -- start start 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0dafe00640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 0x7f0d940a4770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0da8005f10 tx=0x7f0da8010fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 Processor -- start 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42b9e4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b42b8e9060 con 0x55b42b9e6950 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b42b8e8d00 con 0x55b42b9ddd70 2024-03-24T06:47:44.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b42b6b21c0 con 0x55b42b9dd450 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0db9410740 10 client.0 _ll_register_callbacks cb 0x55b42b654be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51624/0 (socket says 172.21.15.22:51624) 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.605+0000 7f0daf400640 1 -- 172.21.15.22:0/4031470982 learned_addr learned my addr 172.21.15.22:0/4031470982 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daea00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0dafe00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42b9e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 -- 172.21.15.22:0/4031470982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 msgr2=0x55b42b9e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 -- 172.21.15.22:0/4031470982 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 msgr2=0x55b42b9e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42b9e4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 -- 172.21.15.22:0/4031470982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da8009bf0 con 0x55b42b9dd450 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daea00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0dafe00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42b9e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0daf400640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0da00ee1c0 tx=0x7f0da00044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c8a198aeb00310 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0d99600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0005640 con 0x55b42b9dd450 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f0da00c1150 con 0x55b42b9dd450 2024-03-24T06:47:44.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.606+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b42b6437e0 con 0x55b42b9dd450 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0da00c5710 con 0x55b42b9dd450 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 10 client.27181 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b42b942a70 con 0x55b42b9dd450 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b42b6437e0 con 0x55b42b9dd450 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0da003c070 con 0x55b42b9dd450 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 10 client.27181 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 10 client.27181 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:47:44.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 20 client.27181 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42b9ff3e0 0x55b42ba017d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 4 client.27181 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.607+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b42b680720 con 0x55b42b9ff3e0 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.608+0000 7f0daea00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42b9ff3e0 0x55b42ba017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.608+0000 7f0daea00640 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42b9ff3e0 0x55b42ba017d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=655c051d63390198 server_cookie=9c0bf972bc1e5fda in_seq=0 out_seq=0 2024-03-24T06:47:44.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.608+0000 7f0d99600640 10 client.27181 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:44.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0d99600640 1 -- 172.21.15.22:0/4031470982 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b42b680720 con 0x55b42b9ff3e0 2024-03-24T06:47:44.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0d99600640 10 client.27181 handle_command_reply: tid=0 2024-03-24T06:47:44.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 client.27181 shutdown 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 msgr2=0x55b42b9e3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0da00ee1c0 tx=0x7f0da00044c0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 shutdown_connections 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42b9ff3e0 0x55b42ba017d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42b9e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 --2- 172.21.15.22:0/4031470982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e3990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0d99600640 0 client.27181 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:44.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 >> 172.21.15.22:0/4031470982 conn(0x55b42b9d7b80 msgr2=0x55b42b9d98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.609+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 shutdown_connections 2024-03-24T06:47:44.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.610+0000 7f0db9410740 1 -- 172.21.15.22:0/4031470982 wait complete. 2024-03-24T06:47:44.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.610+0000 7f0db9410740 20 client.27181 trim_cache size 0 max 16384 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 Processor -- start 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 -- start start 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42ba070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b42b9e4960 con 0x55b42b9e6950 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b42b680720 con 0x55b42b9ddd70 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.625+0000 7f0db9410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b42b9e9620 con 0x55b42b9dd450 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51636/0 (socket says 172.21.15.22:51636) 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 -- 172.21.15.22:0/3785640787 learned_addr learned my addr 172.21.15.22:0/3785640787 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0dafe00640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42ba070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 -- 172.21.15.22:0/3785640787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 msgr2=0x55b42b9faad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9faad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 -- 172.21.15.22:0/3785640787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 msgr2=0x55b42ba070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42ba070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 -- 172.21.15.22:0/3785640787 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b42b8deb00 con 0x55b42b9dd450 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daea00640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0dafe00640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42ba070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.626+0000 7f0daf400640 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0da00edd80 tx=0x7f0da002bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c454854f00a94fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0d9be00640 1 -- 172.21.15.22:0/3785640787 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00c1150 con 0x55b42b9dd450 2024-03-24T06:47:44.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0d9be00640 1 -- 172.21.15.22:0/3785640787 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da00c5710 con 0x55b42b9dd450 2024-03-24T06:47:44.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0d9be00640 1 -- 172.21.15.22:0/3785640787 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00f03e0 con 0x55b42b9dd450 2024-03-24T06:47:44.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 -- 172.21.15.22:0/3785640787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 msgr2=0x55b42b9e60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0da00edd80 tx=0x7f0da002bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 -- 172.21.15.22:0/3785640787 shutdown_connections 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9e6950 0x55b42ba070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 --2- 172.21.15.22:0/3785640787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9dd450 0x55b42b9e60c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 -- 172.21.15.22:0/3785640787 >> 172.21.15.22:0/3785640787 conn(0x55b42b9d7b80 msgr2=0x55b42b9d6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.627+0000 7f0db9410740 1 -- 172.21.15.22:0/3785640787 shutdown_connections 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.628+0000 7f0db9410740 1 -- 172.21.15.22:0/3785640787 wait complete. 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.628+0000 7f0db9410740 1 -- start start 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.628+0000 7f0db9410740 1 Processor -- start 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.628+0000 7f0db9410740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55b42b9e1020 con 0x55b42b9dd450 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b42b9e1190 con 0x55b42b9ddd70 2024-03-24T06:47:44.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b42ba646f0 con 0x55b42b9e6950 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0db9410740 10 client.0 _ll_register_callbacks cb 0x7f0da0009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0dafe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daf400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0dafe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51648/0 (socket says 172.21.15.22:51648) 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44698/0 (socket says 172.21.15.22:44698) 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0dafe00640 1 -- 172.21.15.22:0/1043911126 learned_addr learned my addr 172.21.15.22:0/1043911126 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daf400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60624/0 (socket says 172.21.15.22:60624) 2024-03-24T06:47:44.636 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 -- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 msgr2=0x55b42b9e0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 -- 172.21.15.22:0/1043911126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 msgr2=0x55b42b9f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.629+0000 7f0daea00640 1 -- 172.21.15.22:0/1043911126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c009c20 con 0x55b42b9ddd70 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0daea00640 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x55b42ba02030 tx=0x7f0d9c010be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225a6c0333696867 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0d99600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00fc00 con 0x55b42b9ddd70 2024-03-24T06:47:44.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f0d9c00fd60 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da8015070 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0d9c020070 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0db9410740 10 client.27161 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b42b942a70 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.630+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da8015070 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0d9c029940 con 0x55b42b9ddd70 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 10 client.27161 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 10 client.27161 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 20 client.27161 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42ba65ff0 0x55b42ba68410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 4 client.27161 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b42b6cf890 con 0x55b42ba65ff0 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0daf400640 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42ba65ff0 0x55b42ba68410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0daf400640 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42ba65ff0 0x55b42ba68410 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9686b24ef0305dc8 server_cookie=8b607a5f288456f8 in_seq=0 out_seq=0 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.631+0000 7f0d99600640 10 client.27161 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:44.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0d99600640 1 -- 172.21.15.22:0/1043911126 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b42b6cf890 con 0x55b42ba65ff0 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0d99600640 10 client.27161 handle_command_reply: tid=0 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 client.27161 shutdown 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 msgr2=0x55b42b9e0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x55b42ba02030 tx=0x7f0d9c010be0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 shutdown_connections 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55b42ba65ff0 0x55b42ba68410 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b42b9e6950 0x55b42b9e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b42b9ddd70 0x55b42b9e0350 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.640 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 --2- 172.21.15.22:0/1043911126 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b42b9dd450 0x55b42b9f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0d99600640 0 client.27161 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 >> 172.21.15.22:0/1043911126 conn(0x55b42b9d7b80 msgr2=0x55b42b9d6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 shutdown_connections 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 1 -- 172.21.15.22:0/1043911126 wait complete. 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.632+0000 7f0db9410740 20 client.27161 trim_cache size 0 max 16384 2024-03-24T06:47:44.641 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:44.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 msgr2=0x7f0d940a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 0x7f0d940a4770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0da8005f10 tx=0x7f0da8010fd0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 msgr2=0x7f0db0199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0da0000c00 tx=0x7f0da002cd70 comp rx=0 tx=0).stop 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 shutdown_connections 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0d940a22b0 0x7f0d940a4770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db01097a0 0x7f0db019e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0db0104150 0x7f0db019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 --2- 172.21.15.22:0/622532214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0db0103790 0x7f0db0199e60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 >> 172.21.15.22:0/622532214 conn(0x7f0db00ff220 msgr2=0x7f0db00ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 shutdown_connections 2024-03-24T06:47:44.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.633+0000 7f0db5200640 1 -- 172.21.15.22:0/622532214 wait complete. 2024-03-24T06:47:44.653 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 -- 172.21.15.22:0/892151884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9541038b0 msgr2=0x7fc954103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 --2- 172.21.15.22:0/892151884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9541038b0 0x7fc954103d30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc940009930 tx=0x7fc94002bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 -- 172.21.15.22:0/892151884 shutdown_connections 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 --2- 172.21.15.22:0/892151884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc954104270 0x7fc954110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 --2- 172.21.15.22:0/892151884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9541038b0 0x7fc954103d30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 --2- 172.21.15.22:0/892151884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc954109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.867+0000 7fc95ae00640 1 -- 172.21.15.22:0/892151884 >> 172.21.15.22:0/892151884 conn(0x7fc9540ff3c0 msgr2=0x7fc9541017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.868+0000 7fc95ae00640 1 -- 172.21.15.22:0/892151884 shutdown_connections 2024-03-24T06:47:44.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.868+0000 7fc95ae00640 1 -- 172.21.15.22:0/892151884 wait complete. 2024-03-24T06:47:44.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.869+0000 7fc95ae00640 1 Processor -- start 2024-03-24T06:47:44.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 -- start start 2024-03-24T06:47:44.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9541038b0 0x7fc954199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc95419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc95419ee80 con 0x7fc9541038b0 2024-03-24T06:47:44.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc95419eff0 con 0x7fc9541098c0 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.870+0000 7fc95ae00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc95419f1b0 con 0x7fc954104270 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9541038b0 0x7fc954199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc958e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc95419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33714/0 (socket says 172.21.15.22:33714) 2024-03-24T06:47:44.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 -- 172.21.15.22:0/907625539 learned_addr learned my addr 172.21.15.22:0/907625539 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 -- 172.21.15.22:0/907625539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 msgr2=0x7fc95419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc95419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 -- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9541038b0 msgr2=0x7fc954199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9541038b0 0x7fc954199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc953400640 1 -- 172.21.15.22:0/907625539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc940009560 con 0x7fc954104270 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.871+0000 7fc958e00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc95419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc953400640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc940000c00 tx=0x7fc94002cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc950c00640 1 -- 172.21.15.22:0/907625539 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94002dea0 con 0x7fc954104270 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95419f430 con 0x7fc954104270 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc950c00640 1 -- 172.21.15.22:0/907625539 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc940004350 con 0x7fc954104270 2024-03-24T06:47:44.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc950c00640 1 -- 172.21.15.22:0/907625539 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940003b50 con 0x7fc954104270 2024-03-24T06:47:44.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.872+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541a6d10 con 0x7fc954104270 2024-03-24T06:47:44.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.875+0000 7fc95da20740 1 Processor -- start 2024-03-24T06:47:44.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.875+0000 7fc95da20740 1 -- start start 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x55992450c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5599244716e0 con 0x559924503ba0 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55992446ec10 con 0x55992450c780 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc95da20740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55992446a3f0 con 0x559924503280 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc953e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x55992450c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33722/0 (socket says 172.21.15.22:33722) 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 -- 172.21.15.22:0/2854275933 learned_addr learned my addr 172.21.15.22:0/2854275933 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 -- 172.21.15.22:0/2854275933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 msgr2=0x55992450eb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 -- 172.21.15.22:0/2854275933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 msgr2=0x55992450c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x55992450c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc958e00640 1 -- 172.21.15.22:0/2854275933 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55992442be40 con 0x559924503280 2024-03-24T06:47:44.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.876+0000 7fc953e00640 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x55992450c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.877+0000 7fc950c00640 1 -- 172.21.15.22:0/907625539 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc940003cb0 con 0x7fc954104270 2024-03-24T06:47:44.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.877+0000 7fc958e00640 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0107e0 tx=0x7fc94c010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5805befe629cedd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.877+0000 7fc949600640 1 -- 172.21.15.22:0/2854275933 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c003b70 con 0x559924503280 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.877+0000 7fc949600640 1 -- 172.21.15.22:0/2854275933 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c015070 con 0x559924503280 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.877+0000 7fc949600640 1 -- 172.21.15.22:0/2854275933 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c004530 con 0x559924503280 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 -- 172.21.15.22:0/2854275933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 msgr2=0x559924503660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0107e0 tx=0x7fc94c010ca0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc950c00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 0x7fc9380a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 -- 172.21.15.22:0/2854275933 shutdown_connections 2024-03-24T06:47:44.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc950c00640 1 -- 172.21.15.22:0/907625539 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fc9400f0870 con 0x7fc954104270 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x55992450c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 --2- 172.21.15.22:0/2854275933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924503660 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 -- 172.21.15.22:0/2854275933 >> 172.21.15.22:0/2854275933 conn(0x5599244fd9b0 msgr2=0x5599244fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 -- 172.21.15.22:0/2854275933 shutdown_connections 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc95da20740 1 -- 172.21.15.22:0/2854275933 wait complete. 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.878+0000 7fc953e00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 0x7fc9380a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.879+0000 7fc95da20740 1 -- start start 2024-03-24T06:47:44.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.879+0000 7fc953e00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 0x7fc9380a4770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x559924502d50 tx=0x7fc944005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.879+0000 7fc95da20740 1 Processor -- start 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x559924509d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55992440f060 con 0x559924503ba0 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55992440ed00 con 0x55992450c780 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5599241d81c0 con 0x559924503280 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc95da20740 10 client.0 _ll_register_callbacks cb 0x55992417abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc958e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33726/0 (socket says 172.21.15.22:33726) 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc953400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x559924509d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.880+0000 7fc953e00640 1 -- 172.21.15.22:0/4221035426 learned_addr learned my addr 172.21.15.22:0/4221035426 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 -- 172.21.15.22:0/4221035426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 msgr2=0x559924509800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 -- 172.21.15.22:0/4221035426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 msgr2=0x559924509d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x559924509d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 -- 172.21.15.22:0/4221035426 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94c010410 con 0x55992450c780 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc953400640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x559924509d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc953e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc958e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a280 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc94c000f80 tx=0x7fc94c004070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d13fc74e1f79ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc93e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c004c50 con 0x55992450c780 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fc94c004db0 con 0x55992450c780 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.881+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5599241697e0 con 0x55992450c780 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc94c00e930 con 0x55992450c780 2024-03-24T06:47:44.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 10 client.27170 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559924468a70 con 0x55992450c780 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5599241697e0 con 0x55992450c780 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc94c005050 con 0x55992450c780 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 10 client.27170 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 10 client.27170 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 20 client.27170 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55992451cc50 0x55992451f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 4 client.27170 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:44.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.882+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599241a6720 con 0x55992451cc50 2024-03-24T06:47:44.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.883+0000 7fc953e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55992451cc50 0x55992451f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.883+0000 7fc953e00640 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55992451cc50 0x55992451f040 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90f66bae00578a89 server_cookie=597a51e875470916 in_seq=0 out_seq=0 2024-03-24T06:47:44.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.883+0000 7fc93e000640 10 client.27170 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc93e000640 1 -- 172.21.15.22:0/4221035426 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599241a6720 con 0x55992451cc50 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc93e000640 10 client.27170 handle_command_reply: tid=0 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 client.27170 shutdown 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 msgr2=0x55992450a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a280 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc94c000f80 tx=0x7fc94c004070 comp rx=0 tx=0).stop 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 shutdown_connections 2024-03-24T06:47:44.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55992451cc50 0x55992451f040 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a280 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x559924509d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 --2- 172.21.15.22:0/4221035426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924509800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc93e000640 0 client.27170 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 >> 172.21.15.22:0/4221035426 conn(0x5599244fd9b0 msgr2=0x5599244fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.884+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 shutdown_connections 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.885+0000 7fc95da20740 1 -- 172.21.15.22:0/4221035426 wait complete. 2024-03-24T06:47:44.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.885+0000 7fc95da20740 20 client.27170 trim_cache size 0 max 16384 2024-03-24T06:47:44.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 Processor -- start 2024-03-24T06:47:44.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 -- start start 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x5599244fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5599244febe0 con 0x559924503ba0 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55992450f140 con 0x55992450c780 2024-03-24T06:47:44.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc95da20740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55992451f6d0 con 0x559924503280 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc958e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33734/0 (socket says 172.21.15.22:33734) 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc953e00640 1 -- 172.21.15.22:0/2082402393 learned_addr learned my addr 172.21.15.22:0/2082402393 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.905+0000 7fc953400640 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x5599244fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 -- 172.21.15.22:0/2082402393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 msgr2=0x55992450a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 -- 172.21.15.22:0/2082402393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 msgr2=0x5599244fc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x5599244fc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 -- 172.21.15.22:0/2082402393 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559924404b00 con 0x559924503280 2024-03-24T06:47:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc953e00640 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc944012cb0 tx=0x7fc94401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb820314bb2c7731 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc948c00640 1 -- 172.21.15.22:0/2082402393 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94402f070 con 0x559924503280 2024-03-24T06:47:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc948c00640 1 -- 172.21.15.22:0/2082402393 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc944003d70 con 0x559924503280 2024-03-24T06:47:44.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.906+0000 7fc948c00640 1 -- 172.21.15.22:0/2082402393 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944025040 con 0x559924503280 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 -- 172.21.15.22:0/2082402393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 msgr2=0x559924518040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc944012cb0 tx=0x7fc94401a000 comp rx=0 tx=0).stop 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 -- 172.21.15.22:0/2082402393 shutdown_connections 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55992450c780 0x55992450a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559924503ba0 0x5599244fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 --2- 172.21.15.22:0/2082402393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x559924518040 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 -- 172.21.15.22:0/2082402393 >> 172.21.15.22:0/2082402393 conn(0x5599244fd9b0 msgr2=0x5599244fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 -- 172.21.15.22:0/2082402393 shutdown_connections 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.907+0000 7fc95da20740 1 -- 172.21.15.22:0/2082402393 wait complete. 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.908+0000 7fc95da20740 1 -- start start 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.908+0000 7fc95da20740 1 Processor -- start 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559924503ba0 0x559924506920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55992450c780 0x559924506e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559924514ee0 con 0x55992450c780 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559924581f40 con 0x559924503ba0 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5599245820d0 con 0x559924503280 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc95da20740 10 client.0 _ll_register_callbacks cb 0x7fc9440097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc958e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55992450c780 0x559924506e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33744/0 (socket says 172.21.15.22:33744) 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559924503ba0 0x559924506920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 -- 172.21.15.22:0/3222940458 learned_addr learned my addr 172.21.15.22:0/3222940458 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:44.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 -- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559924503ba0 msgr2=0x559924506920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559924503ba0 0x559924506920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 -- 172.21.15.22:0/3222940458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55992450c780 msgr2=0x559924506e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55992450c780 0x559924506e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.909+0000 7fc953e00640 1 -- 172.21.15.22:0/3222940458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc944009d10 con 0x559924503280 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc953e00640 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc944003f80 tx=0x7fc944003fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e7e6598bea1118 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc93e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944025050 con 0x559924503280 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc94c015070 con 0x559924503280 2024-03-24T06:47:44.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fc94402b470 con 0x559924503280 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc94402f040 con 0x559924503280 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc95da20740 10 client.27205 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.910+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55992442a6a0 con 0x559924503280 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c015070 con 0x559924503280 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc944004ad0 con 0x559924503280 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 10 client.27205 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 10 client.27205 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 20 client.27205 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:44.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5599245839a0 0x559924585dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:44.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 4 client.27205 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:44.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599241f5890 con 0x5599245839a0 2024-03-24T06:47:44.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.911+0000 7fc953400640 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5599245839a0 0x559924585dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:44.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc953400640 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5599245839a0 0x559924585dc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81e5ec92b487aaa2 server_cookie=89f6a0e795a3527a in_seq=0 out_seq=0 2024-03-24T06:47:44.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc93e000640 10 client.27205 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:44.918 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc93e000640 1 -- 172.21.15.22:0/3222940458 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599241f5890 con 0x5599245839a0 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc93e000640 10 client.27205 handle_command_reply: tid=0 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc95da20740 1 client.27205 shutdown 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 msgr2=0x5599245063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.912+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc944003f80 tx=0x7fc944003fb0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 shutdown_connections 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55992450c780 0x559924506e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5599245839a0 0x559924585dc0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.919 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559924503ba0 0x559924506920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc93e000640 0 client.27205 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 --2- 172.21.15.22:0/3222940458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559924503280 0x5599245063e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 >> 172.21.15.22:0/3222940458 conn(0x5599244fd9b0 msgr2=0x5599244fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 shutdown_connections 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 1 -- 172.21.15.22:0/3222940458 wait complete. 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.913+0000 7fc95da20740 20 client.27205 trim_cache size 0 max 16384 2024-03-24T06:47:44.920 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:44.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 msgr2=0x7fc9380a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 0x7fc9380a4770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x559924502d50 tx=0x7fc944005f90 comp rx=0 tx=0).stop 2024-03-24T06:47:44.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 msgr2=0x7fc95419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:44.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc940000c00 tx=0x7fc94002cd70 comp rx=0 tx=0).stop 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 shutdown_connections 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9380a22b0 0x7fc9380a4770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9541098c0 0x7fc95419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954104270 0x7fc95419a500 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 --2- 172.21.15.22:0/907625539 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9541038b0 0x7fc954199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 >> 172.21.15.22:0/907625539 conn(0x7fc9540ff3c0 msgr2=0x7fc9540ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 shutdown_connections 2024-03-24T06:47:44.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:44.914+0000 7fc95ae00640 1 -- 172.21.15.22:0/907625539 wait complete. 2024-03-24T06:47:44.940 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:47:45.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.141+0000 7f9e6c000640 1 -- 172.21.15.22:0/3093294448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 msgr2=0x7f9e64109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.141+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3093294448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e64109ca0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009930 tx=0x7f9e5802bc20 comp rx=0 tx=0).stop 2024-03-24T06:47:45.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 -- 172.21.15.22:0/3093294448 shutdown_connections 2024-03-24T06:47:45.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3093294448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64104270 0x7f9e64110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3093294448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e641038b0 0x7f9e64103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3093294448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e64109ca0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 -- 172.21.15.22:0/3093294448 >> 172.21.15.22:0/3093294448 conn(0x7f9e640ff3c0 msgr2=0x7f9e641017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 -- 172.21.15.22:0/3093294448 shutdown_connections 2024-03-24T06:47:45.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.142+0000 7f9e6c000640 1 -- 172.21.15.22:0/3093294448 wait complete. 2024-03-24T06:47:45.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.143+0000 7f9e6c000640 1 Processor -- start 2024-03-24T06:47:45.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.143+0000 7f9e6c000640 1 -- start start 2024-03-24T06:47:45.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 0x7f9e64199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e64104270 0x7f9e6419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6419ee40 con 0x7f9e64104270 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6419efb0 con 0x7f9e641038b0 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6c000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6419f170 con 0x7f9e641098c0 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e68c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e64104270 0x7f9e6419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33758/0 (socket says 172.21.15.22:33758) 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e69600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 0x7f9e64199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.144+0000 7f9e6a000640 1 -- 172.21.15.22:0/3775988132 learned_addr learned my addr 172.21.15.22:0/3775988132 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 -- 172.21.15.22:0/3775988132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 msgr2=0x7f9e64199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 0x7f9e64199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 -- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e64104270 msgr2=0x7f9e6419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e64104270 0x7f9e6419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 -- 172.21.15.22:0/3775988132 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c009840 con 0x7f9e641098c0 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e69600640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 0x7f9e64199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:45.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00efd0 tx=0x7f9e5c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e62000640 1 -- 172.21.15.22:0/3775988132 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c009b40 con 0x7f9e641098c0 2024-03-24T06:47:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e62000640 1 -- 172.21.15.22:0/3775988132 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c003b70 con 0x7f9e641098c0 2024-03-24T06:47:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e58009560 con 0x7f9e641098c0 2024-03-24T06:47:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e62000640 1 -- 172.21.15.22:0/3775988132 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0108f0 con 0x7f9e641098c0 2024-03-24T06:47:45.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.145+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e641a6af0 con 0x7f9e641098c0 2024-03-24T06:47:45.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.150+0000 7f9e6ec68740 1 Processor -- start 2024-03-24T06:47:45.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.150+0000 7f9e62000640 1 -- 172.21.15.22:0/3775988132 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f9e5c003ce0 con 0x7f9e641098c0 2024-03-24T06:47:45.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.150+0000 7f9e6ec68740 1 -- start start 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab8316660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab8321b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ab82846e0 con 0x558ab8316280 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ab8281c10 con 0x558ab831f780 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ab827d3f0 con 0x558ab8316ba0 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33760/0 (socket says 172.21.15.22:33760) 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 -- 172.21.15.22:0/3007441086 learned_addr learned my addr 172.21.15.22:0/3007441086 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab8316660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab8321b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 -- 172.21.15.22:0/3007441086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 msgr2=0x558ab8321b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab8321b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 -- 172.21.15.22:0/3007441086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 msgr2=0x558ab8316660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab8316660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e69600640 1 -- 172.21.15.22:0/3007441086 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ab823ee40 con 0x558ab8316ba0 2024-03-24T06:47:45.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.151+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab8316660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:45.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e62000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 0x7f9e4c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e69600640 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9e5802c230 tx=0x7f9e5802c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c0856965eb86969 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e62000640 1 -- 172.21.15.22:0/3775988132 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f9e5c014070 con 0x7f9e641098c0 2024-03-24T06:47:45.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 0x7f9e4c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e56600640 1 -- 172.21.15.22:0/3007441086 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5802dea0 con 0x558ab8316ba0 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e56600640 1 -- 172.21.15.22:0/3007441086 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58004040 con 0x558ab8316ba0 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.152+0000 7f9e56600640 1 -- 172.21.15.22:0/3007441086 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58005370 con 0x558ab8316ba0 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3007441086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 msgr2=0x558ab831f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9e5802c230 tx=0x7f9e5802c700 comp rx=0 tx=0).stop 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 0x7f9e4c0a4770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x558ab8311b40 tx=0x7f9e50005c90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3007441086 shutdown_connections 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab8321b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831f240 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3007441086 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab8316660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3007441086 >> 172.21.15.22:0/3007441086 conn(0x558ab83109b0 msgr2=0x558ab8310d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3007441086 shutdown_connections 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.153+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3007441086 wait complete. 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.154+0000 7f9e6ec68740 1 -- start start 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.154+0000 7f9e6ec68740 1 Processor -- start 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab831c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ab8222060 con 0x558ab8316280 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ab8221d00 con 0x558ab831f780 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ab7feb1c0 con 0x558ab8316ba0 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6ec68740 10 client.0 _ll_register_callbacks cb 0x558ab7f8dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e68c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e69600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab831c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46568/0 (socket says 172.21.15.22:46568) 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.155+0000 7f9e6a000640 1 -- 172.21.15.22:0/935295284 learned_addr learned my addr 172.21.15.22:0/935295284 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 -- 172.21.15.22:0/935295284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 msgr2=0x558ab831d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 -- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 msgr2=0x558ab831c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab831c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 -- 172.21.15.22:0/935295284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e64111080 con 0x558ab8316ba0 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e6a000640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e68c00640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831ccc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9e50005d00 tx=0x7f9e50004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf0e820146bbbdd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50005020 con 0x558ab8316ba0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f9e50005180 con 0x558ab8316ba0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.156+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ab7f7c7e0 con 0x558ab8316ba0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e50005340 con 0x558ab8316ba0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 10 client.27217 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558ab827ba70 con 0x558ab8316ba0 2024-03-24T06:47:45.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ab7f7c7e0 con 0x558ab8316ba0 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9e500205d0 con 0x558ab8316ba0 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 10 client.27217 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 10 client.27217 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 20 client.27217 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab832fc50 0x558ab8332070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 4 client.27217 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.157+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ab7fb9720 con 0x558ab832fc50 2024-03-24T06:47:45.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.158+0000 7f9e69600640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab832fc50 0x558ab8332070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.158+0000 7f9e69600640 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab832fc50 0x558ab8332070 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ed8f303ef49d58 server_cookie=f3fc9d314b799137 in_seq=0 out_seq=0 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.158+0000 7f9e4aa00640 10 client.27217 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e4aa00640 1 -- 172.21.15.22:0/935295284 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ab7fb9720 con 0x558ab832fc50 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e4aa00640 10 client.27217 handle_command_reply: tid=0 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 client.27217 shutdown 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 msgr2=0x558ab831ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831ccc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9e50005d00 tx=0x7f9e50004730 comp rx=0 tx=0).stop 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 shutdown_connections 2024-03-24T06:47:45.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab832fc50 0x558ab8332070 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab831f780 0x558ab831d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab8316ba0 0x558ab831ccc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e4aa00640 0 client.27217 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/935295284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab831c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.159+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 >> 172.21.15.22:0/935295284 conn(0x558ab83109b0 msgr2=0x558ab8312700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.160+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 shutdown_connections 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.160+0000 7f9e6ec68740 1 -- 172.21.15.22:0/935295284 wait complete. 2024-03-24T06:47:45.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.160+0000 7f9e6ec68740 20 client.27217 trim_cache size 0 max 16384 2024-03-24T06:47:45.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.174+0000 7f9e6ec68740 1 Processor -- start 2024-03-24T06:47:45.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.174+0000 7f9e6ec68740 1 -- start start 2024-03-24T06:47:45.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab831d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8337960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ab83119b0 con 0x558ab8316280 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ab832c5b0 con 0x558ab8316ba0 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ab8315ee0 con 0x558ab831f780 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e69600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8337960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e69600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43160/0 (socket says 172.21.15.22:43160) 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e68c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab831d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e69600640 1 -- 172.21.15.22:0/1794954023 learned_addr learned my addr 172.21.15.22:0/1794954023 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:45.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 -- 172.21.15.22:0/1794954023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 msgr2=0x558ab831d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab831d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 -- 172.21.15.22:0/1794954023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 msgr2=0x558ab832afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e6a000640 1 -- 172.21.15.22:0/1794954023 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ab8217b00 con 0x558ab831f780 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e68c00640 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab831d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.175+0000 7f9e69600640 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e6a000640 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8337960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0a3190 tx=0x7f9e5c09a990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64314732adaf1b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e55c00640 1 -- 172.21.15.22:0/1794954023 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c09f070 con 0x558ab831f780 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e55c00640 1 -- 172.21.15.22:0/1794954023 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e5c0a36c0 con 0x558ab831f780 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e55c00640 1 -- 172.21.15.22:0/1794954023 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0ce740 con 0x558ab831f780 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e6ec68740 1 -- 172.21.15.22:0/1794954023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 msgr2=0x558ab8337960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.176+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8337960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0a3190 tx=0x7f9e5c09a990 comp rx=0 tx=0).stop 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 -- 172.21.15.22:0/1794954023 shutdown_connections 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8337960 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab831d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/1794954023 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab832afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 -- 172.21.15.22:0/1794954023 >> 172.21.15.22:0/1794954023 conn(0x558ab83109b0 msgr2=0x558ab8311d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 -- 172.21.15.22:0/1794954023 shutdown_connections 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 -- 172.21.15.22:0/1794954023 wait complete. 2024-03-24T06:47:45.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.177+0000 7f9e6ec68740 1 -- start start 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 Processor -- start 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab83275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ab8323480 con 0x558ab8316280 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ab83235f0 con 0x558ab8316ba0 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ab83237b0 con 0x558ab831f780 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6ec68740 10 client.0 _ll_register_callbacks cb 0x7f9e5000b570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:45.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e68c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e69600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab83275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33784/0 (socket says 172.21.15.22:33784) 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e68c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46584/0 (socket says 172.21.15.22:46584) 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.178+0000 7f9e6a000640 1 -- 172.21.15.22:0/3558010888 learned_addr learned my addr 172.21.15.22:0/3558010888 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 -- 172.21.15.22:0/3558010888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 msgr2=0x558ab8322a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 -- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 msgr2=0x558ab83275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:45.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab83275d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 -- 172.21.15.22:0/3558010888 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ab823ee40 con 0x558ab831f780 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e68c00640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6a000640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00cba0 tx=0x7f9e5c004890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a50e88ae262a56a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0ce6c0 con 0x558ab831f780 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f9e5c004b40 con 0x558ab831f780 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.179+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e58004040 con 0x558ab831f780 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e5c004d00 con 0x558ab831f780 2024-03-24T06:47:45.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 10 client.27223 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558ab827ba70 con 0x558ab831f780 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58004040 con 0x558ab831f780 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9e5c099de0 con 0x558ab831f780 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 10 client.27223 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 10 client.27223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 20 client.27223 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab83960e0 0x558ab8398500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 4 client.27223 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:47:45.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ab8008890 con 0x558ab83960e0 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.180+0000 7f9e69600640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab83960e0 0x558ab8398500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.181+0000 7f9e69600640 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab83960e0 0x558ab8398500 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55db0cf9e8f70cee server_cookie=c56eca25437ab8b3 in_seq=0 out_seq=0 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.181+0000 7f9e4aa00640 10 client.27223 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.181+0000 7f9e4aa00640 1 -- 172.21.15.22:0/3558010888 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ab8008890 con 0x558ab83960e0 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.181+0000 7f9e4aa00640 10 client.27223 handle_command_reply: tid=0 2024-03-24T06:47:45.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.181+0000 7f9e6ec68740 1 client.27223 shutdown 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 msgr2=0x558ab8322f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00cba0 tx=0x7f9e5c004890 comp rx=0 tx=0).stop 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 shutdown_connections 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ab831f780 0x558ab8322f40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ab83960e0 0x558ab8398500 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ab8316ba0 0x558ab8322a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 --2- 172.21.15.22:0/3558010888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ab8316280 0x558ab83275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e4aa00640 0 client.27223 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:45.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 >> 172.21.15.22:0/3558010888 conn(0x558ab83109b0 msgr2=0x558ab830fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 shutdown_connections 2024-03-24T06:47:45.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 1 -- 172.21.15.22:0/3558010888 wait complete. 2024-03-24T06:47:45.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.182+0000 7f9e6ec68740 20 client.27223 trim_cache size 0 max 16384 2024-03-24T06:47:45.190 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 msgr2=0x7f9e4c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 0x7f9e4c0a4770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x558ab8311b40 tx=0x7f9e50005c90 comp rx=0 tx=0).stop 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 msgr2=0x7f9e6419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00efd0 tx=0x7f9e5c00c6a0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 shutdown_connections 2024-03-24T06:47:45.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a22b0 0x7f9e4c0a4770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e641098c0 0x7f9e6419e840 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e64104270 0x7f9e6419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 --2- 172.21.15.22:0/3775988132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e641038b0 0x7f9e64199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 >> 172.21.15.22:0/3775988132 conn(0x7f9e640ff3c0 msgr2=0x7f9e640ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 shutdown_connections 2024-03-24T06:47:45.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:45.183+0000 7f9e6c000640 1 -- 172.21.15.22:0/3775988132 wait complete. 2024-03-24T06:47:46.210 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:47:46.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:47:46.211 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:47:46.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:46 smithi022 ceph-mon[32003]: pgmap v239: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:46 smithi143 ceph-mon[36520]: pgmap v239: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 -- 172.21.15.22:0/2491163626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 msgr2=0x7f9934107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 --2- 172.21.15.22:0/2491163626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f9934107a90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f991c009930 tx=0x7f991c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 -- 172.21.15.22:0/2491163626 shutdown_connections 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 --2- 172.21.15.22:0/2491163626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 --2- 172.21.15.22:0/2491163626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 --2- 172.21.15.22:0/2491163626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f9934107a90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 -- 172.21.15.22:0/2491163626 >> 172.21.15.22:0/2491163626 conn(0x7f9934078b10 msgr2=0x7f9934078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 -- 172.21.15.22:0/2491163626 shutdown_connections 2024-03-24T06:47:46.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.443+0000 7f993b200640 1 -- 172.21.15.22:0/2491163626 wait complete. 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.444+0000 7f993b200640 1 Processor -- start 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 -- start start 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f993419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f993419ee20 con 0x7f9934102060 2024-03-24T06:47:46.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f993419ef90 con 0x7f99341016a0 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f993b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f993419f150 con 0x7f99341076b0 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f9933e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f9933e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46596/0 (socket says 172.21.15.22:46596) 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f9933e00640 1 -- 172.21.15.22:0/1933095764 learned_addr learned my addr 172.21.15.22:0/1933095764 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f9933400640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.445+0000 7f9939200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f993419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 -- 172.21.15.22:0/1933095764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 msgr2=0x7f993419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f993419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 -- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 msgr2=0x7f993419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 -- 172.21.15.22:0/1933095764 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9924009840 con 0x7f99341016a0 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933400640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9939200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f993419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9933e00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c02c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9930c00640 1 -- 172.21.15.22:0/1933095764 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c0042d0 con 0x7f99341016a0 2024-03-24T06:47:46.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9930c00640 1 -- 172.21.15.22:0/1933095764 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f991c004430 con 0x7f99341016a0 2024-03-24T06:47:46.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f9930c00640 1 -- 172.21.15.22:0/1933095764 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c003c50 con 0x7f99341016a0 2024-03-24T06:47:46.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991c009560 con 0x7f99341016a0 2024-03-24T06:47:46.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.446+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99341a6ad0 con 0x7f99341016a0 2024-03-24T06:47:46.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f9930c00640 1 -- 172.21.15.22:0/1933095764 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f991c004df0 con 0x7f99341016a0 2024-03-24T06:47:46.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 Processor -- start 2024-03-24T06:47:46.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 -- start start 2024-03-24T06:47:46.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c3750 0x56463c1cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56463c1312d0 con 0x56463c1c3750 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56463c12e800 con 0x56463c1c2e30 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f993de1d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56463c129fe0 con 0x56463c1cc330 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.450+0000 7f9939200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9933400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46612/0 (socket says 172.21.15.22:46612) 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 -- 172.21.15.22:0/532771420 learned_addr learned my addr 172.21.15.22:0/532771420 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9933e00640 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c3750 0x56463c1cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 -- 172.21.15.22:0/532771420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 msgr2=0x56463c1ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 -- 172.21.15.22:0/532771420 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c3750 msgr2=0x56463c1cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c3750 0x56463c1cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 -- 172.21.15.22:0/532771420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56463c0ebe90 con 0x56463c1c2e30 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9933400640 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.451+0000 7f9939200640 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f992c009c20 tx=0x7f992c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc744ac71be4d4bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9929600640 1 -- 172.21.15.22:0/532771420 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c009720 con 0x56463c1c2e30 2024-03-24T06:47:46.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9929600640 1 -- 172.21.15.22:0/532771420 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f992c012930 con 0x56463c1c2e30 2024-03-24T06:47:46.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9929600640 1 -- 172.21.15.22:0/532771420 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992c00faf0 con 0x56463c1c2e30 2024-03-24T06:47:46.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9930c00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 0x7f99180a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 -- 172.21.15.22:0/532771420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 msgr2=0x56463c1c3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f992c009c20 tx=0x7f992c0117b0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9933e00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 0x7f99180a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 -- 172.21.15.22:0/532771420 shutdown_connections 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f9930c00640 1 -- 172.21.15.22:0/1933095764 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f991c0f0b30 con 0x7f99341016a0 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c3750 0x56463c1cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 --2- 172.21.15.22:0/532771420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1c3210 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.452+0000 7f993de1d740 1 -- 172.21.15.22:0/532771420 >> 172.21.15.22:0/532771420 conn(0x56463c1bd560 msgr2=0x56463c1bd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.453+0000 7f993de1d740 1 -- 172.21.15.22:0/532771420 shutdown_connections 2024-03-24T06:47:46.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.453+0000 7f993de1d740 1 -- 172.21.15.22:0/532771420 wait complete. 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.453+0000 7f993de1d740 1 -- start start 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.453+0000 7f9933e00640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 0x7f99180a4770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56463c1c2900 tx=0x7f991c009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 Processor -- start 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 0x56463c1c9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56463c0cefc0 con 0x56463c1c2e30 2024-03-24T06:47:46.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56463c0cec60 con 0x56463c1c3750 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56463be720c0 con 0x56463c1cc330 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f993de1d740 10 client.0 _ll_register_callbacks cb 0x56463c0e6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f9933400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f9933400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46616/0 (socket says 172.21.15.22:46616) 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f9933400640 1 -- 172.21.15.22:0/2000968227 learned_addr learned my addr 172.21.15.22:0/2000968227 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.454+0000 7f9939200640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 0x56463c1c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9939200640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 -- 172.21.15.22:0/2000968227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 msgr2=0x56463c1c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 -- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 msgr2=0x56463c1c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 0x56463c1c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 -- 172.21.15.22:0/2000968227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f992c009880 con 0x56463c1c3750 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9939200640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 0x56463c1c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9933400640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9924011d10 tx=0x7f99240107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d316a11718fcbea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9922000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.455+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924016070 con 0x56463c1c3750 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56463c14ee90 con 0x56463c1c3750 2024-03-24T06:47:46.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f992400e5d0 con 0x56463c1c3750 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f992400e790 con 0x56463c1c3750 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f993de1d740 10 client.27206 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56463c128660 con 0x56463c1c3750 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56463c14ee90 con 0x56463c1c3750 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.456+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9924019900 con 0x56463c1c3750 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 10 client.27206 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 10 client.27206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 20 client.27206 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:46.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c1dc940 0x56463c1ded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 4 client.27206 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56463c120210 con 0x56463c1dc940 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.457+0000 7f9939200640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c1dc940 0x56463c1ded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.458+0000 7f9922000640 10 client.27206 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.458+0000 7f9939200640 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c1dc940 0x56463c1ded60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3698562dbc5803 server_cookie=e3f1fc3438053efb in_seq=0 out_seq=0 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.458+0000 7f9922000640 1 -- 172.21.15.22:0/2000968227 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56463c120210 con 0x56463c1dc940 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.458+0000 7f9922000640 10 client.27206 handle_command_reply: tid=0 2024-03-24T06:47:46.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.458+0000 7f993de1d740 1 client.27206 shutdown 2024-03-24T06:47:46.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 msgr2=0x56463c1c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9924011d10 tx=0x7f99240107a0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 shutdown_connections 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1cc330 0x56463c1c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c1dc940 0x56463c1ded60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1c98c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 --2- 172.21.15.22:0/2000968227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1c2e30 0x56463c1c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f9922000640 0 client.27206 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 >> 172.21.15.22:0/2000968227 conn(0x56463c1bd560 msgr2=0x56463c1bf2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 shutdown_connections 2024-03-24T06:47:46.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 1 -- 172.21.15.22:0/2000968227 wait complete. 2024-03-24T06:47:46.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.459+0000 7f993de1d740 20 client.27206 trim_cache size 0 max 16384 2024-03-24T06:47:46.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.478+0000 7f993de1d740 1 Processor -- start 2024-03-24T06:47:46.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.478+0000 7f993de1d740 1 -- start start 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c2e30 0x56463c1cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1e4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56463c1d8190 con 0x56463c1cc330 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56463c1bb9b0 con 0x56463c1c3750 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f993de1d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56463c1b8530 con 0x56463c1c2e30 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9939200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c2e30 0x56463c1cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46622/0 (socket says 172.21.15.22:46622) 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 -- 172.21.15.22:0/662082445 learned_addr learned my addr 172.21.15.22:0/662082445 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9939200640 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 -- 172.21.15.22:0/662082445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c2e30 msgr2=0x56463c1cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c2e30 0x56463c1cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 -- 172.21.15.22:0/662082445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 msgr2=0x56463c1e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1e4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9933400640 1 -- 172.21.15.22:0/662082445 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56463c0c4ae0 con 0x56463c1c3750 2024-03-24T06:47:46.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.479+0000 7f9939200640 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f9933400640 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f992400a010 tx=0x7f9924009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73bff6f730df875f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f9928c00640 1 -- 172.21.15.22:0/662082445 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924016040 con 0x56463c1c3750 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f9928c00640 1 -- 172.21.15.22:0/662082445 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9924003970 con 0x56463c1c3750 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f9928c00640 1 -- 172.21.15.22:0/662082445 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99240194e0 con 0x56463c1c3750 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f993de1d740 1 -- 172.21.15.22:0/662082445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 msgr2=0x56463c1d7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.480+0000 7f993de1d740 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f992400a010 tx=0x7f9924009c40 comp rx=0 tx=0).stop 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 -- 172.21.15.22:0/662082445 shutdown_connections 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c3750 0x56463c1d7c50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 --2- 172.21.15.22:0/662082445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c2e30 0x56463c1cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 -- 172.21.15.22:0/662082445 >> 172.21.15.22:0/662082445 conn(0x56463c1bd560 msgr2=0x56463c1be920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 -- 172.21.15.22:0/662082445 shutdown_connections 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 -- 172.21.15.22:0/662082445 wait complete. 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 -- start start 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.481+0000 7f993de1d740 1 Processor -- start 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1d4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56463c1d4ac0 con 0x56463c1cc330 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56463c1d1cd0 con 0x56463c1c2e30 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56463c1d1e90 con 0x56463c1c3750 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f993de1d740 10 client.0 _ll_register_callbacks cb 0x7f9924009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f9939200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1d4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f9933400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f9933400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33852/0 (socket says 172.21.15.22:33852) 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f9933400640 1 -- 172.21.15.22:0/1423115347 learned_addr learned my addr 172.21.15.22:0/1423115347 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.482+0000 7f9939200640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 -- 172.21.15.22:0/1423115347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 msgr2=0x56463c1d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1d4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 -- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 msgr2=0x56463c1d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 -- 172.21.15.22:0/1423115347 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56463c128660 con 0x56463c1c3750 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9933400640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99240107e0 tx=0x7f9924003d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=137a9ac95ea63061 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9922000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924016040 con 0x56463c1c3750 2024-03-24T06:47:46.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7f9924004c10 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f992c012930 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.483+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9924004dd0 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 10 client.27241 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56463c0ea6f0 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f992c012930 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f992403b070 con 0x56463c1c3750 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 10 client.27241 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:46.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 10 client.27241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:46.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 20 client.27241 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:46.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c243ee0 0x56463c246300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 4 client.27241 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:47:46.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56463be29860 con 0x56463c243ee0 2024-03-24T06:47:46.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.484+0000 7f9939200640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c243ee0 0x56463c246300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.485+0000 7f9939200640 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c243ee0 0x56463c246300 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f965b9eba58f542d server_cookie=2a1c7200ba9339a in_seq=0 out_seq=0 2024-03-24T06:47:46.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.485+0000 7f9922000640 10 client.27241 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f9922000640 1 -- 172.21.15.22:0/1423115347 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56463be29860 con 0x56463c243ee0 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f9922000640 10 client.27241 handle_command_reply: tid=0 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 client.27241 shutdown 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 msgr2=0x56463c1d1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f99240107e0 tx=0x7f9924003d10 comp rx=0 tx=0).stop 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 shutdown_connections 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56463c1cc330 0x56463c1d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56463c1c3750 0x56463c1d1250 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x56463c243ee0 0x56463c246300 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 --2- 172.21.15.22:0/1423115347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56463c1c2e30 0x56463c1d4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 >> 172.21.15.22:0/1423115347 conn(0x56463c1bd560 msgr2=0x56463c1c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f9922000640 0 client.27241 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 shutdown_connections 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.486+0000 7f993de1d740 1 -- 172.21.15.22:0/1423115347 wait complete. 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.487+0000 7f993de1d740 20 client.27241 trim_cache size 0 max 16384 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:46.493 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:47:46.494 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "2c774ba3-d4d0-40b1-ad65-711cd0a26e4f", 2024-03-24T06:47:46.494 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:47:46.494 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:46.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.487+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 msgr2=0x7f99180a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.487+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 0x7f99180a4770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x56463c1c2900 tx=0x7f991c009260 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.487+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 msgr2=0x7f9934199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.487+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c02c5f0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 shutdown_connections 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f99180a22b0 0x7f99180a4770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99341076b0 0x7f993419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9934102060 0x7f993419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 --2- 172.21.15.22:0/1933095764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99341016a0 0x7f9934199f60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 >> 172.21.15.22:0/1933095764 conn(0x7f9934078b10 msgr2=0x7f993410cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 shutdown_connections 2024-03-24T06:47:46.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.488+0000 7f993b200640 1 -- 172.21.15.22:0/1933095764 wait complete. 2024-03-24T06:47:46.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:47:46.512 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:47:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:46 smithi155 ceph-mon[36262]: pgmap v239: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 -- 172.21.15.22:0/3233909106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234100530 msgr2=0x7fa234100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 --2- 172.21.15.22:0/3233909106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234100530 0x7fa234100910 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa224009900 tx=0x7fa22402bd10 comp rx=0 tx=0).stop 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 -- 172.21.15.22:0/3233909106 shutdown_connections 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 --2- 172.21.15.22:0/3233909106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234105930 0x7fa234105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 --2- 172.21.15.22:0/3233909106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa234105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 --2- 172.21.15.22:0/3233909106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234100530 0x7fa234100910 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 -- 172.21.15.22:0/3233909106 >> 172.21.15.22:0/3233909106 conn(0x7fa2340fbf60 msgr2=0x7fa2340fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 -- 172.21.15.22:0/3233909106 shutdown_connections 2024-03-24T06:47:46.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.717+0000 7fa239e00640 1 -- 172.21.15.22:0/3233909106 wait complete. 2024-03-24T06:47:46.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.718+0000 7fa239e00640 1 Processor -- start 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.718+0000 7fa239e00640 1 -- start start 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234100530 0x7fa23419e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 0x7fa2341a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa2341a30a0 con 0x7fa234100530 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa2341a3210 con 0x7fa234105930 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa2341a33d0 con 0x7fa234100e50 2024-03-24T06:47:46.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa238a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234100530 0x7fa23419e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 0x7fa2341a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa22be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 0x7fa2341a2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46652/0 (socket says 172.21.15.22:46652) 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa239400640 1 -- 172.21.15.22:0/3553016065 learned_addr learned my addr 172.21.15.22:0/3553016065 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.719+0000 7fa22be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33864/0 (socket says 172.21.15.22:33864) 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 -- 172.21.15.22:0/3553016065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 msgr2=0x7fa2341a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 0x7fa2341a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 -- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234100530 msgr2=0x7fa23419e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234100530 0x7fa23419e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 -- 172.21.15.22:0/3553016065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa224009560 con 0x7fa234100e50 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa22be00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00e7c0 tx=0x7fa21c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa232000640 1 -- 172.21.15.22:0/3553016065 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00ccc0 con 0x7fa234100e50 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa232000640 1 -- 172.21.15.22:0/3553016065 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c009530 con 0x7fa234100e50 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2341a36b0 con 0x7fa234100e50 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.720+0000 7fa232000640 1 -- 172.21.15.22:0/3553016065 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c004a00 con 0x7fa234100e50 2024-03-24T06:47:46.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.721+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2341aaf30 con 0x7fa234100e50 2024-03-24T06:47:46.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.724+0000 7fa232000640 1 -- 172.21.15.22:0/3553016065 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa21c005020 con 0x7fa234100e50 2024-03-24T06:47:46.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.724+0000 7fa23e1f3740 1 Processor -- start 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.724+0000 7fa23e1f3740 1 -- start start 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.724+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x56173412e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.724+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x56173413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56173409d790 con 0x5617341382d0 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56173409acc0 con 0x56173412fc40 2024-03-24T06:47:46.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5617340964a0 con 0x56173412ef60 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa239400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x56173412e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33874/0 (socket says 172.21.15.22:33874) 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa239400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x56173412e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46668/0 (socket says 172.21.15.22:46668) 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 -- 172.21.15.22:0/2167331267 learned_addr learned my addr 172.21.15.22:0/2167331267 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa22be00640 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x56173413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 -- 172.21.15.22:0/2167331267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 msgr2=0x56173412e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x56173412e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 -- 172.21.15.22:0/2167331267 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 msgr2=0x56173413a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x56173413a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.725+0000 7fa238a00640 1 -- 172.21.15.22:0/2167331267 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561734057c20 con 0x56173412ef60 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa232000640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 0x7fa2180a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa238a00640 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa224000c00 tx=0x7fa22402c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=552a4511bf918b29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa232000640 1 -- 172.21.15.22:0/3553016065 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fa21c014070 con 0x7fa234100e50 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa229c00640 1 -- 172.21.15.22:0/2167331267 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224004900 con 0x56173412ef60 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa229c00640 1 -- 172.21.15.22:0/2167331267 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa224004a60 con 0x56173412ef60 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa229c00640 1 -- 172.21.15.22:0/2167331267 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224005050 con 0x56173412ef60 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.726+0000 7fa239400640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 0x7fa2180a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- 172.21.15.22:0/2167331267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 msgr2=0x56173412f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa224000c00 tx=0x7fa22402c7a0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa239400640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 0x7fa2180a4770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009270 tx=0x7fa22c009680 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- 172.21.15.22:0/2167331267 shutdown_connections 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x56173413a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x56173412f3e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/2167331267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x56173412e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- 172.21.15.22:0/2167331267 >> 172.21.15.22:0/2167331267 conn(0x5617340f5b30 msgr2=0x5617341299b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- 172.21.15.22:0/2167331267 shutdown_connections 2024-03-24T06:47:46.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- 172.21.15.22:0/2167331267 wait complete. 2024-03-24T06:47:46.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.727+0000 7fa23e1f3740 1 -- start start 2024-03-24T06:47:46.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.728+0000 7fa23e1f3740 1 Processor -- start 2024-03-24T06:47:46.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.728+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.728+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x5617341357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.728+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 0x561734135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x56173403af20 con 0x56173412fc40 2024-03-24T06:47:46.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56173403abc0 con 0x5617341382d0 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561733e041c0 con 0x56173412ef60 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa23e1f3740 10 client.0 _ll_register_callbacks cb 0x561733da6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa22be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x5617341357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33886/0 (socket says 172.21.15.22:33886) 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa238a00640 1 -- 172.21.15.22:0/3618720313 learned_addr learned my addr 172.21.15.22:0/3618720313 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 0x561734135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 -- 172.21.15.22:0/3618720313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 msgr2=0x561734135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 -- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 msgr2=0x5617341357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x5617341357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa239400640 1 -- 172.21.15.22:0/3618720313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa234115290 con 0x5617341382d0 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa238a00640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.729+0000 7fa22be00640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x5617341357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa239400640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 0x561734135cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009fd0 tx=0x7fa22c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444890a9df01da22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa222000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c017070 con 0x5617341382d0 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56173408e160 con 0x5617341382d0 2024-03-24T06:47:46.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fa22c020940 con 0x5617341382d0 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa22c020b00 con 0x5617341382d0 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa23e1f3740 10 client.27224 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561734094b20 con 0x5617341382d0 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.730+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56173408e160 con 0x5617341382d0 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa22c01d3f0 con 0x5617341382d0 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 10 client.27224 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 10 client.27224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 20 client.27224 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341488c0 0x56173414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 4 client.27224 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561733d957e0 con 0x5617341488c0 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.731+0000 7fa238a00640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341488c0 0x56173414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.732+0000 7fa238a00640 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341488c0 0x56173414ace0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8f6ce41a2e6e488 server_cookie=b7243cb7fb8e7e3c in_seq=0 out_seq=0 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.732+0000 7fa222000640 10 client.27224 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa222000640 1 -- 172.21.15.22:0/3618720313 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561733d957e0 con 0x5617341488c0 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa222000640 10 client.27224 handle_command_reply: tid=0 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 client.27224 shutdown 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 msgr2=0x561734135cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.738 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 0x561734135cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009fd0 tx=0x7fa22c004a40 comp rx=0 tx=0).stop 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 shutdown_connections 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341488c0 0x56173414ace0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617341382d0 0x561734135cf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x5617341357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa222000640 0 client.27224 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3618720313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x561734135270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 >> 172.21.15.22:0/3618720313 conn(0x5617340f5b30 msgr2=0x56173412b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 shutdown_connections 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3618720313 wait complete. 2024-03-24T06:47:46.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.733+0000 7fa23e1f3740 20 client.27224 trim_cache size 0 max 16384 2024-03-24T06:47:46.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.753+0000 7fa23e1f3740 1 Processor -- start 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.753+0000 7fa23e1f3740 1 -- start start 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.753+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x561734143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x5617341505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561734147030 con 0x5617341382d0 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561734129f30 con 0x56173412fc40 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561734145270 con 0x56173412ef60 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa22be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x561734143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:33896/0 (socket says 172.21.15.22:33896) 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 -- 172.21.15.22:0/3755603661 learned_addr learned my addr 172.21.15.22:0/3755603661 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa239400640 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x5617341505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 -- 172.21.15.22:0/3755603661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 msgr2=0x561734143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x561734143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 -- 172.21.15.22:0/3755603661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 msgr2=0x5617341505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x5617341505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.754+0000 7fa238a00640 1 -- 172.21.15.22:0/3755603661 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617340309a0 con 0x56173412ef60 2024-03-24T06:47:46.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa238a00640 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa224006010 tx=0x7fa224041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfcad7c0dd0d866c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa229200640 1 -- 172.21.15.22:0/3755603661 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22404d070 con 0x56173412ef60 2024-03-24T06:47:46.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa229200640 1 -- 172.21.15.22:0/3755603661 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa224050740 con 0x56173412ef60 2024-03-24T06:47:46.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa229200640 1 -- 172.21.15.22:0/3755603661 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2240053d0 con 0x56173412ef60 2024-03-24T06:47:46.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3755603661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 msgr2=0x5617341294e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.755+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa224006010 tx=0x7fa224041040 comp rx=0 tx=0).stop 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3755603661 shutdown_connections 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5617341382d0 0x5617341505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412fc40 0x561734143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3755603661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56173412ef60 0x5617341294e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3755603661 >> 172.21.15.22:0/3755603661 conn(0x5617340f5b30 msgr2=0x561734128bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3755603661 shutdown_connections 2024-03-24T06:47:46.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3755603661 wait complete. 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 -- start start 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.756+0000 7fa23e1f3740 1 Processor -- start 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x561734132440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 0x561734132980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5617341adbe0 con 0x56173412fc40 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617341add20 con 0x56173412ef60 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5617341adee0 con 0x5617341382d0 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa23e1f3740 10 client.0 _ll_register_callbacks cb 0x7fa22c009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:47:46.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa239400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 0x561734132980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:46700/0 (socket says 172.21.15.22:46700) 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 -- 172.21.15.22:0/3759595596 learned_addr learned my addr 172.21.15.22:0/3759595596 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa22be00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x561734132440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 -- 172.21.15.22:0/3759595596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 msgr2=0x561734132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 0x561734132980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 -- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 msgr2=0x561734132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:47:46.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x561734132440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa238a00640 1 -- 172.21.15.22:0/3759595596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561734057c20 con 0x56173412ef60 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.757+0000 7fa239400640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 0x561734132980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa238a00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa22402c320 tx=0x7fa22402c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=765aae8c9d8a1bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa222000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22404d040 con 0x56173412ef60 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa224004a60 con 0x56173412ef60 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9467+0+0 (secure 0 0 0) 0x7fa224050260 con 0x56173412ef60 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa224050420 con 0x56173412ef60 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa23e1f3740 10 client.27230 fetch_fsmap learned FSMap version 37 2024-03-24T06:47:46.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.758+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561734056480 con 0x56173412ef60 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa224004a60 con 0x56173412ef60 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa22403c050 con 0x56173412ef60 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 10 client.27230 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 10 client.27230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 20 client.27230 populate_metadata read hostname 'smithi022' 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341af380 0x5617341b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 4 client.27230 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:47:46.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561733dd2720 con 0x5617341af380 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.759+0000 7fa22be00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341af380 0x5617341b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa22be00640 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341af380 0x5617341b17a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63e276fc425b078b server_cookie=995f9e26c3941524 in_seq=0 out_seq=0 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa222000640 10 client.27230 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa222000640 1 -- 172.21.15.22:0/3759595596 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561733dd2720 con 0x5617341af380 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa222000640 10 client.27230 handle_command_reply: tid=0 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 client.27230 shutdown 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 msgr2=0x561734131f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa22402c320 tx=0x7fa22402c740 comp rx=0 tx=0).stop 2024-03-24T06:47:46.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 shutdown_connections 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5617341382d0 0x561734132980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56173412fc40 0x561734132440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5617341af380 0x5617341b17a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 --2- 172.21.15.22:0/3759595596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56173412ef60 0x561734131f00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 >> 172.21.15.22:0/3759595596 conn(0x5617340f5b30 msgr2=0x561734136920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.760+0000 7fa222000640 0 client.27230 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.761+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 shutdown_connections 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.761+0000 7fa23e1f3740 1 -- 172.21.15.22:0/3759595596 wait complete. 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.761+0000 7fa23e1f3740 20 client.27230 trim_cache size 0 max 16384 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:47:46.769 INFO:teuthology.orchestra.run.smithi022.stdout: "2c774ba3-d4d0-40b1-ad65-711cd0a26e4f": { 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "2c774ba3-d4d0-40b1-ad65-711cd0a26e4f", 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:47:46.770 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 msgr2=0x7fa2180a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 0x7fa2180a4770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009270 tx=0x7fa22c009680 comp rx=0 tx=0).stop 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 msgr2=0x7fa23419e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00e7c0 tx=0x7fa21c00ec80 comp rx=0 tx=0).stop 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 shutdown_connections 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa2180a22b0 0x7fa2180a4770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa234105930 0x7fa2341a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa234100e50 0x7fa23419e720 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 --2- 172.21.15.22:0/3553016065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa234100530 0x7fa23419e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:47:46.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 >> 172.21.15.22:0/3553016065 conn(0x7fa2340fbf60 msgr2=0x7fa2341032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:47:46.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 shutdown_connections 2024-03-24T06:47:46.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:47:46.762+0000 7fa239e00640 1 -- 172.21.15.22:0/3553016065 wait complete. 2024-03-24T06:47:46.781 INFO:tasks.cephfs.filesystem:scrub status for tag:2c774ba3-d4d0-40b1-ad65-711cd0a26e4f - {'path': '/', 'tag': '2c774ba3-d4d0-40b1-ad65-711cd0a26e4f', 'options': 'recursive,force'} 2024-03-24T06:47:47.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:47.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:47 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:47:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:47.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:47 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:47:47.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:47.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:47 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:48 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:47:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:48 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:48.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:48 smithi022 ceph-mon[32003]: pgmap v240: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:48 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:47:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:48 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:48 smithi143 ceph-mon[36520]: pgmap v240: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:48 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:47:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:48 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:47:48.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:48 smithi155 ceph-mon[36262]: pgmap v240: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:47:50.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:50 smithi022 ceph-mon[32003]: pgmap v241: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:50 smithi143 ceph-mon[36520]: pgmap v241: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:50 smithi155 ceph-mon[36262]: pgmap v241: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:47:52.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:52 smithi022 ceph-mon[32003]: pgmap v242: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:47:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:52 smithi143 ceph-mon[36520]: pgmap v242: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:47:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:52 smithi155 ceph-mon[36262]: pgmap v242: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:47:54.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:54 smithi022 ceph-mon[32003]: pgmap v243: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:54 smithi143 ceph-mon[36520]: pgmap v243: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:54 smithi155 ceph-mon[36262]: pgmap v243: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:56 smithi022 ceph-mon[32003]: pgmap v244: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:47:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:56 smithi143 ceph-mon[36520]: pgmap v244: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:47:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:56 smithi155 ceph-mon[36262]: pgmap v244: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:47:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:58 smithi022 ceph-mon[32003]: pgmap v245: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:58.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:58 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:58.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:58 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:58.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:58 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:58.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:58 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:47:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:58 smithi143 ceph-mon[36520]: pgmap v245: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:58 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:58 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:58 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:58.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:58 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:58 smithi155 ceph-mon[36262]: pgmap v245: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:47:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:58 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:58 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:58 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:58 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:59.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:47:59 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:47:59.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:47:59 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:47:59.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:47:59 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:00.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:00 smithi143 ceph-mon[36520]: pgmap v246: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:48:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:00 smithi155 ceph-mon[36262]: pgmap v246: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:48:00.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:00 smithi022 ceph-mon[32003]: pgmap v246: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:48:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:02 smithi143 ceph-mon[36520]: pgmap v247: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:48:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:02 smithi155 ceph-mon[36262]: pgmap v247: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:48:02.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:02 smithi022 ceph-mon[32003]: pgmap v247: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:48:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:04 smithi143 ceph-mon[36520]: pgmap v248: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:48:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:04 smithi155 ceph-mon[36262]: pgmap v248: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:48:04.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:04 smithi022 ceph-mon[32003]: pgmap v248: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:48:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:06 smithi143 ceph-mon[36520]: pgmap v249: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:48:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:06 smithi155 ceph-mon[36262]: pgmap v249: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:48:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:06 smithi022 ceph-mon[32003]: pgmap v249: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:48:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:08 smithi143 ceph-mon[36520]: pgmap v250: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:48:08.535 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:08 smithi155 ceph-mon[36262]: pgmap v250: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:48:08.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:08 smithi022 ceph-mon[32003]: pgmap v250: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:48:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:09 smithi155 ceph-mon[36262]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T06:48:09.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:09 smithi022 ceph-mon[32003]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T06:48:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:09 smithi143 ceph-mon[36520]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T06:48:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:10 smithi155 ceph-mon[36262]: pgmap v252: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:10.550 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:10 smithi022 ceph-mon[32003]: pgmap v252: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:10.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:10 smithi143 ceph-mon[36520]: pgmap v252: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:12 smithi155 ceph-mon[36262]: pgmap v253: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:12.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:12 smithi022 ceph-mon[32003]: pgmap v253: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:12.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:12 smithi143 ceph-mon[36520]: pgmap v253: 129 pgs: 6 active+clean+snaptrim, 106 active+clean, 17 active+clean+snaptrim_wait; 17 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:48:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:14 smithi155 ceph-mon[36262]: pgmap v254: 129 pgs: 8 active+clean+snaptrim, 97 active+clean, 24 active+clean+snaptrim_wait; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:48:14.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:14 smithi022 ceph-mon[32003]: pgmap v254: 129 pgs: 8 active+clean+snaptrim, 97 active+clean, 24 active+clean+snaptrim_wait; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:48:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:14 smithi143 ceph-mon[36520]: pgmap v254: 129 pgs: 8 active+clean+snaptrim, 97 active+clean, 24 active+clean+snaptrim_wait; 15 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:48:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:16 smithi155 ceph-mon[36262]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:16.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:16 smithi022 ceph-mon[32003]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:16.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:16 smithi143 ceph-mon[36520]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 14 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:16.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:48:16.782 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:48:17.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 -- 172.21.15.22:0/673033729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc102740 msgr2=0x7f3ecc102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 --2- 172.21.15.22:0/673033729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc102740 0x7f3ecc102b20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009900 tx=0x7f3ec002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 -- 172.21.15.22:0/673033729 shutdown_connections 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 --2- 172.21.15.22:0/673033729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc10bb60 0x7f3ecc10e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 --2- 172.21.15.22:0/673033729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc103060 0x7f3ecc10b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 --2- 172.21.15.22:0/673033729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc102740 0x7f3ecc102b20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 -- 172.21.15.22:0/673033729 >> 172.21.15.22:0/673033729 conn(0x7f3ecc0fbf60 msgr2=0x7f3ecc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 -- 172.21.15.22:0/673033729 shutdown_connections 2024-03-24T06:48:17.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.017+0000 7f3ed2400640 1 -- 172.21.15.22:0/673033729 wait complete. 2024-03-24T06:48:17.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.018+0000 7f3ed2400640 1 Processor -- start 2024-03-24T06:48:17.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.018+0000 7f3ed2400640 1 -- start start 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc10bb60 0x7f3ecc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc198040 con 0x7f3ecc102740 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1981b0 con 0x7f3ecc10bb60 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed2400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc198370 con 0x7f3ecc103060 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed1a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc10bb60 0x7f3ecc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59584/0 (socket says 172.21.15.22:59584) 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60060/0 (socket says 172.21.15.22:60060) 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 -- 172.21.15.22:0/199966077 learned_addr learned my addr 172.21.15.22:0/199966077 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 -- 172.21.15.22:0/199966077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc10bb60 msgr2=0x7f3ecc1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc10bb60 0x7f3ecc1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 -- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 msgr2=0x7f3ecc19d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ecbe00640 1 -- 172.21.15.22:0/199966077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec0009560 con 0x7f3ecc103060 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.019+0000 7f3ed1000640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9f0 tx=0x7f3eb800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ec9600640 1 -- 172.21.15.22:0/199966077 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8009670 con 0x7f3ecc103060 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ec9600640 1 -- 172.21.15.22:0/199966077 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8005500 con 0x7f3ecc103060 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ec9600640 1 -- 172.21.15.22:0/199966077 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8010430 con 0x7f3ecc103060 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc198650 con 0x7f3ecc103060 2024-03-24T06:48:17.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.020+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1004a0 con 0x7f3ecc103060 2024-03-24T06:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 Processor -- start 2024-03-24T06:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 -- start start 2024-03-24T06:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ec9600640 1 -- 172.21.15.22:0/199966077 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3eb8005020 con 0x7f3ecc103060 2024-03-24T06:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd473e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563d4bcb5790 con 0x563d4bd46f60 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563d4bcb2cc0 con 0x563d4bd502d0 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563d4bcae4a0 con 0x563d4bd47c40 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed1a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59588/0 (socket says 172.21.15.22:59588) 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ed1a00640 1 -- 172.21.15.22:0/3502808614 learned_addr learned my addr 172.21.15.22:0/3502808614 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.024+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1000640 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 -- 172.21.15.22:0/3502808614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 msgr2=0x563d4bd526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 -- 172.21.15.22:0/3502808614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 msgr2=0x563d4bd473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 -- 172.21.15.22:0/3502808614 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d4bc6fc20 con 0x563d4bd47c40 2024-03-24T06:48:17.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009f50 tx=0x7f3ec4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d198dae64e36a19d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.025+0000 7f3ebe000640 1 -- 172.21.15.22:0/3502808614 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4015070 con 0x563d4bd47c40 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ebe000640 1 -- 172.21.15.22:0/3502808614 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec4016040 con 0x563d4bd47c40 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ebe000640 1 -- 172.21.15.22:0/3502808614 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400b900 con 0x563d4bd47c40 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ec9600640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 0x7f3eb40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ec9600640 1 -- 172.21.15.22:0/199966077 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f3eb8014070 con 0x7f3ecc103060 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ed1000640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 0x7f3eb40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ed67ae740 1 -- 172.21.15.22:0/3502808614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 msgr2=0x563d4bd46990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.026+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009f50 tx=0x7f3ec4009500 comp rx=0 tx=0).stop 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 -- 172.21.15.22:0/3502808614 shutdown_connections 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/3502808614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd46990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 -- 172.21.15.22:0/3502808614 >> 172.21.15.22:0/3502808614 conn(0x563d4bd0db30 msgr2=0x563d4bd419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 -- 172.21.15.22:0/3502808614 shutdown_connections 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed67ae740 1 -- 172.21.15.22:0/3502808614 wait complete. 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.027+0000 7f3ed1000640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 0x7f3eb40a4770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0005ab0 tx=0x7f3ec0005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.028+0000 7f3ed67ae740 1 -- start start 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.028+0000 7f3ed67ae740 1 Processor -- start 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 0x563d4bd4d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd502d0 0x563d4bd4dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563d4bc52f20 con 0x563d4bd47c40 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563d4bc52bc0 con 0x563d4bd46f60 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563d4ba1c1c0 con 0x563d4bd502d0 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed67ae740 10 client.0 _ll_register_callbacks cb 0x563d4b9bebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:17.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40682/0 (socket says 172.21.15.22:40682) 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 -- 172.21.15.22:0/2891772468 learned_addr learned my addr 172.21.15.22:0/2891772468 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 0x563d4bd4d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd502d0 0x563d4bd4dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 -- 172.21.15.22:0/2891772468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd502d0 msgr2=0x563d4bd4dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd502d0 0x563d4bd4dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 -- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 msgr2=0x563d4bd4d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 0x563d4bd4d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ed1000640 1 -- 172.21.15.22:0/2891772468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec4009c00 con 0x563d4bd46f60 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.029+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 0x563d4bd4d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0035000 tx=0x7f3ec0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf9af4844dede13e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3eb2a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec002cdc0 con 0x563d4bd46f60 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563d4bca6160 con 0x563d4bd46f60 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f3ec0004df0 con 0x563d4bd46f60 2024-03-24T06:48:17.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3ec0056910 con 0x563d4bd46f60 2024-03-24T06:48:17.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3ed67ae740 10 client.27242 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:17.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.030+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563d4bcacb20 con 0x563d4bd46f60 2024-03-24T06:48:17.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563d4bca6160 con 0x563d4bd46f60 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3ec003c040 con 0x563d4bd46f60 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 10 client.27242 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 10 client.27242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 20 client.27242 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bd60820 0x563d4bd62c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 4 client.27242 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d4b9ad7e0 con 0x563d4bd60820 2024-03-24T06:48:17.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.031+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bd60820 0x563d4bd62c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.032+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bd60820 0x563d4bd62c80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=890c9d9e124fc1cb server_cookie=3496ec766ea1b6ff in_seq=0 out_seq=0 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.032+0000 7f3eb2a00640 10 client.27242 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2891772468 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563d4b9ad7e0 con 0x563d4bd60820 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3eb2a00640 10 client.27242 handle_command_reply: tid=0 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 client.27242 shutdown 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 msgr2=0x563d4bd4d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0035000 tx=0x7f3ec0009260 comp rx=0 tx=0).stop 2024-03-24T06:48:17.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 shutdown_connections 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd502d0 0x563d4bd4dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd47c40 0x563d4bd4d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bd60820 0x563d4bd62c80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2891772468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd4d260 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 >> 172.21.15.22:0/2891772468 conn(0x563d4bd0db30 msgr2=0x563d4bd43700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.033+0000 7f3eb2a00640 0 client.27242 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.034+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 shutdown_connections 2024-03-24T06:48:17.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.034+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2891772468 wait complete. 2024-03-24T06:48:17.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.034+0000 7f3ed67ae740 20 client.27242 trim_cache size 0 max 16384 2024-03-24T06:48:17.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 Processor -- start 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 -- start start 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd414e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd4db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563d4bd3d5f0 con 0x563d4bd46f60 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563d4bd45f90 con 0x563d4bd502d0 2024-03-24T06:48:17.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.053+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563d4bd4b7f0 con 0x563d4bd47c40 2024-03-24T06:48:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd4db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59600/0 (socket says 172.21.15.22:59600) 2024-03-24T06:48:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ecbe00640 1 -- 172.21.15.22:0/233279345 learned_addr learned my addr 172.21.15.22:0/233279345 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1000640 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 -- 172.21.15.22:0/233279345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 msgr2=0x563d4bd5bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 -- 172.21.15.22:0/233279345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 msgr2=0x563d4bd414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 -- 172.21.15.22:0/233279345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d4bc489a0 con 0x563d4bd502d0 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.054+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd4db00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009f20 tx=0x7f3ec400b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=641df0ca42e0a754 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ebd600640 1 -- 172.21.15.22:0/233279345 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4015040 con 0x563d4bd502d0 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ebd600640 1 -- 172.21.15.22:0/233279345 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec4004dc0 con 0x563d4bd502d0 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ebd600640 1 -- 172.21.15.22:0/233279345 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec400b5b0 con 0x563d4bd502d0 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 -- 172.21.15.22:0/233279345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 msgr2=0x563d4bd4db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd4db00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009f20 tx=0x7f3ec400b540 comp rx=0 tx=0).stop 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 -- 172.21.15.22:0/233279345 shutdown_connections 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd502d0 0x563d4bd4db00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd5bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.055+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/233279345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd46f60 0x563d4bd414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.056+0000 7f3ed67ae740 1 -- 172.21.15.22:0/233279345 >> 172.21.15.22:0/233279345 conn(0x563d4bd0db30 msgr2=0x563d4bd40bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.056+0000 7f3ed67ae740 1 -- 172.21.15.22:0/233279345 shutdown_connections 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.056+0000 7f3ed67ae740 1 -- 172.21.15.22:0/233279345 wait complete. 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.056+0000 7f3ed67ae740 1 -- start start 2024-03-24T06:48:17.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.056+0000 7f3ed67ae740 1 Processor -- start 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd55220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 0x563d4bd55760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x563d4bd55e60 con 0x563d4bd502d0 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563d4bd55fd0 con 0x563d4bd46f60 2024-03-24T06:48:17.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563d4bdc6bd0 con 0x563d4bd47c40 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed67ae740 10 client.0 _ll_register_callbacks cb 0x7f3ec0043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd55220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40692/0 (socket says 172.21.15.22:40692) 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.057+0000 7f3ed1000640 1 -- 172.21.15.22:0/2376275231 learned_addr learned my addr 172.21.15.22:0/2376275231 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 0x563d4bd55760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 -- 172.21.15.22:0/2376275231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 msgr2=0x563d4bd55220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd55220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 -- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 msgr2=0x563d4bd55760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 0x563d4bd55760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 -- 172.21.15.22:0/2376275231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1004a0 con 0x563d4bd46f60 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1a00640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 0x563d4bd55760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd55220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed1000640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00037b0 tx=0x7f3ec0005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea21cd5a79de6df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3eb2a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00055a0 con 0x563d4bd46f60 2024-03-24T06:48:17.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f3ec0003830 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.058+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563d4bd41b40 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3ec00039f0 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 10 client.27248 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563d4bc6fc20 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563d4bd41b40 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3ec003c040 con 0x563d4bd46f60 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 10 client.27248 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:17.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 20 client.27248 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bdc80f0 0x563d4bdca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 4 client.27248 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.059+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f3ec4016040 con 0x563d4bdc80f0 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.060+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bdc80f0 0x563d4bdca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.060+0000 7f3ecbe00640 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bdc80f0 0x563d4bdca510 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7952243a1a49eb74 server_cookie=bbf7e369af09c8a4 in_seq=0 out_seq=0 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.060+0000 7f3eb2a00640 10 client.27248 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3eb2a00640 1 -- 172.21.15.22:0/2376275231 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f3ec4016040 con 0x563d4bdc80f0 2024-03-24T06:48:17.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3eb2a00640 10 client.27248 handle_command_reply: tid=0 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 client.27248 shutdown 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 msgr2=0x563d4bd58520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00037b0 tx=0x7f3ec0005020 comp rx=0 tx=0).stop 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 shutdown_connections 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x563d4bd502d0 0x563d4bd55760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563d4bd47c40 0x563d4bd55220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x563d4bdc80f0 0x563d4bdca510 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 --2- 172.21.15.22:0/2376275231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563d4bd46f60 0x563d4bd58520 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3eb2a00640 0 client.27248 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 >> 172.21.15.22:0/2376275231 conn(0x563d4bd0db30 msgr2=0x563d4bd42cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 shutdown_connections 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 1 -- 172.21.15.22:0/2376275231 wait complete. 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.061+0000 7f3ed67ae740 20 client.27248 trim_cache size 0 max 16384 2024-03-24T06:48:17.069 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:17.070 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:48:17.070 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:48:17.070 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 msgr2=0x7f3eb40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 0x7f3eb40a4770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0005ab0 tx=0x7f3ec0005a20 comp rx=0 tx=0).stop 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 msgr2=0x7f3ecc19dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9f0 tx=0x7f3eb800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 shutdown_connections 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3eb40a22b0 0x7f3eb40a4770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ecc10bb60 0x7f3ecc1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ecc103060 0x7f3ecc19dd80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 --2- 172.21.15.22:0/199966077 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ecc102740 0x7f3ecc19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 >> 172.21.15.22:0/199966077 conn(0x7f3ecc0fbf60 msgr2=0x7f3ecc0fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 shutdown_connections 2024-03-24T06:48:17.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.062+0000 7f3ed2400640 1 -- 172.21.15.22:0/199966077 wait complete. 2024-03-24T06:48:17.081 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:48:17.082 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 -- 172.21.15.22:0/3845668592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 msgr2=0x7f4b10073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 --2- 172.21.15.22:0/3845668592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b10073e90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009900 tx=0x7f4b0002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 -- 172.21.15.22:0/3845668592 shutdown_connections 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 --2- 172.21.15.22:0/3845668592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b101126b0 0x7f4b10114aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 --2- 172.21.15.22:0/3845668592 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b100743d0 0x7f4b10112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 --2- 172.21.15.22:0/3845668592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b10073e90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 -- 172.21.15.22:0/3845668592 >> 172.21.15.22:0/3845668592 conn(0x7f4b100fd1b0 msgr2=0x7f4b100ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 -- 172.21.15.22:0/3845668592 shutdown_connections 2024-03-24T06:48:17.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.297+0000 7f4b16800640 1 -- 172.21.15.22:0/3845668592 wait complete. 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.298+0000 7f4b16800640 1 Processor -- start 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.298+0000 7f4b16800640 1 -- start start 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 0x7f4b1019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b101126b0 0x7f4b101a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f4b101a3260 con 0x7f4b101126b0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4b101a33d0 con 0x7f4b10073ab0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b16800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b101a3590 con 0x7f4b100743d0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b0f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b0f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40704/0 (socket says 172.21.15.22:40704) 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b0f400640 1 -- 172.21.15.22:0/958241379 learned_addr learned my addr 172.21.15.22:0/958241379 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b0ea00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 0x7f4b1019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.299+0000 7f4b0fe00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b101126b0 0x7f4b101a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 -- 172.21.15.22:0/958241379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 msgr2=0x7f4b1019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 0x7f4b1019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 -- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b101126b0 msgr2=0x7f4b101a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b101126b0 0x7f4b101a2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 -- 172.21.15.22:0/958241379 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b00009560 con 0x7f4b10073ab0 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0ea00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 0x7f4b1019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b0f400640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4b00000c00 tx=0x7f4b0002c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0002dbe0 con 0x7f4b10073ab0 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b0002dd40 con 0x7f4b10073ab0 2024-03-24T06:48:17.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00003b20 con 0x7f4b10073ab0 2024-03-24T06:48:17.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b101a3810 con 0x7f4b10073ab0 2024-03-24T06:48:17.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.300+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b101ab0f0 con 0x7f4b10073ab0 2024-03-24T06:48:17.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.302+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1019f6f0 con 0x7f4b10073ab0 2024-03-24T06:48:17.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.303+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f4b00003c80 con 0x7f4b10073ab0 2024-03-24T06:48:17.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.305+0000 7f4b07e00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 0x7f4af40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.305+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f4b000f0890 con 0x7f4b10073ab0 2024-03-24T06:48:17.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.305+0000 7f4b0ea00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 0x7f4af40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.306+0000 7f4b0ea00640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 0x7f4af40a4770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4b100ff8b0 tx=0x7f4af8005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.309+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b000b9250 con 0x7f4b10073ab0 2024-03-24T06:48:17.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.655+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b10068900 con 0x7f4b10073ab0 2024-03-24T06:48:17.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.656+0000 7f4b07e00640 1 -- 172.21.15.22:0/958241379 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f4b000bc910 con 0x7f4b10073ab0 2024-03-24T06:48:17.657 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:48:17.657 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:48:17.657 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:48:17.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 msgr2=0x7f4af40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 0x7f4af40a4770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4b100ff8b0 tx=0x7f4af8005f90 comp rx=0 tx=0).stop 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 msgr2=0x7f4b1019e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4b00000c00 tx=0x7f4b0002c6d0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 shutdown_connections 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f4af40a22b0 0x7f4af40a4770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b101126b0 0x7f4b101a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b100743d0 0x7f4b1019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 --2- 172.21.15.22:0/958241379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b10073ab0 0x7f4b1019e3a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 >> 172.21.15.22:0/958241379 conn(0x7f4b100fd1b0 msgr2=0x7f4b100fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 shutdown_connections 2024-03-24T06:48:17.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.659+0000 7f4b16800640 1 -- 172.21.15.22:0/958241379 wait complete. 2024-03-24T06:48:17.679 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:48:17.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 -- 172.21.15.22:0/3076556205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c10bb60 msgr2=0x7fb39c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 --2- 172.21.15.22:0/3076556205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c10bb60 0x7fb39c10e000 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb394009be0 tx=0x7fb39402bf80 comp rx=0 tx=0).stop 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 -- 172.21.15.22:0/3076556205 shutdown_connections 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 --2- 172.21.15.22:0/3076556205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c10bb60 0x7fb39c10e000 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 --2- 172.21.15.22:0/3076556205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c103060 0x7fb39c10b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 --2- 172.21.15.22:0/3076556205 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c102740 0x7fb39c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.908+0000 7fb3a2600640 1 -- 172.21.15.22:0/3076556205 >> 172.21.15.22:0/3076556205 conn(0x7fb39c0fbf60 msgr2=0x7fb39c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.909+0000 7fb3a2600640 1 -- 172.21.15.22:0/3076556205 shutdown_connections 2024-03-24T06:48:17.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.909+0000 7fb3a2600640 1 -- 172.21.15.22:0/3076556205 wait complete. 2024-03-24T06:48:17.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.910+0000 7fb3a2600640 1 Processor -- start 2024-03-24T06:48:17.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.910+0000 7fb3a2600640 1 -- start start 2024-03-24T06:48:17.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c19ec80 con 0x7fb39c10bb60 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c19edf0 con 0x7fb39c102740 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a2600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c19efb0 con 0x7fb39c103060 2024-03-24T06:48:17.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb39be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb39be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59642/0 (socket says 172.21.15.22:59642) 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb39be00640 1 -- 172.21.15.22:0/1042197698 learned_addr learned my addr 172.21.15.22:0/1042197698 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60130/0 (socket says 172.21.15.22:60130) 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.911+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40730/0 (socket says 172.21.15.22:40730) 2024-03-24T06:48:17.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 -- 172.21.15.22:0/1042197698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 msgr2=0x7fb39c199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 -- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 msgr2=0x7fb39c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 -- 172.21.15.22:0/1042197698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb388009560 con 0x7fb39c103060 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb3a1200640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb39be00640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb38800e910 tx=0x7fb38800edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb399600640 1 -- 172.21.15.22:0/1042197698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38800cd80 con 0x7fb39c103060 2024-03-24T06:48:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb399600640 1 -- 172.21.15.22:0/1042197698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3880035f0 con 0x7fb39c103060 2024-03-24T06:48:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb394009840 con 0x7fb39c103060 2024-03-24T06:48:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.912+0000 7fb399600640 1 -- 172.21.15.22:0/1042197698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388010690 con 0x7fb39c103060 2024-03-24T06:48:17.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.913+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c1a6930 con 0x7fb39c103060 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.916+0000 7fb399600640 1 -- 172.21.15.22:0/1042197698 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb388003760 con 0x7fb39c103060 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.916+0000 7fb3a698b740 1 Processor -- start 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.916+0000 7fb3a698b740 1 -- start start 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.916+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.916+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7ca3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a698b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561fb7c066e0 con 0x561fb7ca1950 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a698b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561fb7c03c10 con 0x561fb7c98d70 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a698b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fb7bff3f0 con 0x561fb7c98450 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59654/0 (socket says 172.21.15.22:59654) 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 -- 172.21.15.22:0/450851517 learned_addr learned my addr 172.21.15.22:0/450851517 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1200640 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb39be00640 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7ca3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 -- 172.21.15.22:0/450851517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 msgr2=0x561fb7ca1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 -- 172.21.15.22:0/450851517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 msgr2=0x561fb7ca3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7ca3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb3a1c00640 1 -- 172.21.15.22:0/450851517 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb7bc0e40 con 0x561fb7c98450 2024-03-24T06:48:17.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.917+0000 7fb399600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 0x7fb3840a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb394009be0 tx=0x7fb394033e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ba200968322e5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb399600640 1 -- 172.21.15.22:0/1042197698 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fb388014070 con 0x7fb39c103060 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb3a1200640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 0x7fb3840a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb38e000640 1 -- 172.21.15.22:0/450851517 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39402c920 con 0x561fb7c98450 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb38e000640 1 -- 172.21.15.22:0/450851517 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3940338c0 con 0x561fb7c98450 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.918+0000 7fb38e000640 1 -- 172.21.15.22:0/450851517 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394004200 con 0x561fb7c98450 2024-03-24T06:48:17.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- 172.21.15.22:0/450851517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 msgr2=0x561fb7c98830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a1200640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 0x7fb3840a4770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x561fb7c97f20 tx=0x7fb390005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb394009be0 tx=0x7fb394033e30 comp rx=0 tx=0).stop 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- 172.21.15.22:0/450851517 shutdown_connections 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7ca3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 --2- 172.21.15.22:0/450851517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c98830 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- 172.21.15.22:0/450851517 >> 172.21.15.22:0/450851517 conn(0x561fb7c92b80 msgr2=0x561fb7c92f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- 172.21.15.22:0/450851517 shutdown_connections 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- 172.21.15.22:0/450851517 wait complete. 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.919+0000 7fb3a698b740 1 -- start start 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 Processor -- start 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 0x561fb7c9ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7ca1950 0x561fb7c9f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561fb7ba4060 con 0x561fb7c98d70 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561fb7ba3d00 con 0x561fb7ca1950 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fb796d1c0 con 0x561fb7c98450 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a698b740 10 client.0 _ll_register_callbacks cb 0x561fb790fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59660/0 (socket says 172.21.15.22:59660) 2024-03-24T06:48:17.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a1200640 1 -- 172.21.15.22:0/19969834 learned_addr learned my addr 172.21.15.22:0/19969834 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb39be00640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 0x561fb7c9ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.920+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7ca1950 0x561fb7c9f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 -- 172.21.15.22:0/19969834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7ca1950 msgr2=0x561fb7c9f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7ca1950 0x561fb7c9f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 -- 172.21.15.22:0/19969834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 msgr2=0x561fb7c9ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 0x561fb7c9ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 -- 172.21.15.22:0/19969834 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c10ec80 con 0x561fb7c98450 2024-03-24T06:48:17.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb39be00640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 0x561fb7c9ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb3a1200640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb390009560 tx=0x7fb390004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2efd363f27f49eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb382a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.921+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390023070 con 0x561fb7c98450 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fb390004020 con 0x561fb7c98450 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561fb78fe7e0 con 0x561fb7c98450 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb390017070 con 0x561fb7c98450 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 10 client.27295 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:17.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561fb7bfda70 con 0x561fb7c98450 2024-03-24T06:48:17.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561fb78fe7e0 con 0x561fb7c98450 2024-03-24T06:48:17.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb390003680 con 0x561fb7c98450 2024-03-24T06:48:17.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 10 client.27295 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:17.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 10 client.27295 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:48:17.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 20 client.27295 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7cb1ea0 0x561fb7cb42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 4 client.27295 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.922+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fb793b720 con 0x561fb7cb1ea0 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.923+0000 7fb39be00640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7cb1ea0 0x561fb7cb42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.923+0000 7fb39be00640 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7cb1ea0 0x561fb7cb42e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46d984ff49f03dfb server_cookie=3666a4b1bff36c8e in_seq=0 out_seq=0 2024-03-24T06:48:17.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.923+0000 7fb382a00640 10 client.27295 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.924+0000 7fb382a00640 1 -- 172.21.15.22:0/19969834 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fb793b720 con 0x561fb7cb1ea0 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.924+0000 7fb382a00640 10 client.27295 handle_command_reply: tid=0 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.924+0000 7fb3a698b740 1 client.27295 shutdown 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 msgr2=0x561fb7c9e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb390009560 tx=0x7fb390004730 comp rx=0 tx=0).stop 2024-03-24T06:48:17.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 shutdown_connections 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7cb1ea0 0x561fb7cb42e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7ca1950 0x561fb7c9f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7c98d70 0x561fb7c9ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 --2- 172.21.15.22:0/19969834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7c9e9f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb382a00640 0 client.27295 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 >> 172.21.15.22:0/19969834 conn(0x561fb7c92b80 msgr2=0x561fb7c948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 shutdown_connections 2024-03-24T06:48:17.937 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 1 -- 172.21.15.22:0/19969834 wait complete. 2024-03-24T06:48:17.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.925+0000 7fb3a698b740 20 client.27295 trim_cache size 0 max 16384 2024-03-24T06:48:17.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 Processor -- start 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 -- start start 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 0x561fb7c8db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 0x561fb7cad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561fb7c94e90 con 0x561fb7ca1950 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561fb7c9a5c0 con 0x561fb7c98450 2024-03-24T06:48:17.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a698b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fb7c923d0 con 0x561fb7c98d70 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a1c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60150/0 (socket says 172.21.15.22:60150) 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a1c00640 1 -- 172.21.15.22:0/1629923721 learned_addr learned my addr 172.21.15.22:0/1629923721 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb3a1200640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 0x561fb7c8db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.940+0000 7fb39be00640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 0x561fb7cad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 -- 172.21.15.22:0/1629923721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 msgr2=0x561fb7cad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 0x561fb7cad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 -- 172.21.15.22:0/1629923721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 msgr2=0x561fb7c8db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 0x561fb7c8db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 -- 172.21.15.22:0/1629923721 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb7b99b00 con 0x561fb7ca1950 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb39be00640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 0x561fb7cad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1200640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 0x561fb7c8db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:17.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb394009be0 tx=0x7fb394005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f83738720340aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb38d600640 1 -- 172.21.15.22:0/1629923721 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39402cb20 con 0x561fb7ca1950 2024-03-24T06:48:17.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb38d600640 1 -- 172.21.15.22:0/1629923721 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39404e070 con 0x561fb7ca1950 2024-03-24T06:48:17.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.941+0000 7fb38d600640 1 -- 172.21.15.22:0/1629923721 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394004710 con 0x561fb7ca1950 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 -- 172.21.15.22:0/1629923721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 msgr2=0x561fb7cb9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb394009be0 tx=0x7fb394005fd0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 -- 172.21.15.22:0/1629923721 shutdown_connections 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7cb9bf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98d70 0x561fb7cad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 --2- 172.21.15.22:0/1629923721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98450 0x561fb7c8db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 -- 172.21.15.22:0/1629923721 >> 172.21.15.22:0/1629923721 conn(0x561fb7c92b80 msgr2=0x561fb7c91cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 -- 172.21.15.22:0/1629923721 shutdown_connections 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.942+0000 7fb3a698b740 1 -- 172.21.15.22:0/1629923721 wait complete. 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 -- start start 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 Processor -- start 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7d17220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561fb7ca6490 con 0x561fb7ca1950 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561fb7d17760 con 0x561fb7c98d70 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fb7d17920 con 0x561fb7c98450 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a698b740 10 client.0 _ll_register_callbacks cb 0x7fb390009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.943+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59678/0 (socket says 172.21.15.22:59678) 2024-03-24T06:48:17.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb39be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 -- 172.21.15.22:0/2202142723 learned_addr learned my addr 172.21.15.22:0/2202142723 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7d17220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 -- 172.21.15.22:0/2202142723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 msgr2=0x561fb7ca5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 -- 172.21.15.22:0/2202142723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 msgr2=0x561fb7d17220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7d17220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 -- 172.21.15.22:0/2202142723 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb7bc0e40 con 0x561fb7c98450 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb39be00640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1c00640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7d17220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb3a1200640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb390000c00 tx=0x7fb390003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed4a489a1aa3900d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb382a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.944+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390023070 con 0x561fb7c98450 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fb390003800 con 0x561fb7c98450 2024-03-24T06:48:17.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3940338c0 con 0x561fb7c98450 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb39001e070 con 0x561fb7c98450 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 10 client.27301 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561fb7bbf6a0 con 0x561fb7c98450 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3940338c0 con 0x561fb7c98450 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb39001e6a0 con 0x561fb7c98450 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 10 client.27301 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:17.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 10 client.27301 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 20 client.27301 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7d19100 0x561fb7d1b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 4 client.27301 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.945+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fb798a890 con 0x561fb7d19100 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb39be00640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7d19100 0x561fb7d1b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb39be00640 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7d19100 0x561fb7d1b4f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5ebc2328a8618ca0 server_cookie=98b9eb72493c0ecd in_seq=0 out_seq=0 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb382a00640 10 client.27301 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:17.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb382a00640 1 -- 172.21.15.22:0/2202142723 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fb798a890 con 0x561fb7d19100 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb382a00640 10 client.27301 handle_command_reply: tid=0 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.946+0000 7fb3a698b740 1 client.27301 shutdown 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 msgr2=0x561fb7ca57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb390000c00 tx=0x7fb390003ed0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 shutdown_connections 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x561fb7d19100 0x561fb7d1b4f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561fb7ca1950 0x561fb7d17220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561fb7c98d70 0x561fb7ca5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 --2- 172.21.15.22:0/2202142723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fb7c98450 0x561fb7ca57f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 >> 172.21.15.22:0/2202142723 conn(0x561fb7c92b80 msgr2=0x561fb7c93db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb382a00640 0 client.27301 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 shutdown_connections 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 1 -- 172.21.15.22:0/2202142723 wait complete. 2024-03-24T06:48:17.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a698b740 20 client.27301 trim_cache size 0 max 16384 2024-03-24T06:48:17.959 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:17.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 msgr2=0x7fb3840a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.947+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 0x7fb3840a4770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x561fb7c97f20 tx=0x7fb390005f90 comp rx=0 tx=0).stop 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 msgr2=0x7fb39c19a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb38800e910 tx=0x7fb38800edd0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 shutdown_connections 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb3840a22b0 0x7fb3840a4770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb39c10bb60 0x7fb39c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb39c103060 0x7fb39c19a300 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 --2- 172.21.15.22:0/1042197698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb39c102740 0x7fb39c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 >> 172.21.15.22:0/1042197698 conn(0x7fb39c0fbf60 msgr2=0x7fb39c10de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 shutdown_connections 2024-03-24T06:48:17.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:17.948+0000 7fb3a2600640 1 -- 172.21.15.22:0/1042197698 wait complete. 2024-03-24T06:48:17.966 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:48:18.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 -- 172.21.15.22:0/2949160369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 msgr2=0x7f326410b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 --2- 172.21.15.22:0/2949160369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326410b750 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3250009860 tx=0x7f325002bc10 comp rx=0 tx=0).stop 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 -- 172.21.15.22:0/2949160369 shutdown_connections 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 --2- 172.21.15.22:0/2949160369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 0x7f326410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 --2- 172.21.15.22:0/2949160369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326410b750 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 --2- 172.21.15.22:0/2949160369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 -- 172.21.15.22:0/2949160369 >> 172.21.15.22:0/2949160369 conn(0x7f3264076990 msgr2=0x7f3264076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.177+0000 7f326ba00640 1 -- 172.21.15.22:0/2949160369 shutdown_connections 2024-03-24T06:48:18.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.178+0000 7f326ba00640 1 -- 172.21.15.22:0/2949160369 wait complete. 2024-03-24T06:48:18.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.180+0000 7f326ba00640 1 Processor -- start 2024-03-24T06:48:18.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.180+0000 7f326ba00640 1 -- start start 2024-03-24T06:48:18.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 0x7f326419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f326419ee80 con 0x7f326410bd20 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f326419eff0 con 0x7f3264102950 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f326ba00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f326419f1b0 con 0x7f3264103270 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f3269000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f3269000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40778/0 (socket says 172.21.15.22:40778) 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f3269000640 1 -- 172.21.15.22:0/2720893703 learned_addr learned my addr 172.21.15.22:0/2720893703 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f3269a00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 0x7f326419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.181+0000 7f3263e00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 -- 172.21.15.22:0/2720893703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 msgr2=0x7f326419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 -- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 msgr2=0x7f326419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 0x7f326419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 -- 172.21.15.22:0/2720893703 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c009900 con 0x7f3264102950 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3269000640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9f0 tx=0x7f325800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3261600640 1 -- 172.21.15.22:0/2720893703 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258009720 con 0x7f3264102950 2024-03-24T06:48:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3250009490 con 0x7f3264102950 2024-03-24T06:48:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3261600640 1 -- 172.21.15.22:0/2720893703 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325800cd80 con 0x7f3264102950 2024-03-24T06:48:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f3261600640 1 -- 172.21.15.22:0/2720893703 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258010690 con 0x7f3264102950 2024-03-24T06:48:18.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.182+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32641a6b30 con 0x7f3264102950 2024-03-24T06:48:18.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.187+0000 7f3261600640 1 -- 172.21.15.22:0/2720893703 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f32580046c0 con 0x7f3264102950 2024-03-24T06:48:18.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.187+0000 7f326e77f740 1 Processor -- start 2024-03-24T06:48:18.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.187+0000 7f326e77f740 1 -- start start 2024-03-24T06:48:18.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415343240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415345b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614152a86e0 con 0x56141533aba0 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614152a5c10 con 0x56141533a280 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f326e77f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614152a13f0 con 0x561415343780 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40786/0 (socket says 172.21.15.22:40786) 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f3269a00640 1 -- 172.21.15.22:0/2244179716 learned_addr learned my addr 172.21.15.22:0/2244179716 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f3269000640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415343240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.188+0000 7f3263e00640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415345b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 -- 172.21.15.22:0/2244179716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 msgr2=0x561415345b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415345b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 -- 172.21.15.22:0/2244179716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 msgr2=0x561415343240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415343240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 -- 172.21.15.22:0/2244179716 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561415262e40 con 0x56141533a280 2024-03-24T06:48:18.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269000640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415343240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3261600640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 0x7f324c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269a00640 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f325c00f430 tx=0x7f325c00f8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=388ed0606dea5ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3261600640 1 -- 172.21.15.22:0/2720893703 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f3258014070 con 0x7f3264102950 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3269000640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 0x7f324c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.189+0000 7f3256000640 1 -- 172.21.15.22:0/2244179716 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c014070 con 0x56141533a280 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.190+0000 7f3256000640 1 -- 172.21.15.22:0/2244179716 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325c00cb20 con 0x56141533a280 2024-03-24T06:48:18.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.190+0000 7f3256000640 1 -- 172.21.15.22:0/2244179716 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c018c90 con 0x56141533a280 2024-03-24T06:48:18.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.190+0000 7f326e77f740 1 -- 172.21.15.22:0/2244179716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 msgr2=0x56141533a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.190+0000 7f3269000640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 0x7f324c0a46f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32580cbd90 tx=0x7f3258004b30 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.190+0000 7f326e77f740 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f325c00f430 tx=0x7f325c00f8f0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 -- 172.21.15.22:0/2244179716 shutdown_connections 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415345b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415343240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 --2- 172.21.15.22:0/2244179716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141533a660 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 -- 172.21.15.22:0/2244179716 >> 172.21.15.22:0/2244179716 conn(0x5614153349b0 msgr2=0x561415334d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 -- 172.21.15.22:0/2244179716 shutdown_connections 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 -- 172.21.15.22:0/2244179716 wait complete. 2024-03-24T06:48:18.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 -- start start 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.191+0000 7f326e77f740 1 Processor -- start 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561415343780 0x561415341270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561415246060 con 0x561415343780 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561415245d00 con 0x56141533a280 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56141500f1c0 con 0x56141533aba0 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f326e77f740 10 client.0 _ll_register_callbacks cb 0x561414fb1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3263e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40802/0 (socket says 172.21.15.22:40802) 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3269a00640 1 -- 172.21.15.22:0/1799819436 learned_addr learned my addr 172.21.15.22:0/1799819436 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3263e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59704/0 (socket says 172.21.15.22:59704) 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.192+0000 7f3269a00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561415343780 0x561415341270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 -- 172.21.15.22:0/1799819436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 msgr2=0x5614153407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 -- 172.21.15.22:0/1799819436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561415343780 msgr2=0x561415341270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561415343780 0x561415341270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 -- 172.21.15.22:0/1799819436 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c009ce0 con 0x56141533aba0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3269a00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f3263e00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3250037040 tx=0x7f32500043d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80f22cf2330323c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f324aa00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250034850 con 0x56141533aba0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f32500349b0 con 0x56141533aba0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561414fa07e0 con 0x56141533aba0 2024-03-24T06:48:18.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.193+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3250005030 con 0x56141533aba0 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 10 client.27313 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56141529fa70 con 0x56141533aba0 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561414fa07e0 con 0x56141533aba0 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3250045020 con 0x56141533aba0 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 10 client.27313 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 10 client.27313 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 20 client.27313 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x561415353c30 0x561415356050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 4 client.27313 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561414fdd720 con 0x561415353c30 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.194+0000 7f3269a00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x561415353c30 0x561415356050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.195+0000 7f3269a00640 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x561415353c30 0x561415356050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=81745a1bf5c0f22 server_cookie=140d38a08fd06d4 in_seq=0 out_seq=0 2024-03-24T06:48:18.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.195+0000 7f324aa00640 10 client.27313 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:18.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f324aa00640 1 -- 172.21.15.22:0/1799819436 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561414fdd720 con 0x561415353c30 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f324aa00640 10 client.27313 handle_command_reply: tid=0 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 client.27313 shutdown 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 msgr2=0x561415340d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3250037040 tx=0x7f32500043d0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 shutdown_connections 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x561415353c30 0x561415356050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x561415343780 0x561415341270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x561415340d30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 --2- 172.21.15.22:0/1799819436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x5614153407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f324aa00640 0 client.27313 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:18.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.196+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 >> 172.21.15.22:0/1799819436 conn(0x5614153349b0 msgr2=0x561415336700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.197+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 shutdown_connections 2024-03-24T06:48:18.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.197+0000 7f326e77f740 1 -- 172.21.15.22:0/1799819436 wait complete. 2024-03-24T06:48:18.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.197+0000 7f326e77f740 20 client.27313 trim_cache size 0 max 16384 2024-03-24T06:48:18.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 Processor -- start 2024-03-24T06:48:18.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 -- start start 2024-03-24T06:48:18.223 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x56141534f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561415343780 0x56141535b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5614153506e0 con 0x56141533a280 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56141534f550 con 0x561415343780 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.222+0000 7f326e77f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614153566b0 con 0x56141533aba0 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60206/0 (socket says 172.21.15.22:60206) 2024-03-24T06:48:18.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 -- 172.21.15.22:0/591867461 learned_addr learned my addr 172.21.15.22:0/591867461 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561415343780 0x56141535b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3263e00640 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x56141534f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 -- 172.21.15.22:0/591867461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 msgr2=0x56141534f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x56141534f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 -- 172.21.15.22:0/591867461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561415343780 msgr2=0x56141535b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561415343780 0x56141535b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 -- 172.21.15.22:0/591867461 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56141523bb00 con 0x56141533a280 2024-03-24T06:48:18.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.223+0000 7f3269a00640 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f325c005fd0 tx=0x7f325c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f1e5c48c825b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.224+0000 7f3255600640 1 -- 172.21.15.22:0/591867461 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c014040 con 0x56141533a280 2024-03-24T06:48:18.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.224+0000 7f3255600640 1 -- 172.21.15.22:0/591867461 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325c015040 con 0x56141533a280 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.224+0000 7f3255600640 1 -- 172.21.15.22:0/591867461 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c005020 con 0x56141533a280 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 -- 172.21.15.22:0/591867461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 msgr2=0x561415342f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f325c005fd0 tx=0x7f325c0095c0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 -- 172.21.15.22:0/591867461 shutdown_connections 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561415343780 0x56141535b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56141533aba0 0x56141534f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 --2- 172.21.15.22:0/591867461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533a280 0x561415342f20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.229 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 -- 172.21.15.22:0/591867461 >> 172.21.15.22:0/591867461 conn(0x5614153349b0 msgr2=0x56141533af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 -- 172.21.15.22:0/591867461 shutdown_connections 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.225+0000 7f326e77f740 1 -- 172.21.15.22:0/591867461 wait complete. 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.226+0000 7f326e77f740 1 -- start start 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.226+0000 7f326e77f740 1 Processor -- start 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415346dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415347510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x561415347c10 con 0x56141533aba0 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5614153b8f50 con 0x56141533a280 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5614153b9090 con 0x561415343780 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f326e77f740 10 client.0 _ll_register_callbacks cb 0x7f3250037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3263e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415346dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3269a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40824/0 (socket says 172.21.15.22:40824) 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3269a00640 1 -- 172.21.15.22:0/3501146572 learned_addr learned my addr 172.21.15.22:0/3501146572 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3269a00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415347510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3263e00640 1 -- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 msgr2=0x561415347510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3263e00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415347510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3263e00640 1 -- 172.21.15.22:0/3501146572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 msgr2=0x56141534ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.227+0000 7f3263e00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f3263e00640 1 -- 172.21.15.22:0/3501146572 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561415262e40 con 0x56141533aba0 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f3269a00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f3263e00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415346dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f325002cdc0 tx=0x7f3250004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2efa91a5cd33c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f324aa00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250004440 con 0x56141533aba0 2024-03-24T06:48:18.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f325002c5b0 con 0x56141533aba0 2024-03-24T06:48:18.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325c00cb20 con 0x56141533aba0 2024-03-24T06:48:18.234 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3250034a60 con 0x56141533aba0 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f326e77f740 10 client.16932 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.228+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5614152616a0 con 0x56141533aba0 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325c00cb20 con 0x56141533aba0 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f325005e070 con 0x56141533aba0 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 10 client.16932 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 10 client.16932 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 20 client.16932 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5614153ba610 0x5614153bca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.235 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 4 client.16932 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56141502c890 con 0x5614153ba610 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.229+0000 7f3269a00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5614153ba610 0x5614153bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f3269a00640 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5614153ba610 0x5614153bca30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b74e17bf72bca512 server_cookie=87084969c5f88b0c in_seq=0 out_seq=0 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f324aa00640 10 client.16932 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f324aa00640 1 -- 172.21.15.22:0/3501146572 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56141502c890 con 0x5614153ba610 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f324aa00640 10 client.16932 handle_command_reply: tid=0 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 client.16932 shutdown 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 msgr2=0x561415346dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415346dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f325002cdc0 tx=0x7f3250004990 comp rx=0 tx=0).stop 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 shutdown_connections 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x5614153ba610 0x5614153bca30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561415343780 0x561415347510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x56141533aba0 0x561415346dd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 --2- 172.21.15.22:0/3501146572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56141533a280 0x56141534ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f324aa00640 0 client.16932 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.230+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 >> 172.21.15.22:0/3501146572 conn(0x5614153349b0 msgr2=0x561415340790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 shutdown_connections 2024-03-24T06:48:18.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326e77f740 1 -- 172.21.15.22:0/3501146572 wait complete. 2024-03-24T06:48:18.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326e77f740 20 client.16932 trim_cache size 0 max 16384 2024-03-24T06:48:18.238 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 msgr2=0x7f324c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 0x7f324c0a46f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32580cbd90 tx=0x7f3258004b30 comp rx=0 tx=0).stop 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 msgr2=0x7f3264199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.231+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f325800e9f0 tx=0x7f325800edf0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 shutdown_connections 2024-03-24T06:48:18.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f324c0a2230 0x7f324c0a46f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f326410bd20 0x7f326419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3264103270 0x7f326419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 --2- 172.21.15.22:0/2720893703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3264102950 0x7f3264199fc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 >> 172.21.15.22:0/2720893703 conn(0x7f3264076990 msgr2=0x7f3264100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 shutdown_connections 2024-03-24T06:48:18.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.232+0000 7f326ba00640 1 -- 172.21.15.22:0/2720893703 wait complete. 2024-03-24T06:48:18.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:18 smithi022 ceph-mon[32003]: pgmap v256: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:18.247 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:18 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/958241379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:18.251 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:48:18.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 -- 172.21.15.22:0/248845089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73941038b0 msgr2=0x7f7394103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 --2- 172.21.15.22:0/248845089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73941038b0 0x7f7394103d30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7380009cd0 tx=0x7f738002c0e0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 -- 172.21.15.22:0/248845089 shutdown_connections 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 --2- 172.21.15.22:0/248845089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7394104270 0x7f7394110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 --2- 172.21.15.22:0/248845089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73941038b0 0x7f7394103d30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 --2- 172.21.15.22:0/248845089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 0x7f7394109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 -- 172.21.15.22:0/248845089 >> 172.21.15.22:0/248845089 conn(0x7f73940ff380 msgr2=0x7f73941017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 -- 172.21.15.22:0/248845089 shutdown_connections 2024-03-24T06:48:18.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.451+0000 7f739b200640 1 -- 172.21.15.22:0/248845089 wait complete. 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.452+0000 7f739b200640 1 Processor -- start 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.452+0000 7f739b200640 1 -- start start 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 0x7f7394199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 0x7f739419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f739419ee50 con 0x7f73941038b0 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f739419efc0 con 0x7f73941098c0 2024-03-24T06:48:18.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f739b200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f739419f180 con 0x7f7394104270 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f7393400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f7393400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59752/0 (socket says 172.21.15.22:59752) 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f7393400640 1 -- 172.21.15.22:0/1825465948 learned_addr learned my addr 172.21.15.22:0/1825465948 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f7399200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 0x7f739419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.453+0000 7f7393e00640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 0x7f7394199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 -- 172.21.15.22:0/1825465948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 msgr2=0x7f739419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 0x7f739419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 -- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 msgr2=0x7f7394199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 0x7f7394199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 -- 172.21.15.22:0/1825465948 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7380009900 con 0x7f7394104270 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393e00640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 0x7f7394199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7393400640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7380004af0 tx=0x7f73800325e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7390c00640 1 -- 172.21.15.22:0/1825465948 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380034e20 con 0x7f7394104270 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7390c00640 1 -- 172.21.15.22:0/1825465948 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7380003e50 con 0x7f7394104270 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f7390c00640 1 -- 172.21.15.22:0/1825465948 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380004760 con 0x7f7394104270 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.454+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739419f400 con 0x7f7394104270 2024-03-24T06:48:18.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.455+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73941a6b00 con 0x7f7394104270 2024-03-24T06:48:18.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.459+0000 7f739dfd2740 1 Processor -- start 2024-03-24T06:48:18.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.459+0000 7f739dfd2740 1 -- start start 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7390c00640 1 -- 172.21.15.22:0/1825465948 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f738000a040 con 0x7f7394104270 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7beb5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7beb7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be7be1a6e0 con 0x55be7beb5780 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be7be17c10 con 0x55be7beac280 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f739dfd2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7be133f0 con 0x55be7beacba0 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40842/0 (socket says 172.21.15.22:40842) 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 -- 172.21.15.22:0/937863622 learned_addr learned my addr 172.21.15.22:0/937863622 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7393e00640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7beb5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7393400640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7beb7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 -- 172.21.15.22:0/937863622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 msgr2=0x55be7beb5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7beb5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.460+0000 7f7399200640 1 -- 172.21.15.22:0/937863622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 msgr2=0x55be7beb7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7399200640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7beb7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7399200640 1 -- 172.21.15.22:0/937863622 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7bdd4e40 con 0x55be7beac280 2024-03-24T06:48:18.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7393400640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7beb7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7399200640 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f738c009ca0 tx=0x7f738c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c9252c522b3abb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7389600640 1 -- 172.21.15.22:0/937863622 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c019630 con 0x55be7beac280 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7389600640 1 -- 172.21.15.22:0/937863622 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f738c009e50 con 0x55be7beac280 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.461+0000 7f7389600640 1 -- 172.21.15.22:0/937863622 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c00eb80 con 0x55be7beac280 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 -- 172.21.15.22:0/937863622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 msgr2=0x55be7beac660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f738c009ca0 tx=0x7f738c011810 comp rx=0 tx=0).stop 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 -- 172.21.15.22:0/937863622 shutdown_connections 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7beb7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7beb5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f7390c00640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 0x7f73780a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 --2- 172.21.15.22:0/937863622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7beac660 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 -- 172.21.15.22:0/937863622 >> 172.21.15.22:0/937863622 conn(0x55be7bea69b0 msgr2=0x55be7bea6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f7390c00640 1 -- 172.21.15.22:0/1825465948 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f73800f0c10 con 0x7f7394104270 2024-03-24T06:48:18.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f7399200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 0x7f73780a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.462+0000 7f739dfd2740 1 -- 172.21.15.22:0/937863622 shutdown_connections 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.463+0000 7f739dfd2740 1 -- 172.21.15.22:0/937863622 wait complete. 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.463+0000 7f739dfd2740 1 -- start start 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.463+0000 7f7399200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 0x7f73780a4770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e6c0 tx=0x7f738c000bc0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.463+0000 7f739dfd2740 1 Processor -- start 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beacba0 0x55be7beb2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beb5780 0x55be7beb31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be7bdb8060 con 0x55be7beacba0 2024-03-24T06:48:18.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be7bdb7d00 con 0x55be7beb5780 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7bb811c0 con 0x55be7beac280 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f739dfd2740 10 client.0 _ll_register_callbacks cb 0x55be7bb23be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f7393e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f7393400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beacba0 0x55be7beb2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f7393e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59776/0 (socket says 172.21.15.22:59776) 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f7393e00640 1 -- 172.21.15.22:0/4267208060 learned_addr learned my addr 172.21.15.22:0/4267208060 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.464+0000 7f7399200640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beb5780 0x55be7beb31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 -- 172.21.15.22:0/4267208060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beb5780 msgr2=0x55be7beb31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beb5780 0x55be7beb31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 -- 172.21.15.22:0/4267208060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beacba0 msgr2=0x55be7beb2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beacba0 0x55be7beb2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 -- 172.21.15.22:0/4267208060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c009950 con 0x55be7beac280 2024-03-24T06:48:18.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f7393e00640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7384009cf0 tx=0x7f73840039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c52e78a3c72de482 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f737e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384012860 con 0x55be7beac280 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be7bb127e0 con 0x55be7beac280 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.465+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f73840129c0 con 0x55be7beac280 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7384012b80 con 0x55be7beac280 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 10 client.27331 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55be7be11a70 con 0x55be7beac280 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55be7bb127e0 con 0x55be7beac280 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7384011450 con 0x55be7beac280 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 10 client.27331 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 10 client.27331 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 20 client.27331 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bec5c30 0x55be7bec8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 4 client.27331 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.466+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be7bb4f720 con 0x55be7bec5c30 2024-03-24T06:48:18.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.467+0000 7f7393400640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bec5c30 0x55be7bec8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.467+0000 7f7393400640 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bec5c30 0x55be7bec8090 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7bd3da853448a67 server_cookie=8cbbc769c554fce5 in_seq=0 out_seq=0 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.467+0000 7f737e000640 10 client.27331 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f737e000640 1 -- 172.21.15.22:0/4267208060 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be7bb4f720 con 0x55be7bec5c30 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f737e000640 10 client.27331 handle_command_reply: tid=0 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 client.27331 shutdown 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 msgr2=0x55be7beb2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7384009cf0 tx=0x7f73840039c0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 shutdown_connections 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bec5c30 0x55be7bec8090 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beb5780 0x55be7beb31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beacba0 0x55be7beb2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 --2- 172.21.15.22:0/4267208060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beac280 0x55be7beb2760 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f737e000640 0 client.27331 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.468+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 >> 172.21.15.22:0/4267208060 conn(0x55be7bea69b0 msgr2=0x55be7bea8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.469+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 shutdown_connections 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.469+0000 7f739dfd2740 1 -- 172.21.15.22:0/4267208060 wait complete. 2024-03-24T06:48:18.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.469+0000 7f739dfd2740 20 client.27331 trim_cache size 0 max 16384 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 Processor -- start 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 -- start start 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7bea58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be7bea8cc0 con 0x55be7beb5780 2024-03-24T06:48:18.485 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be7beabc50 con 0x55be7beac280 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f739dfd2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7beb0cb0 con 0x55be7beacba0 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7393400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7393e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7393400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59778/0 (socket says 172.21.15.22:59778) 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7393e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40866/0 (socket says 172.21.15.22:40866) 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7393e00640 1 -- 172.21.15.22:0/58242743 learned_addr learned my addr 172.21.15.22:0/58242743 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.484+0000 7f7399200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7bea58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 -- 172.21.15.22:0/58242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 msgr2=0x55be7becd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 -- 172.21.15.22:0/58242743 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 msgr2=0x55be7bea58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7bea58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 -- 172.21.15.22:0/58242743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7bdadb00 con 0x55be7beac280 2024-03-24T06:48:18.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393400640 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7393e00640 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7384000c00 tx=0x7f73840097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c2d7a39cab31c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7388c00640 1 -- 172.21.15.22:0/58242743 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840098b0 con 0x55be7beac280 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7388c00640 1 -- 172.21.15.22:0/58242743 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7384011650 con 0x55be7beac280 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.485+0000 7f7388c00640 1 -- 172.21.15.22:0/58242743 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840048e0 con 0x55be7beac280 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- 172.21.15.22:0/58242743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 msgr2=0x55be7bec1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7384000c00 tx=0x7f73840097d0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- 172.21.15.22:0/58242743 shutdown_connections 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beb5780 0x55be7bea58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beacba0 0x55be7becd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 --2- 172.21.15.22:0/58242743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beac280 0x55be7bec1010 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- 172.21.15.22:0/58242743 >> 172.21.15.22:0/58242743 conn(0x55be7bea69b0 msgr2=0x55be7bea51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- 172.21.15.22:0/58242743 shutdown_connections 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- 172.21.15.22:0/58242743 wait complete. 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.486+0000 7f739dfd2740 1 -- start start 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 Processor -- start 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beac280 0x55be7beaf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beb5780 0x55be7beaff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55be7bf2b0f0 con 0x55be7beac280 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be7bf2b260 con 0x55be7beacba0 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7bf2b420 con 0x55be7beb5780 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f739dfd2740 10 client.0 _ll_register_callbacks cb 0x7f7384004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f7393400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f7393e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beac280 0x55be7beaf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f7393400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40876/0 (socket says 172.21.15.22:40876) 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f7393400640 1 -- 172.21.15.22:0/3939413154 learned_addr learned my addr 172.21.15.22:0/3939413154 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.487+0000 7f7399200640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beb5780 0x55be7beaff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 -- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beb5780 msgr2=0x55be7beaff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beb5780 0x55be7beaff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 -- 172.21.15.22:0/3939413154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beac280 msgr2=0x55be7beaf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beac280 0x55be7beaf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 -- 172.21.15.22:0/3939413154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73800b7e70 con 0x55be7beacba0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f7393400640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f73800b9cd0 tx=0x7f7380009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f5eb277a6e9ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f737e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73800bda00 con 0x55be7beacba0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f73800bd300 con 0x55be7beacba0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.488+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f738c009e50 con 0x55be7beacba0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f73800c1450 con 0x55be7beacba0 2024-03-24T06:48:18.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 10 client.27308 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55be7be11a70 con 0x55be7beacba0 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f738c009e50 con 0x55be7beacba0 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f73800ee1d0 con 0x55be7beacba0 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 10 client.27308 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 10 client.27308 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 20 client.27308 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bf2ca10 0x55be7bf2ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 4 client.27308 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:18.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be7bb9e890 con 0x55be7bf2ca10 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.489+0000 7f7393e00640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bf2ca10 0x55be7bf2ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f7393e00640 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bf2ca10 0x55be7bf2ee00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65bee61c98f0ca59 server_cookie=8e6b0f74f3ddb473 in_seq=0 out_seq=0 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f737e000640 10 client.27308 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f737e000640 1 -- 172.21.15.22:0/3939413154 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be7bb9e890 con 0x55be7bf2ca10 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f737e000640 10 client.27308 handle_command_reply: tid=0 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 client.27308 shutdown 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 msgr2=0x55be7beafa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f73800b9cd0 tx=0x7f7380009800 comp rx=0 tx=0).stop 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 shutdown_connections 2024-03-24T06:48:18.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55be7bf2ca10 0x55be7bf2ee00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7beb5780 0x55be7beaff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be7beacba0 0x55be7beafa50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 --2- 172.21.15.22:0/3939413154 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55be7beac280 0x55be7beaf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 >> 172.21.15.22:0/3939413154 conn(0x55be7bea69b0 msgr2=0x55be7bea8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.490+0000 7f737e000640 0 client.27308 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 shutdown_connections 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739dfd2740 1 -- 172.21.15.22:0/3939413154 wait complete. 2024-03-24T06:48:18.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739dfd2740 20 client.27308 trim_cache size 0 max 16384 2024-03-24T06:48:18.498 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 msgr2=0x7f73780a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 0x7f73780a4770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e6c0 tx=0x7f738c000bc0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 msgr2=0x7f739419a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7380004af0 tx=0x7f73800325e0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 shutdown_connections 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f73780a22b0 0x7f73780a4770 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73941098c0 0x7f739419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394104270 0x7f739419a4d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 --2- 172.21.15.22:0/1825465948 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f73941038b0 0x7f7394199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.491+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 >> 172.21.15.22:0/1825465948 conn(0x7f73940ff380 msgr2=0x7f73940ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.492+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 shutdown_connections 2024-03-24T06:48:18.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.492+0000 7f739b200640 1 -- 172.21.15.22:0/1825465948 wait complete. 2024-03-24T06:48:18.508 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:48:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:18 smithi155 ceph-mon[36262]: pgmap v256: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:18 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/958241379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:18 smithi143 ceph-mon[36520]: pgmap v256: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:18.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:18 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/958241379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.712+0000 7fa56a400640 1 -- 172.21.15.22:0/879300745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410aac0 msgr2=0x7fa56410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.712+0000 7fa56a400640 1 --2- 172.21.15.22:0/879300745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410aac0 0x7fa56410eb00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009930 tx=0x7fa54c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 -- 172.21.15.22:0/879300745 shutdown_connections 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 --2- 172.21.15.22:0/879300745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410f1e0 0x7fa564116db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 --2- 172.21.15.22:0/879300745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410aac0 0x7fa56410eb00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 --2- 172.21.15.22:0/879300745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56410a110 0x7fa56410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 -- 172.21.15.22:0/879300745 >> 172.21.15.22:0/879300745 conn(0x7fa564072cc0 msgr2=0x7fa5640730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 -- 172.21.15.22:0/879300745 shutdown_connections 2024-03-24T06:48:18.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.713+0000 7fa56a400640 1 -- 172.21.15.22:0/879300745 wait complete. 2024-03-24T06:48:18.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.714+0000 7fa56a400640 1 Processor -- start 2024-03-24T06:48:18.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.714+0000 7fa56a400640 1 -- start start 2024-03-24T06:48:18.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410a110 0x7fa56406af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56406b980 0x7fa5641ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa5641aed90 con 0x7fa56410a110 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa5641aef00 con 0x7fa56410f1e0 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa56a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa5641af0c0 con 0x7fa56406b980 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa563400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410a110 0x7fa56406af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa563e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56406b980 0x7fa5641ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40898/0 (socket says 172.21.15.22:40898) 2024-03-24T06:48:18.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.715+0000 7fa55aa00640 1 -- 172.21.15.22:0/902800427 learned_addr learned my addr 172.21.15.22:0/902800427 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 -- 172.21.15.22:0/902800427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56406b980 msgr2=0x7fa5641ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56406b980 0x7fa5641ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 -- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410a110 msgr2=0x7fa56406af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410a110 0x7fa56406af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 -- 172.21.15.22:0/902800427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa554009840 con 0x7fa56410f1e0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa55aa00640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa54c005ab0 tx=0x7fa54c02c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa560c00640 1 -- 172.21.15.22:0/902800427 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0042d0 con 0x7fa56410f1e0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa560c00640 1 -- 172.21.15.22:0/902800427 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c004430 con 0x7fa56410f1e0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa54c009560 con 0x7fa56410f1e0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.716+0000 7fa560c00640 1 -- 172.21.15.22:0/902800427 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c003be0 con 0x7fa56410f1e0 2024-03-24T06:48:18.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.717+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5641af6e0 con 0x7fa56410f1e0 2024-03-24T06:48:18.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 Processor -- start 2024-03-24T06:48:18.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa560c00640 1 -- 172.21.15.22:0/902800427 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa54c003d40 con 0x7fa56410f1e0 2024-03-24T06:48:18.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 -- start start 2024-03-24T06:48:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecca0780 0x5568ecca2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5568ecc056e0 con 0x5568ecca0780 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568ecc02c10 con 0x5568ecc97ba0 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.720+0000 7fa56d201740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5568ecbfe3f0 con 0x5568ecc97280 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59810/0 (socket says 172.21.15.22:59810) 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 -- 172.21.15.22:0/2928985637 learned_addr learned my addr 172.21.15.22:0/2928985637 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa55aa00640 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecca0780 0x5568ecca2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 -- 172.21.15.22:0/2928985637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 msgr2=0x5568ecca0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 -- 172.21.15.22:0/2928985637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecca0780 msgr2=0x5568ecca2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecca0780 0x5568ecca2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 -- 172.21.15.22:0/2928985637 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ecbbfe40 con 0x5568ecc97280 2024-03-24T06:48:18.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.721+0000 7fa563e00640 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009c20 tx=0x7fa55c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d85d20e63536dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa560c00640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 0x7fa5480a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa560c00640 1 -- 172.21.15.22:0/902800427 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fa54c0f0a10 con 0x7fa56410f1e0 2024-03-24T06:48:18.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa558c00640 1 -- 172.21.15.22:0/2928985637 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c019630 con 0x5568ecc97280 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa558c00640 1 -- 172.21.15.22:0/2928985637 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa55c009e90 con 0x5568ecc97280 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa558c00640 1 -- 172.21.15.22:0/2928985637 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00eaf0 con 0x5568ecc97280 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.722+0000 7fa563400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 0x7fa5480a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa563400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 0x7fa5480a47a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa554009fd0 tx=0x7fa554009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 -- 172.21.15.22:0/2928985637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 msgr2=0x5568ecc97660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009c20 tx=0x7fa55c011810 comp rx=0 tx=0).stop 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 -- 172.21.15.22:0/2928985637 shutdown_connections 2024-03-24T06:48:18.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecca0780 0x5568ecca2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 --2- 172.21.15.22:0/2928985637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97280 0x5568ecc97660 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 -- 172.21.15.22:0/2928985637 >> 172.21.15.22:0/2928985637 conn(0x5568ecc919b0 msgr2=0x5568ecc91d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 -- 172.21.15.22:0/2928985637 shutdown_connections 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.723+0000 7fa56d201740 1 -- 172.21.15.22:0/2928985637 wait complete. 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.724+0000 7fa56d201740 1 -- start start 2024-03-24T06:48:18.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.724+0000 7fa56d201740 1 Processor -- start 2024-03-24T06:48:18.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecc9e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5568ecba3060 con 0x5568ecc97280 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568ecba2d00 con 0x5568ecc97ba0 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5568ec96c1c0 con 0x5568ecca0780 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa56d201740 10 client.0 _ll_register_callbacks cb 0x5568ec90ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa563400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa563e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecc9e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40914/0 (socket says 172.21.15.22:40914) 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 -- 172.21.15.22:0/894164308 learned_addr learned my addr 172.21.15.22:0/894164308 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 -- 172.21.15.22:0/894164308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 msgr2=0x5568ecc9e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecc9e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 -- 172.21.15.22:0/894164308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 msgr2=0x5568ecc9d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.725+0000 7fa55aa00640 1 -- 172.21.15.22:0/894164308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55c0098d0 con 0x5568ecc97ba0 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa55aa00640 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0bd030 tx=0x7fa54c003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2c34a087b597ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa551600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:18.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0c55b0 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fa54c0bd0a0 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568ec8fd7e0 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa54c0ee180 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa56d201740 10 client.27320 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.726+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5568ecbfca70 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5568ec8fd7e0 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa54c03c070 con 0x5568ecc97ba0 2024-03-24T06:48:18.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 10 client.27320 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 10 client.27320 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 20 client.27320 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568eccb0900 0x5568eccb2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 4 client.27320 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568ec93a720 con 0x5568eccb0900 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.727+0000 7fa563400640 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568eccb0900 0x5568eccb2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.728+0000 7fa551600640 10 client.27320 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.728+0000 7fa563400640 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568eccb0900 0x5568eccb2cf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cbc920fdd408e4f2 server_cookie=36683cca113dab7f in_seq=0 out_seq=0 2024-03-24T06:48:18.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa551600640 1 -- 172.21.15.22:0/894164308 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568ec93a720 con 0x5568eccb0900 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa551600640 10 client.27320 handle_command_reply: tid=0 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 client.27320 shutdown 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 msgr2=0x5568ecc9dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0bd030 tx=0x7fa54c003680 comp rx=0 tx=0).stop 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 shutdown_connections 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecc9e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568eccb0900 0x5568eccb2cf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecc9dd20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa551600640 0 client.27320 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:18.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 --2- 172.21.15.22:0/894164308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 >> 172.21.15.22:0/894164308 conn(0x5568ecc919b0 msgr2=0x5568ecc93700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 shutdown_connections 2024-03-24T06:48:18.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 1 -- 172.21.15.22:0/894164308 wait complete. 2024-03-24T06:48:18.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.729+0000 7fa56d201740 20 client.27320 trim_cache size 0 max 16384 2024-03-24T06:48:18.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 Processor -- start 2024-03-24T06:48:18.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 -- start start 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecca0780 0x5568eccb85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5568ec93a720 con 0x5568ecc97280 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568ec8fd7e0 con 0x5568ecca0780 2024-03-24T06:48:18.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa56d201740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5568eccb0710 con 0x5568ecc97ba0 2024-03-24T06:48:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.748+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59830/0 (socket says 172.21.15.22:59830) 2024-03-24T06:48:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 -- 172.21.15.22:0/1139086076 learned_addr learned my addr 172.21.15.22:0/1139086076 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa563e00640 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecca0780 0x5568eccb85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa563400640 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 -- 172.21.15.22:0/1139086076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecca0780 msgr2=0x5568eccb85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecca0780 0x5568eccb85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 -- 172.21.15.22:0/1139086076 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 msgr2=0x5568ecc9ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 -- 172.21.15.22:0/1139086076 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ecb98b00 con 0x5568ecc97ba0 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa54c02c780 tx=0x7fa54c0c10c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab9ca3dcc323f075 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa553e00640 1 -- 172.21.15.22:0/1139086076 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0ee180 con 0x5568ecc97ba0 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.749+0000 7fa553e00640 1 -- 172.21.15.22:0/1139086076 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c0bcbe0 con 0x5568ecc97ba0 2024-03-24T06:48:18.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa553e00640 1 -- 172.21.15.22:0/1139086076 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0c5840 con 0x5568ecc97ba0 2024-03-24T06:48:18.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 -- 172.21.15.22:0/1139086076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 msgr2=0x5568eccac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa54c02c780 tx=0x7fa54c0c10c0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 -- 172.21.15.22:0/1139086076 shutdown_connections 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecca0780 0x5568eccb85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecc97ba0 0x5568eccac010 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 --2- 172.21.15.22:0/1139086076 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecc9ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.750+0000 7fa56d201740 1 -- 172.21.15.22:0/1139086076 >> 172.21.15.22:0/1139086076 conn(0x5568ecc919b0 msgr2=0x5568ecc97f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.751+0000 7fa56d201740 1 -- 172.21.15.22:0/1139086076 shutdown_connections 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.751+0000 7fa56d201740 1 -- 172.21.15.22:0/1139086076 wait complete. 2024-03-24T06:48:18.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.751+0000 7fa56d201740 1 -- start start 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.751+0000 7fa56d201740 1 Processor -- start 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecca4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5568ecca4990 con 0x5568ecc97280 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568ecca4b00 con 0x5568ecc97ba0 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5568ecd15c00 con 0x5568ecca0780 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa56d201740 10 client.0 _ll_register_callbacks cb 0x7fa54c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60332/0 (socket says 172.21.15.22:60332) 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecca4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 -- 172.21.15.22:0/1901342503 learned_addr learned my addr 172.21.15.22:0/1901342503 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 -- 172.21.15.22:0/1901342503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 msgr2=0x5568ecca4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecca4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 -- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 msgr2=0x5568ecca3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.752+0000 7fa563400640 1 -- 172.21.15.22:0/1901342503 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ecbfca70 con 0x5568ecc97280 2024-03-24T06:48:18.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa563400640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5568eccb3570 tx=0x7fa554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fa7e085ee83622 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa551600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554024040 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7fa55402ad10 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa55c009e90 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa554016070 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa56d201740 10 client.16968 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.753+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5568ecbbe6a0 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa55c009e90 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa554004040 con 0x5568ecc97280 2024-03-24T06:48:18.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 10 client.16968 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 10 client.16968 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 20 client.16968 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568ecd173f0 0x5568ecd19810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 4 client.16968 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5568ec989890 con 0x5568ecd173f0 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.754+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568ecd173f0 0x5568ecd19810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa55aa00640 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568ecd173f0 0x5568ecd19810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe804ef5b9828c4d server_cookie=a56d2c25af28aa37 in_seq=0 out_seq=0 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa551600640 10 client.16968 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa551600640 1 -- 172.21.15.22:0/1901342503 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5568ec989890 con 0x5568ecd173f0 2024-03-24T06:48:18.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa551600640 10 client.16968 handle_command_reply: tid=0 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 client.16968 shutdown 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 msgr2=0x5568ecca8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5568eccb3570 tx=0x7fa554004a40 comp rx=0 tx=0).stop 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 shutdown_connections 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568ecca0780 0x5568ecca4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5568ecd173f0 0x5568ecd19810 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568ecc97ba0 0x5568ecca3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 --2- 172.21.15.22:0/1901342503 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5568ecc97280 0x5568ecca8940 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 >> 172.21.15.22:0/1901342503 conn(0x5568ecc919b0 msgr2=0x5568ecc901e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa551600640 0 client.16968 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:18.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 shutdown_connections 2024-03-24T06:48:18.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.755+0000 7fa56d201740 1 -- 172.21.15.22:0/1901342503 wait complete. 2024-03-24T06:48:18.762 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56d201740 20 client.16968 trim_cache size 0 max 16384 2024-03-24T06:48:18.762 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 msgr2=0x7fa5480a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 0x7fa5480a47a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa554009fd0 tx=0x7fa554009500 comp rx=0 tx=0).stop 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 msgr2=0x7fa56406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa54c005ab0 tx=0x7fa54c02c5f0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 shutdown_connections 2024-03-24T06:48:18.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa5480a22e0 0x7fa5480a47a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa56406b980 0x7fa5641ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa56410f1e0 0x7fa56406b440 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 --2- 172.21.15.22:0/902800427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa56410a110 0x7fa56406af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.756+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 >> 172.21.15.22:0/902800427 conn(0x7fa564072cc0 msgr2=0x7fa56406e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.757+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 shutdown_connections 2024-03-24T06:48:18.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.757+0000 7fa56a400640 1 -- 172.21.15.22:0/902800427 wait complete. 2024-03-24T06:48:18.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:48:18.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.977+0000 7f35f8200640 1 -- 172.21.15.22:0/3371139455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f01088d0 msgr2=0x7f35f010c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.977+0000 7f35f8200640 1 --2- 172.21.15.22:0/3371139455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f01088d0 0x7f35f010c910 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009890 tx=0x7f35dc02bb70 comp rx=0 tx=0).stop 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 -- 172.21.15.22:0/3371139455 shutdown_connections 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 --2- 172.21.15.22:0/3371139455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 0x7f35f0114bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 --2- 172.21.15.22:0/3371139455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f01088d0 0x7f35f010c910 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 --2- 172.21.15.22:0/3371139455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f0107f20 0x7f35f0108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 -- 172.21.15.22:0/3371139455 >> 172.21.15.22:0/3371139455 conn(0x7f35f0075c50 msgr2=0x7f35f0078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 -- 172.21.15.22:0/3371139455 shutdown_connections 2024-03-24T06:48:18.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.978+0000 7f35f8200640 1 -- 172.21.15.22:0/3371139455 wait complete. 2024-03-24T06:48:18.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 Processor -- start 2024-03-24T06:48:18.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 -- start start 2024-03-24T06:48:18.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 0x7f35f019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 0x7f35f01a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a3630 con 0x7f35f010cff0 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a37a0 con 0x7f35f0107f20 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f8200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f35f01a3910 con 0x7f35f01a2c40 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.979+0000 7f35f5800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40940/0 (socket says 172.21.15.22:40940) 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 -- 172.21.15.22:0/3801961879 learned_addr learned my addr 172.21.15.22:0/3801961879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f4e00640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 0x7f35f019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f6200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 0x7f35f01a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 -- 172.21.15.22:0/3801961879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 msgr2=0x7f35f01a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 0x7f35f01a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 -- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 msgr2=0x7f35f019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 0x7f35f019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 -- 172.21.15.22:0/3801961879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35dc009490 con 0x7f35f0107f20 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f6200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 0x7f35f01a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.980+0000 7f35f5800640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e9c0 tx=0x7f35e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.981+0000 7f35ee000640 1 -- 172.21.15.22:0/3801961879 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e400ccc0 con 0x7f35f0107f20 2024-03-24T06:48:18.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.981+0000 7f35ee000640 1 -- 172.21.15.22:0/3801961879 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e400ce20 con 0x7f35f0107f20 2024-03-24T06:48:18.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.981+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f01aaef0 con 0x7f35f0107f20 2024-03-24T06:48:18.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.981+0000 7f35ee000640 1 -- 172.21.15.22:0/3801961879 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e4010430 con 0x7f35f0107f20 2024-03-24T06:48:18.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.981+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f01ab310 con 0x7f35f0107f20 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.984+0000 7f35ee000640 1 -- 172.21.15.22:0/3801961879 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f35e4004f80 con 0x7f35f0107f20 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.984+0000 7f35fae19740 1 Processor -- start 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.984+0000 7f35fae19740 1 -- start start 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566e2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea566476e0 con 0x55ea566e2780 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea56644c10 con 0x55ea566d9ba0 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35fae19740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea566403f0 con 0x55ea566d9280 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59868/0 (socket says 172.21.15.22:59868) 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 -- 172.21.15.22:0/3452796307 learned_addr learned my addr 172.21.15.22:0/3452796307 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f5800640 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566e2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f4e00640 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 -- 172.21.15.22:0/3452796307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 msgr2=0x55ea566e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566e2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 -- 172.21.15.22:0/3452796307 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 msgr2=0x55ea566e4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566e4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.985+0000 7f35f6200640 1 -- 172.21.15.22:0/3452796307 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea56601e40 con 0x55ea566d9280 2024-03-24T06:48:18.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35f6200640 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f35e8011790 tx=0x7f35e8011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101aeb73ee44bdb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35ee000640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 0x7f35d80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35e2a00640 1 -- 172.21.15.22:0/3452796307 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8011ea0 con 0x55ea566d9280 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35e2a00640 1 -- 172.21.15.22:0/3452796307 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e800fe80 con 0x55ea566d9280 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35e2a00640 1 -- 172.21.15.22:0/3452796307 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8004840 con 0x55ea566d9280 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35ee000640 1 -- 172.21.15.22:0/3801961879 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f35e4014070 con 0x7f35f0107f20 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.986+0000 7f35f5800640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 0x7f35d80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 -- 172.21.15.22:0/3452796307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 msgr2=0x55ea566d9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f35e8011790 tx=0x7f35e8011c50 comp rx=0 tx=0).stop 2024-03-24T06:48:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 -- 172.21.15.22:0/3452796307 shutdown_connections 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566e4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566e2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35f5800640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 0x7f35d80a4770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55ea566d8d50 tx=0x7f35e40d40e0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 --2- 172.21.15.22:0/3452796307 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566d9660 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 -- 172.21.15.22:0/3452796307 >> 172.21.15.22:0/3452796307 conn(0x55ea566d39b0 msgr2=0x55ea566d3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 -- 172.21.15.22:0/3452796307 shutdown_connections 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.987+0000 7f35fae19740 1 -- 172.21.15.22:0/3452796307 wait complete. 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 -- start start 2024-03-24T06:48:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 Processor -- start 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566d9280 0x55ea566df7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566e2780 0x55ea566e0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea565e5060 con 0x55ea566d9280 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea565e4d00 con 0x55ea566d9ba0 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea563ae1c0 con 0x55ea566e2780 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.988+0000 7f35fae19740 10 client.0 _ll_register_callbacks cb 0x55ea56350be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40954/0 (socket says 172.21.15.22:40954) 2024-03-24T06:48:18.992 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 -- 172.21.15.22:0/1617598686 learned_addr learned my addr 172.21.15.22:0/1617598686 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f6200640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566d9280 0x55ea566df7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f6200640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566e2780 0x55ea566e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 -- 172.21.15.22:0/1617598686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566e2780 msgr2=0x55ea566e0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566e2780 0x55ea566e0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 -- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566d9280 msgr2=0x55ea566df7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566d9280 0x55ea566df7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 -- 172.21.15.22:0/1617598686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e8011440 con 0x55ea566d9ba0 2024-03-24T06:48:18.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f35dc02c180 tx=0x7f35dc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc5390b1b36d5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:18.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.989+0000 7f35d7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:18.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc034640 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea5633f7e0 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f35dc0347a0 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35dc034960 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 10 client.27338 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ea5663ea70 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ea5633f7e0 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35dc032600 con 0x55ea566d9ba0 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 10 client.27338 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:18.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 10 client.27338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 20 client.27338 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.990+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea566f2ae0 0x55ea566f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.991+0000 7f35fae19740 4 client.27338 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.991+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea5637c720 con 0x55ea566f2ae0 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.991+0000 7f35f6200640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea566f2ae0 0x55ea566f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.991+0000 7f35f6200640 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea566f2ae0 0x55ea566f4f00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba221daec30fedb4 server_cookie=ba7d30f3f83fcf1 in_seq=0 out_seq=0 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.991+0000 7f35d7400640 10 client.27338 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35d7400640 1 -- 172.21.15.22:0/1617598686 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea5637c720 con 0x55ea566f2ae0 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35d7400640 10 client.27338 handle_command_reply: tid=0 2024-03-24T06:48:18.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 client.27338 shutdown 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 msgr2=0x55ea566dfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f35dc02c180 tx=0x7f35dc004270 comp rx=0 tx=0).stop 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 shutdown_connections 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566e2780 0x55ea566e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea566f2ae0 0x55ea566f4f00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9ba0 0x55ea566dfce0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 --2- 172.21.15.22:0/1617598686 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566d9280 0x55ea566df7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 >> 172.21.15.22:0/1617598686 conn(0x55ea566d39b0 msgr2=0x55ea566d5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:18.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.992+0000 7f35d7400640 0 client.27338 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:18.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.993+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 shutdown_connections 2024-03-24T06:48:18.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.993+0000 7f35fae19740 1 -- 172.21.15.22:0/1617598686 wait complete. 2024-03-24T06:48:18.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:18.993+0000 7f35fae19740 20 client.27338 trim_cache size 0 max 16384 2024-03-24T06:48:19.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.007+0000 7f35fae19740 1 Processor -- start 2024-03-24T06:48:19.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 -- start start 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566dfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea566d8ee0 con 0x55ea566e2780 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea566d49b0 con 0x55ea566d9280 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35fae19740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea566db3b0 con 0x55ea566d9ba0 2024-03-24T06:48:19.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f6200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f6200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40964/0 (socket says 172.21.15.22:40964) 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f6200640 1 -- 172.21.15.22:0/1731047365 learned_addr learned my addr 172.21.15.22:0/1731047365 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59892/0 (socket says 172.21.15.22:59892) 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.008+0000 7f35f6200640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566dfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 -- 172.21.15.22:0/1731047365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 msgr2=0x55ea566e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 -- 172.21.15.22:0/1731047365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 msgr2=0x55ea566dfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566dfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 -- 172.21.15.22:0/1731047365 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea565dab00 con 0x55ea566d9ba0 2024-03-24T06:48:19.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f6200640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35f4e00640 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0329b0 tx=0x7f35dc02bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a0ec21395967db4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35e2000640 1 -- 172.21.15.22:0/1731047365 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc032e20 con 0x55ea566d9ba0 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35e2000640 1 -- 172.21.15.22:0/1731047365 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35dc032600 con 0x55ea566d9ba0 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.009+0000 7f35e2000640 1 -- 172.21.15.22:0/1731047365 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0347b0 con 0x55ea566d9ba0 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 -- 172.21.15.22:0/1731047365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 msgr2=0x55ea566ee010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0329b0 tx=0x7f35dc02bb70 comp rx=0 tx=0).stop 2024-03-24T06:48:19.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 -- 172.21.15.22:0/1731047365 shutdown_connections 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea566e2780 0x55ea566dfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9ba0 0x55ea566ee010 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 --2- 172.21.15.22:0/1731047365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566d9280 0x55ea566e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 -- 172.21.15.22:0/1731047365 >> 172.21.15.22:0/1731047365 conn(0x55ea566d39b0 msgr2=0x55ea566d3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 -- 172.21.15.22:0/1731047365 shutdown_connections 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.010+0000 7f35fae19740 1 -- 172.21.15.22:0/1731047365 wait complete. 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 -- start start 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 Processor -- start 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566dcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea56751df0 0x55ea56752270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ea567527b0 con 0x55ea56751df0 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea56752920 con 0x55ea566e2780 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ea56752ae0 con 0x55ea566d9280 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.011+0000 7f35fae19740 10 client.0 _ll_register_callbacks cb 0x7f35e4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566dcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f6200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566dcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59896/0 (socket says 172.21.15.22:59896) 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40976/0 (socket says 172.21.15.22:40976) 2024-03-24T06:48:19.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f6200640 1 -- 172.21.15.22:0/3552001833 learned_addr learned my addr 172.21.15.22:0/3552001833 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f6200640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea56751df0 0x55ea56752270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 -- 172.21.15.22:0/3552001833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 msgr2=0x55ea566dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 -- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea56751df0 msgr2=0x55ea56752270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea56751df0 0x55ea56752270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 -- 172.21.15.22:0/3552001833 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e801e040 con 0x55ea566e2780 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35f4e00640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55ea566d4520 tx=0x7f35dc032e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f14b26b134b044c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35d7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.012+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc02cb30 con 0x55ea566e2780 2024-03-24T06:48:19.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35e800fe80 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9555+0+0 (secure 0 0 0) 0x7f35dc000e40 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35dc032770 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 10 client.27344 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ea5663ea70 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e800fe80 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f35dc0037b0 con 0x55ea566e2780 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 10 client.27344 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 10 client.27344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:19.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 20 client.27344 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea56759e70 0x55ea5675c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 4 client.27344 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.013+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea563cb890 con 0x55ea56759e70 2024-03-24T06:48:19.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.014+0000 7f35f6200640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea56759e70 0x55ea5675c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.014+0000 7f35f6200640 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea56759e70 0x55ea5675c260 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93bb7423a0a01fd7 server_cookie=dbf36a1523a91cd2 in_seq=0 out_seq=0 2024-03-24T06:48:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.014+0000 7f35d7400640 10 client.27344 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35d7400640 1 -- 172.21.15.22:0/3552001833 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea563cb890 con 0x55ea56759e70 2024-03-24T06:48:19.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35d7400640 10 client.27344 handle_command_reply: tid=0 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 client.27344 shutdown 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 msgr2=0x55ea566dd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55ea566d4520 tx=0x7f35dc032e30 comp rx=0 tx=0).stop 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 shutdown_connections 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ea56751df0 0x55ea56752270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ea56759e70 0x55ea5675c260 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea566e2780 0x55ea566dd0a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 --2- 172.21.15.22:0/3552001833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea566d9280 0x55ea566dcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35d7400640 0 client.27344 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 >> 172.21.15.22:0/3552001833 conn(0x55ea566d39b0 msgr2=0x55ea566d2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 shutdown_connections 2024-03-24T06:48:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 1 -- 172.21.15.22:0/3552001833 wait complete. 2024-03-24T06:48:19.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.015+0000 7f35fae19740 20 client.27344 trim_cache size 0 max 16384 2024-03-24T06:48:19.022 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 msgr2=0x7f35d80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 0x7f35d80a4770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55ea566d8d50 tx=0x7f35e40d40e0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 msgr2=0x7f35f019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f35e400e9c0 tx=0x7f35e400ee80 comp rx=0 tx=0).stop 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 shutdown_connections 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f35d80a22b0 0x7f35d80a4770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35f01a2c40 0x7f35f01a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35f010cff0 0x7f35f019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 --2- 172.21.15.22:0/3801961879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35f0107f20 0x7f35f019e3d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:19.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 >> 172.21.15.22:0/3801961879 conn(0x7f35f0075c50 msgr2=0x7f35f0076f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:19.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 shutdown_connections 2024-03-24T06:48:19.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:19.016+0000 7f35f8200640 1 -- 172.21.15.22:0/3801961879 wait complete. 2024-03-24T06:48:20.035 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:48:20.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:48:20.036 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:48:20.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 -- 172.21.15.22:0/2626164529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34381032c0 msgr2=0x7f343810b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 --2- 172.21.15.22:0/2626164529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34381032c0 0x7f343810b7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3424009900 tx=0x7f342402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 -- 172.21.15.22:0/2626164529 shutdown_connections 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 --2- 172.21.15.22:0/2626164529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f343810bce0 0x7f343810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 --2- 172.21.15.22:0/2626164529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34381032c0 0x7f343810b7a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 --2- 172.21.15.22:0/2626164529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381029a0 0x7f3438102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 -- 172.21.15.22:0/2626164529 >> 172.21.15.22:0/2626164529 conn(0x7f3438077770 msgr2=0x7f3438077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 -- 172.21.15.22:0/2626164529 shutdown_connections 2024-03-24T06:48:20.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.275+0000 7f343d000640 1 -- 172.21.15.22:0/2626164529 wait complete. 2024-03-24T06:48:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.276+0000 7f343d000640 1 Processor -- start 2024-03-24T06:48:20.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 -- start start 2024-03-24T06:48:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 0x7f343819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f343819cbf0 con 0x7f34381029a0 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f343819cd60 con 0x7f34381032c0 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f343d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f343819cf20 con 0x7f343810bce0 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f3435e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.277+0000 7f3435400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3435e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60408/0 (socket says 172.21.15.22:60408) 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3435400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40988/0 (socket says 172.21.15.22:40988) 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3435e00640 1 -- 172.21.15.22:0/3304076998 learned_addr learned my addr 172.21.15.22:0/3304076998 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 0x7f343819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 -- 172.21.15.22:0/3304076998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 msgr2=0x7f34381982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 -- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 msgr2=0x7f3438197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 -- 172.21.15.22:0/3304076998 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3424009560 con 0x7f343810bce0 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3435e00640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3435400640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.278+0000 7f3436800640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 0x7f343819c5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3430005fd0 tx=0x7f343000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.279+0000 7f342ea00640 1 -- 172.21.15.22:0/3304076998 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343000cd80 con 0x7f343810bce0 2024-03-24T06:48:20.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.279+0000 7f342ea00640 1 -- 172.21.15.22:0/3304076998 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3430005020 con 0x7f343810bce0 2024-03-24T06:48:20.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.279+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343819d200 con 0x7f343810bce0 2024-03-24T06:48:20.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.279+0000 7f342ea00640 1 -- 172.21.15.22:0/3304076998 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430010690 con 0x7f343810bce0 2024-03-24T06:48:20.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.279+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3438100700 con 0x7f343810bce0 2024-03-24T06:48:20.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.282+0000 7f343fe1f740 1 Processor -- start 2024-03-24T06:48:20.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.282+0000 7f342ea00640 1 -- 172.21.15.22:0/3304076998 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3430005500 con 0x7f343810bce0 2024-03-24T06:48:20.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 -- start start 2024-03-24T06:48:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098708210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x559098713720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5590986762d0 con 0x559098707e30 2024-03-24T06:48:20.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559098673800 con 0x559098711330 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f343fe1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55909866efe0 con 0x559098708750 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f3435400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098708210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59926/0 (socket says 172.21.15.22:59926) 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.283+0000 7f3435e00640 1 -- 172.21.15.22:0/3675593816 learned_addr learned my addr 172.21.15.22:0/3675593816 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435400640 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x559098713720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 -- 172.21.15.22:0/3675593816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 msgr2=0x559098713720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x559098713720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 -- 172.21.15.22:0/3675593816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 msgr2=0x559098708210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098708210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 -- 172.21.15.22:0/3675593816 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559098630e90 con 0x559098708750 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f342ea00640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 0x7f341c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f3435e00640 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3428011650 tx=0x7f3428009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb69166ff2de99cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.284+0000 7f342ea00640 1 -- 172.21.15.22:0/3304076998 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f3430014070 con 0x7f343810bce0 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f3435400640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 0x7f341c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f3423400640 1 -- 172.21.15.22:0/3675593816 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800f6c0 con 0x559098708750 2024-03-24T06:48:20.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f3423400640 1 -- 172.21.15.22:0/3675593816 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342800fd00 con 0x559098708750 2024-03-24T06:48:20.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f3423400640 1 -- 172.21.15.22:0/3675593816 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428019e00 con 0x559098708750 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 -- 172.21.15.22:0/3675593816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 msgr2=0x559098710df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3428011650 tx=0x7f3428009f90 comp rx=0 tx=0).stop 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f3435400640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 0x7f341c0a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3424005ab0 tx=0x7f3424005e60 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 -- 172.21.15.22:0/3675593816 shutdown_connections 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x559098713720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098708750 0x559098710df0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3675593816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098708210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.285+0000 7f343fe1f740 1 -- 172.21.15.22:0/3675593816 >> 172.21.15.22:0/3675593816 conn(0x559098702560 msgr2=0x559098702940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.286+0000 7f343fe1f740 1 -- 172.21.15.22:0/3675593816 shutdown_connections 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.286+0000 7f343fe1f740 1 -- 172.21.15.22:0/3675593816 wait complete. 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.286+0000 7f343fe1f740 1 -- start start 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.286+0000 7f343fe1f740 1 Processor -- start 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098708750 0x55909870e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x55909870ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559098613fc0 con 0x559098708750 2024-03-24T06:48:20.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559098613c60 con 0x559098711330 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5590983b70c0 con 0x559098707e30 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f343fe1f740 10 client.0 _ll_register_callbacks cb 0x55909862be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59942/0 (socket says 172.21.15.22:59942) 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098708750 0x55909870e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 -- 172.21.15.22:0/1233365116 learned_addr learned my addr 172.21.15.22:0/1233365116 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3436800640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x55909870ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 -- 172.21.15.22:0/1233365116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 msgr2=0x55909870ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x55909870ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 -- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098708750 msgr2=0x55909870e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098708750 0x55909870e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.287+0000 7f3435e00640 1 -- 172.21.15.22:0/1233365116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3428009e00 con 0x559098707e30 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f3435e00640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3428019980 tx=0x7f3428009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f958ca1c835132 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f3417e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34280197a0 con 0x559098707e30 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f342800f8d0 con 0x559098707e30 2024-03-24T06:48:20.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559098693e90 con 0x559098707e30 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f342800fa90 con 0x559098707e30 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f343fe1f740 10 client.27385 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.288+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55909866d660 con 0x559098707e30 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559098693e90 con 0x559098707e30 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f34280046c0 con 0x559098707e30 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 10 client.27385 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 10 client.27385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 20 client.27385 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:20.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098721710 0x559098723b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 4 client.27385 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559098665210 con 0x559098721710 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.289+0000 7f3435400640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098721710 0x559098723b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.290+0000 7f3435400640 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098721710 0x559098723b50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461182bbb6f70b2b server_cookie=ebc26d0a240366db in_seq=0 out_seq=0 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.290+0000 7f3417e00640 10 client.27385 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f3417e00640 1 -- 172.21.15.22:0/1233365116 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559098665210 con 0x559098721710 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f3417e00640 10 client.27385 handle_command_reply: tid=0 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 client.27385 shutdown 2024-03-24T06:48:20.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 msgr2=0x55909870e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3428019980 tx=0x7f3428009cc0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 shutdown_connections 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098721710 0x559098723b50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098711330 0x55909870ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f3417e00640 0 client.27385 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098708750 0x55909870e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 --2- 172.21.15.22:0/1233365116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x55909870e3c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.291+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 >> 172.21.15.22:0/1233365116 conn(0x559098702560 msgr2=0x5590987042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.292+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 shutdown_connections 2024-03-24T06:48:20.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.292+0000 7f343fe1f740 1 -- 172.21.15.22:0/1233365116 wait complete. 2024-03-24T06:48:20.299 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.292+0000 7f343fe1f740 20 client.27385 trim_cache size 0 max 16384 2024-03-24T06:48:20.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 Processor -- start 2024-03-24T06:48:20.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 -- start start 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098710af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 0x559098729460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5590986fd7d0 con 0x559098707e30 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559098703560 con 0x559098708750 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.311+0000 7f343fe1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55909870c910 con 0x559098711330 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41004/0 (socket says 172.21.15.22:41004) 2024-03-24T06:48:20.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 -- 172.21.15.22:0/2894988108 learned_addr learned my addr 172.21.15.22:0/2894988108 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435e00640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098710af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3436800640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 0x559098729460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 -- 172.21.15.22:0/2894988108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 msgr2=0x559098729460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 0x559098729460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 -- 172.21.15.22:0/2894988108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 msgr2=0x559098710af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098710af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 -- 172.21.15.22:0/2894988108 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559098609ae0 con 0x559098708750 2024-03-24T06:48:20.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3436800640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 0x559098729460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:20.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.312+0000 7f3435400640 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3424009230 tx=0x7f3424009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80cc1836ab6532e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f3422a00640 1 -- 172.21.15.22:0/2894988108 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424053040 con 0x559098708750 2024-03-24T06:48:20.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f3422a00640 1 -- 172.21.15.22:0/2894988108 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342402ce50 con 0x559098708750 2024-03-24T06:48:20.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f3422a00640 1 -- 172.21.15.22:0/2894988108 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342403c5d0 con 0x559098708750 2024-03-24T06:48:20.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 -- 172.21.15.22:0/2894988108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 msgr2=0x55909871cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3424009230 tx=0x7f3424009260 comp rx=0 tx=0).stop 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 -- 172.21.15.22:0/2894988108 shutdown_connections 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098711330 0x559098729460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x55909871cbc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 --2- 172.21.15.22:0/2894988108 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098707e30 0x559098710af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 -- 172.21.15.22:0/2894988108 >> 172.21.15.22:0/2894988108 conn(0x559098702560 msgr2=0x559098708b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 -- 172.21.15.22:0/2894988108 shutdown_connections 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.313+0000 7f343fe1f740 1 -- 172.21.15.22:0/2894988108 wait complete. 2024-03-24T06:48:20.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 -- start start 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 Processor -- start 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x559098715250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098711330 0x559098715790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559098786bb0 con 0x559098711330 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559098786d20 con 0x559098708750 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.314+0000 7f343fe1f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559098786ee0 con 0x559098707e30 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f343fe1f740 10 client.0 _ll_register_callbacks cb 0x7f342801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:20.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59950/0 (socket says 172.21.15.22:59950) 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 -- 172.21.15.22:0/3880271879 learned_addr learned my addr 172.21.15.22:0/3880271879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3436800640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098711330 0x559098715790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435400640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x559098715250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 -- 172.21.15.22:0/3880271879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 msgr2=0x559098715250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x559098715250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 -- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098711330 msgr2=0x559098715790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098711330 0x559098715790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 -- 172.21.15.22:0/3880271879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343810edd0 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3435e00640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3428011650 tx=0x7f3428004f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee83b211d63475ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.315+0000 7f3417e00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342801aea0 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f342800fd00 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f3428000e40 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3428005d30 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 10 client.27391 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55909862f6f0 con 0x559098707e30 2024-03-24T06:48:20.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342800fd00 con 0x559098707e30 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f34280040c0 con 0x559098707e30 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 10 client.27391 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 10 client.27391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 20 client.27391 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098788420 0x55909878a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 4 client.27391 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.316+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55909836e860 con 0x559098788420 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.317+0000 7f3435400640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098788420 0x55909878a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.317+0000 7f3417e00640 10 client.27391 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.317+0000 7f3435400640 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098788420 0x55909878a810 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1a3dc4db31c1a4e server_cookie=314d7c43e6fd6d87 in_seq=0 out_seq=0 2024-03-24T06:48:20.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.318+0000 7f3417e00640 1 -- 172.21.15.22:0/3880271879 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55909836e860 con 0x559098788420 2024-03-24T06:48:20.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.318+0000 7f3417e00640 10 client.27391 handle_command_reply: tid=0 2024-03-24T06:48:20.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.318+0000 7f343fe1f740 1 client.27391 shutdown 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.318+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 msgr2=0x559098714d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.318+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3428011650 tx=0x7f3428004f50 comp rx=0 tx=0).stop 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 shutdown_connections 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559098711330 0x559098715790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x559098788420 0x55909878a810 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559098708750 0x559098715250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 --2- 172.21.15.22:0/3880271879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559098707e30 0x559098714d10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f3417e00640 0 client.27391 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 >> 172.21.15.22:0/3880271879 conn(0x559098702560 msgr2=0x559098700d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 shutdown_connections 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 1 -- 172.21.15.22:0/3880271879 wait complete. 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.319+0000 7f343fe1f740 20 client.27391 trim_cache size 0 max 16384 2024-03-24T06:48:20.326 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:20.327 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:48:20.327 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "ef1c843e-8763-4411-ba62-1035d49bdf3d", 2024-03-24T06:48:20.327 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:48:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.320+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 msgr2=0x7f341c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.320+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 0x7f341c0a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3424005ab0 tx=0x7f3424005e60 comp rx=0 tx=0).stop 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.320+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 msgr2=0x7f343819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.320+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 0x7f343819c5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3430005fd0 tx=0x7f343000ee60 comp rx=0 tx=0).stop 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 shutdown_connections 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f341c0a22b0 0x7f341c0a4770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f343810bce0 0x7f343819c5f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34381032c0 0x7f34381982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 --2- 172.21.15.22:0/3304076998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34381029a0 0x7f3438197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 >> 172.21.15.22:0/3304076998 conn(0x7f3438077770 msgr2=0x7f34380fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.321+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 shutdown_connections 2024-03-24T06:48:20.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.322+0000 7f343d000640 1 -- 172.21.15.22:0/3304076998 wait complete. 2024-03-24T06:48:20.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:48:20.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:48:20.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:20 smithi022 ceph-mon[32003]: pgmap v257: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 189 B/s wr, 9 op/s 2024-03-24T06:48:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:20 smithi022 ceph-mon[32003]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T06:48:20.479 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:20 smithi143 ceph-mon[36520]: pgmap v257: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 189 B/s wr, 9 op/s 2024-03-24T06:48:20.479 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:20 smithi143 ceph-mon[36520]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T06:48:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:20 smithi155 ceph-mon[36262]: pgmap v257: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 189 B/s wr, 9 op/s 2024-03-24T06:48:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:20 smithi155 ceph-mon[36262]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T06:48:20.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.549+0000 7fa12dc00640 1 -- 172.21.15.22:0/2176451576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa12810bb60 msgr2=0x7fa12810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.549+0000 7fa12dc00640 1 --2- 172.21.15.22:0/2176451576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa12810bb60 0x7fa12810e000 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa120009ca0 tx=0x7fa12002c180 comp rx=0 tx=0).stop 2024-03-24T06:48:20.550 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 -- 172.21.15.22:0/2176451576 shutdown_connections 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 --2- 172.21.15.22:0/2176451576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa12810bb60 0x7fa12810e000 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 --2- 172.21.15.22:0/2176451576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128103060 0x7fa12810b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 --2- 172.21.15.22:0/2176451576 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa128102740 0x7fa128102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 -- 172.21.15.22:0/2176451576 >> 172.21.15.22:0/2176451576 conn(0x7fa1280fbf60 msgr2=0x7fa1280fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 -- 172.21.15.22:0/2176451576 shutdown_connections 2024-03-24T06:48:20.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.550+0000 7fa12dc00640 1 -- 172.21.15.22:0/2176451576 wait complete. 2024-03-24T06:48:20.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 Processor -- start 2024-03-24T06:48:20.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 -- start start 2024-03-24T06:48:20.553 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa12810bb60 0x7fa128100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281a5ef0 con 0x7fa12810bb60 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281a6060 con 0x7fa128102740 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa12dc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281a6220 con 0x7fa128103060 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.551+0000 7fa127e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa12d200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa12810bb60 0x7fa128100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.554 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59968/0 (socket says 172.21.15.22:59968) 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41034/0 (socket says 172.21.15.22:41034) 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127400640 1 -- 172.21.15.22:0/125739206 learned_addr learned my addr 172.21.15.22:0/125739206 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 -- 172.21.15.22:0/125739206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 msgr2=0x7fa1281008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 -- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa12810bb60 msgr2=0x7fa128100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa12810bb60 0x7fa128100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 -- 172.21.15.22:0/125739206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa120009900 con 0x7fa128102740 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127400640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:20.555 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.552+0000 7fa127e00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa11800e9f0 tx=0x7fa11800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.553+0000 7fa124c00640 1 -- 172.21.15.22:0/125739206 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11800ccc0 con 0x7fa128102740 2024-03-24T06:48:20.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.553+0000 7fa124c00640 1 -- 172.21.15.22:0/125739206 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11800ce20 con 0x7fa128102740 2024-03-24T06:48:20.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.553+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281a6500 con 0x7fa128102740 2024-03-24T06:48:20.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.553+0000 7fa124c00640 1 -- 172.21.15.22:0/125739206 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118018920 con 0x7fa128102740 2024-03-24T06:48:20.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.553+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281a6a60 con 0x7fa128102740 2024-03-24T06:48:20.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.557+0000 7fa131f7a740 1 Processor -- start 2024-03-24T06:48:20.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.557+0000 7fa131f7a740 1 -- start start 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa124c00640 1 -- 172.21.15.22:0/125739206 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa118018a80 con 0x7fa128102740 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e93253e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e93306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e9293790 con 0x5636e932e2d0 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e9290cc0 con 0x5636e9324f60 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa131f7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e928c4a0 con 0x5636e9325c40 2024-03-24T06:48:20.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa12d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa127e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e93253e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa12d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59982/0 (socket says 172.21.15.22:59982) 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa12d200640 1 -- 172.21.15.22:0/4028321008 learned_addr learned my addr 172.21.15.22:0/4028321008 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.558+0000 7fa127400640 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e93306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 -- 172.21.15.22:0/4028321008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 msgr2=0x5636e93253e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e93253e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 -- 172.21.15.22:0/4028321008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 msgr2=0x5636e93306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e93306c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 -- 172.21.15.22:0/4028321008 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e924dc20 con 0x5636e9325c40 2024-03-24T06:48:20.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa127400640 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e93306c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa12d200640 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa12002c790 tx=0x7fa120005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14441e89b751c855 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa11d600640 1 -- 172.21.15.22:0/4028321008 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120034440 con 0x5636e9325c40 2024-03-24T06:48:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa11d600640 1 -- 172.21.15.22:0/4028321008 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa120003970 con 0x5636e9325c40 2024-03-24T06:48:20.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa11d600640 1 -- 172.21.15.22:0/4028321008 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120032e30 con 0x5636e9325c40 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.559+0000 7fa124c00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 0x7fa10c0a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa124c00640 1 -- 172.21.15.22:0/125739206 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa118014070 con 0x7fa128102740 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa127e00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 0x7fa10c0a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 -- 172.21.15.22:0/4028321008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 msgr2=0x5636e9324990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa12002c790 tx=0x7fa120005f80 comp rx=0 tx=0).stop 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 -- 172.21.15.22:0/4028321008 shutdown_connections 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e93306c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e93253e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4028321008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e9324990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 -- 172.21.15.22:0/4028321008 >> 172.21.15.22:0/4028321008 conn(0x5636e92ebb30 msgr2=0x5636e931f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.560+0000 7fa131f7a740 1 -- 172.21.15.22:0/4028321008 shutdown_connections 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.561+0000 7fa127e00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 0x7fa10c0a47a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5636e9326190 tx=0x7fa1180d1f50 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.561+0000 7fa131f7a740 1 -- 172.21.15.22:0/4028321008 wait complete. 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.561+0000 7fa131f7a740 1 -- start start 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.561+0000 7fa131f7a740 1 Processor -- start 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e932bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e9230f20 con 0x5636e932e2d0 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e9230bc0 con 0x5636e9324f60 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e8ffa1c0 con 0x5636e9325c40 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa131f7a740 10 client.0 _ll_register_callbacks cb 0x5636e8f9cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:59994/0 (socket says 172.21.15.22:59994) 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 -- 172.21.15.22:0/1666512500 learned_addr learned my addr 172.21.15.22:0/1666512500 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa12d200640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa12d200640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e932bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 -- 172.21.15.22:0/1666512500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 msgr2=0x5636e932b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 -- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 msgr2=0x5636e932bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.562+0000 7fa127400640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e932bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa127400640 1 -- 172.21.15.22:0/1666512500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12810ec80 con 0x5636e9325c40 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa12d200640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e932bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa127400640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa1140129e0 tx=0x7fa114012ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9e71fb5023314 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa112000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114017070 con 0x5636e9325c40 2024-03-24T06:48:20.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5636e9284160 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa11400fa90 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa1140122e0 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa131f7a740 10 client.27403 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.563+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5636e928ab20 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5636e9284160 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa11401c420 con 0x5636e9325c40 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 10 client.27403 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:20.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 10 client.27403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 20 client.27403 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e933e840 0x5636e9340c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 4 client.27403 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:20.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e8f8b7e0 con 0x5636e933e840 2024-03-24T06:48:20.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.564+0000 7fa12d200640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e933e840 0x5636e9340c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.565+0000 7fa12d200640 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e933e840 0x5636e9340c60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae22a5fd90f77396 server_cookie=f4c08f04c8dcb3a3 in_seq=0 out_seq=0 2024-03-24T06:48:20.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.565+0000 7fa112000640 10 client.27403 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa112000640 1 -- 172.21.15.22:0/1666512500 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636e8f8b7e0 con 0x5636e933e840 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa112000640 10 client.27403 handle_command_reply: tid=0 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 client.27403 shutdown 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 msgr2=0x5636e932b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa1140129e0 tx=0x7fa114012ea0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 shutdown_connections 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e932e2d0 0x5636e932bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9325c40 0x5636e932b820 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e933e840 0x5636e9340c60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 --2- 172.21.15.22:0/1666512500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 >> 172.21.15.22:0/1666512500 conn(0x5636e92ebb30 msgr2=0x5636e9321700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa112000640 0 client.27403 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 shutdown_connections 2024-03-24T06:48:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 1 -- 172.21.15.22:0/1666512500 wait complete. 2024-03-24T06:48:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.566+0000 7fa131f7a740 20 client.27403 trim_cache size 0 max 16384 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 Processor -- start 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 -- start start 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9339bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e932e2d0 0x5636e9346530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e933e650 con 0x5636e9325c40 2024-03-24T06:48:20.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e932dd40 con 0x5636e9324f60 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa131f7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e93241e0 con 0x5636e932e2d0 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa12d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa12d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:41052/0 (socket says 172.21.15.22:41052) 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa12d200640 1 -- 172.21.15.22:0/4031703082 learned_addr learned my addr 172.21.15.22:0/4031703082 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa127400640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9339bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.586+0000 7fa12d200640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e932e2d0 0x5636e9346530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 -- 172.21.15.22:0/4031703082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e932e2d0 msgr2=0x5636e9346530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e932e2d0 0x5636e9346530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 -- 172.21.15.22:0/4031703082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 msgr2=0x5636e932a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 -- 172.21.15.22:0/4031703082 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e92269a0 con 0x5636e9325c40 2024-03-24T06:48:20.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa12d200640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa127400640 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9339bc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa11401c850 tx=0x7fa114012360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239eaea2ff3c8bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa11cc00640 1 -- 172.21.15.22:0/4031703082 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114017040 con 0x5636e9325c40 2024-03-24T06:48:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa11cc00640 1 -- 172.21.15.22:0/4031703082 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11401cac0 con 0x5636e9325c40 2024-03-24T06:48:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.587+0000 7fa11cc00640 1 -- 172.21.15.22:0/4031703082 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140053c0 con 0x5636e9325c40 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 -- 172.21.15.22:0/4031703082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 msgr2=0x5636e9339bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9339bc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa11401c850 tx=0x7fa114012360 comp rx=0 tx=0).stop 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 -- 172.21.15.22:0/4031703082 shutdown_connections 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e932e2d0 0x5636e9346530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9339bc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 --2- 172.21.15.22:0/4031703082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e9324f60 0x5636e932a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 -- 172.21.15.22:0/4031703082 >> 172.21.15.22:0/4031703082 conn(0x5636e92ebb30 msgr2=0x5636e9321670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 -- 172.21.15.22:0/4031703082 shutdown_connections 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.588+0000 7fa131f7a740 1 -- 172.21.15.22:0/4031703082 wait complete. 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 -- start start 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 Processor -- start 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9324f60 0x5636e9331be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 0x5636e9332660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5636e93a3b60 con 0x5636e9325c40 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5636e93a3ca0 con 0x5636e932e2d0 2024-03-24T06:48:20.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5636e93a3e60 con 0x5636e9324f60 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.589+0000 7fa131f7a740 10 client.0 _ll_register_callbacks cb 0x7fa114009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa12d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9324f60 0x5636e9331be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60508/0 (socket says 172.21.15.22:60508) 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 -- 172.21.15.22:0/5374081 learned_addr learned my addr 172.21.15.22:0/5374081 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa12d200640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 0x5636e9332660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 -- 172.21.15.22:0/5374081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9324f60 msgr2=0x5636e9331be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9324f60 0x5636e9331be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 -- 172.21.15.22:0/5374081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 msgr2=0x5636e9332660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 0x5636e9332660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa127400640 1 -- 172.21.15.22:0/5374081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e924dc20 con 0x5636e9325c40 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.590+0000 7fa12d200640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 0x5636e9332660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa127400640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa11401c970 tx=0x7fa1140124e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd0422b64132b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa112000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114017070 con 0x5636e9325c40 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa120003970 con 0x5636e9325c40 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fa11400fca0 con 0x5636e9325c40 2024-03-24T06:48:20.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa11402f070 con 0x5636e9325c40 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa131f7a740 10 client.17022 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.591+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5636e928ab20 con 0x5636e9325c40 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa120003970 con 0x5636e9325c40 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa114003b50 con 0x5636e9325c40 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 10 client.17022 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 20 client.17022 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e93a5250 0x5636e93a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:20.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 4 client.17022 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:48:20.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636e8fc8720 con 0x5636e93a5250 2024-03-24T06:48:20.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.592+0000 7fa12d200640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e93a5250 0x5636e93a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa12d200640 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e93a5250 0x5636e93a7670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7300579feaf1f275 server_cookie=80c56ef9c0de1040 in_seq=0 out_seq=0 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa112000640 10 client.17022 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa112000640 1 -- 172.21.15.22:0/5374081 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5636e8fc8720 con 0x5636e93a5250 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa112000640 10 client.17022 handle_command_reply: tid=0 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 client.17022 shutdown 2024-03-24T06:48:20.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 msgr2=0x5636e9332120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa11401c970 tx=0x7fa1140124e0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 shutdown_connections 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5636e93a5250 0x5636e93a7670 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5636e932e2d0 0x5636e9332660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5636e9325c40 0x5636e9332120 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 --2- 172.21.15.22:0/5374081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5636e9324f60 0x5636e9331be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa112000640 0 client.17022 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.593+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 >> 172.21.15.22:0/5374081 conn(0x5636e92ebb30 msgr2=0x5636e9320de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 shutdown_connections 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa131f7a740 1 -- 172.21.15.22:0/5374081 wait complete. 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa131f7a740 20 client.17022 trim_cache size 0 max 16384 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:48:20.601 INFO:teuthology.orchestra.run.smithi022.stdout: "ef1c843e-8763-4411-ba62-1035d49bdf3d": { 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "ef1c843e-8763-4411-ba62-1035d49bdf3d", 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:48:20.602 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 msgr2=0x7fa10c0a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 0x7fa10c0a47a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5636e9326190 tx=0x7fa1180d1f50 comp rx=0 tx=0).stop 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 msgr2=0x7fa128100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.594+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa11800e9f0 tx=0x7fa11800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 shutdown_connections 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa10c0a22e0 0x7fa10c0a47a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa12810bb60 0x7fa128100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa128103060 0x7fa1281008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 --2- 172.21.15.22:0/125739206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa128102740 0x7fa128100390 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 >> 172.21.15.22:0/125739206 conn(0x7fa1280fbf60 msgr2=0x7fa1280fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 shutdown_connections 2024-03-24T06:48:20.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:20.595+0000 7fa12dc00640 1 -- 172.21.15.22:0/125739206 wait complete. 2024-03-24T06:48:20.612 INFO:tasks.cephfs.filesystem:scrub status for tag:ef1c843e-8763-4411-ba62-1035d49bdf3d - {'path': '/', 'tag': 'ef1c843e-8763-4411-ba62-1035d49bdf3d', 'options': 'recursive,force'} 2024-03-24T06:48:21.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:48:21.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:21.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:21.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:21 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:21.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:21.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:21 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:48:21.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:21.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:21.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:21.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:21 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:22 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:22.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:22 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:22.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:22 smithi155 ceph-mon[36262]: pgmap v259: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:22.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:22 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:22.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:22 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:22.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:22 smithi022 ceph-mon[32003]: pgmap v259: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:22 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:22 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:22.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:22 smithi143 ceph-mon[36520]: pgmap v259: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:48:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:24 smithi155 ceph-mon[36262]: pgmap v260: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:24.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:24 smithi022 ceph-mon[32003]: pgmap v260: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:24.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:24 smithi143 ceph-mon[36520]: pgmap v260: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:26 smithi155 ceph-mon[36262]: pgmap v261: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:26.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:26 smithi022 ceph-mon[32003]: pgmap v261: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:26.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:26 smithi143 ceph-mon[36520]: pgmap v261: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:48:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:27 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:48:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:27 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:48:27.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:27 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:48:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:28 smithi155 ceph-mon[36262]: pgmap v262: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:28 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:48:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:28 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:48:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:28 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:48:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:28 smithi022 ceph-mon[32003]: pgmap v262: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:28 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:48:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:28 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:48:28.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:28 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:48:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:28 smithi143 ceph-mon[36520]: pgmap v262: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:28 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:48:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:28 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:48:28.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:28 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:48:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:30 smithi155 ceph-mon[36262]: pgmap v263: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:30 smithi022 ceph-mon[32003]: pgmap v263: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:30 smithi143 ceph-mon[36520]: pgmap v263: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-24T06:48:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:32 smithi155 ceph-mon[36262]: pgmap v264: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:48:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:32 smithi022 ceph-mon[32003]: pgmap v264: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:48:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:32 smithi143 ceph-mon[36520]: pgmap v264: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:48:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:33 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:48:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:33 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:48:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:33 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:48:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:33 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:48:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:33 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:48:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:33 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:33 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:33 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:33 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:33.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:33 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:48:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:33 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:48:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:33 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:48:33.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:33 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:48:33.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:33 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:48:33.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:33 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:48:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:34 smithi155 ceph-mon[36262]: pgmap v265: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:48:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:34 smithi022 ceph-mon[32003]: pgmap v265: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:48:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:34 smithi143 ceph-mon[36520]: pgmap v265: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:48:36.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:36 smithi022 ceph-mon[32003]: pgmap v266: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:36 smithi143 ceph-mon[36520]: pgmap v266: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:36 smithi155 ceph-mon[36262]: pgmap v266: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:38 smithi022 ceph-mon[32003]: pgmap v267: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-24T06:48:38.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:38 smithi143 ceph-mon[36520]: pgmap v267: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-24T06:48:38.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:38 smithi155 ceph-mon[36262]: pgmap v267: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-24T06:48:40.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:40 smithi022 ceph-mon[32003]: pgmap v268: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:40 smithi143 ceph-mon[36520]: pgmap v268: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:40.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:40 smithi155 ceph-mon[36262]: pgmap v268: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:41.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:41 smithi022 ceph-mon[32003]: pgmap v269: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:41.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:41 smithi143 ceph-mon[36520]: pgmap v269: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:41.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:41 smithi155 ceph-mon[36262]: pgmap v269: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:44 smithi022 ceph-mon[32003]: pgmap v270: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:44 smithi143 ceph-mon[36520]: pgmap v270: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:44.478 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:44 smithi155 ceph-mon[36262]: pgmap v270: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:46.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:46 smithi022 ceph-mon[32003]: pgmap v271: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:46 smithi143 ceph-mon[36520]: pgmap v271: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:46 smithi155 ceph-mon[36262]: pgmap v271: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:48.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:48 smithi022 ceph-mon[32003]: pgmap v272: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:48 smithi143 ceph-mon[36520]: pgmap v272: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:48 smithi155 ceph-mon[36262]: pgmap v272: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:50.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:50 smithi022 ceph-mon[32003]: pgmap v273: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:50 smithi143 ceph-mon[36520]: pgmap v273: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:50 smithi155 ceph-mon[36262]: pgmap v273: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:48:50.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:48:50.613 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 -- 172.21.15.22:0/361656913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac1028e0 msgr2=0x7feaac102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 --2- 172.21.15.22:0/361656913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac1028e0 0x7feaac102cc0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fea9c009ca0 tx=0x7fea9c02c180 comp rx=0 tx=0).stop 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 -- 172.21.15.22:0/361656913 shutdown_connections 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 --2- 172.21.15.22:0/361656913 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 0x7feaac10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 --2- 172.21.15.22:0/361656913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac103200 0x7feaac10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 --2- 172.21.15.22:0/361656913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac1028e0 0x7feaac102cc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 -- 172.21.15.22:0/361656913 >> 172.21.15.22:0/361656913 conn(0x7feaac0fc0e0 msgr2=0x7feaac0fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.857+0000 7feab2000640 1 -- 172.21.15.22:0/361656913 shutdown_connections 2024-03-24T06:48:50.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.858+0000 7feab2000640 1 -- 172.21.15.22:0/361656913 wait complete. 2024-03-24T06:48:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.858+0000 7feab2000640 1 Processor -- start 2024-03-24T06:48:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 -- start start 2024-03-24T06:48:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 0x7feaac199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 0x7feaac19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19ee50 con 0x7feaac10bd00 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19efc0 con 0x7feaac1028e0 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feab2000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feaac19f180 con 0x7feaac103200 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaaaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 0x7feaac199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaaaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37266/0 (socket says 172.21.15.22:37266) 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaaaa00640 1 -- 172.21.15.22:0/219796789 learned_addr learned my addr 172.21.15.22:0/219796789 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 0x7feaac199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58912/0 (socket says 172.21.15.22:58912) 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.859+0000 7feaabe00640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 0x7feaac19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 -- 172.21.15.22:0/219796789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 msgr2=0x7feaac199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 0x7feaac199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 -- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 msgr2=0x7feaac19e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 0x7feaac19e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 -- 172.21.15.22:0/219796789 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c009900 con 0x7feaac103200 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaaaa00640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fea9800e9f0 tx=0x7fea9800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaa3e00640 1 -- 172.21.15.22:0/219796789 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98009670 con 0x7feaac103200 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac19f460 con 0x7feaac103200 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaa3e00640 1 -- 172.21.15.22:0/219796789 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea98005500 con 0x7feaac103200 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.860+0000 7feaa3e00640 1 -- 172.21.15.22:0/219796789 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea98010430 con 0x7feaac103200 2024-03-24T06:48:50.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.861+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac1a6ce0 con 0x7feaac103200 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.864+0000 7feab4de9740 1 Processor -- start 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.864+0000 7feab4de9740 1 -- start start 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed922a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556ed918d790 con 0x556ed92280f0 2024-03-24T06:48:50.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ed918acc0 con 0x556ed921fa60 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feab4de9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556ed91864a0 con 0x556ed921ed80 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaabe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37272/0 (socket says 172.21.15.22:37272) 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaabe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58920/0 (socket says 172.21.15.22:58920) 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 -- 172.21.15.22:0/1916099818 learned_addr learned my addr 172.21.15.22:0/1916099818 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:50.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 -- 172.21.15.22:0/1916099818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 msgr2=0x556ed921e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 -- 172.21.15.22:0/1916099818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 msgr2=0x556ed922a4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed922a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaab400640 1 -- 172.21.15.22:0/1916099818 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed9147c20 con 0x556ed921ed80 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.865+0000 7feaabe00640 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:50.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.866+0000 7feaa3e00640 1 -- 172.21.15.22:0/219796789 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fea98005020 con 0x7feaac103200 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.866+0000 7feaab400640 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fea9c037040 tx=0x7fea9c032340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f774679d98af5bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.866+0000 7fea97e00640 1 -- 172.21.15.22:0/1916099818 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c034440 con 0x556ed921ed80 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.866+0000 7fea97e00640 1 -- 172.21.15.22:0/1916099818 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea9c003970 con 0x556ed921ed80 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.866+0000 7fea97e00640 1 -- 172.21.15.22:0/1916099818 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c004040 con 0x556ed921ed80 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- 172.21.15.22:0/1916099818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 msgr2=0x556ed921f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fea9c037040 tx=0x7fea9c032340 comp rx=0 tx=0).stop 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- 172.21.15.22:0/1916099818 shutdown_connections 2024-03-24T06:48:50.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed922a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed921f200 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 --2- 172.21.15.22:0/1916099818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed921e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- 172.21.15.22:0/1916099818 >> 172.21.15.22:0/1916099818 conn(0x556ed91e5b30 msgr2=0x556ed92197d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- 172.21.15.22:0/1916099818 shutdown_connections 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- 172.21.15.22:0/1916099818 wait complete. 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feaa3e00640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 0x7fea900a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feaa3e00640 1 -- 172.21.15.22:0/219796789 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fea98014070 con 0x7feaac103200 2024-03-24T06:48:50.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feaab400640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 0x7fea900a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.867+0000 7feab4de9740 1 -- start start 2024-03-24T06:48:50.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 Processor -- start 2024-03-24T06:48:50.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feaab400640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 0x7fea900a4600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea9c02c790 tx=0x7fea9c041040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921fa60 0x556ed9225680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 0x556ed9225bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556ed912af20 con 0x556ed921fa60 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ed912abc0 con 0x556ed92280f0 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.868+0000 7feab4de9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556ed8ef41c0 con 0x556ed921ed80 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feab4de9740 10 client.0 _ll_register_callbacks cb 0x556ed8e96be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaabe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 0x556ed9225bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37284/0 (socket says 172.21.15.22:37284) 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 -- 172.21.15.22:0/3777982127 learned_addr learned my addr 172.21.15.22:0/3777982127 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaaaa00640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921fa60 0x556ed9225680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 -- 172.21.15.22:0/3777982127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 msgr2=0x556ed9225bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 0x556ed9225bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 -- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921fa60 msgr2=0x556ed9225680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921fa60 0x556ed9225680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 -- 172.21.15.22:0/3777982127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac10ee20 con 0x556ed921ed80 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaabe00640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 0x556ed9225bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7feaab400640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fea9c000f80 tx=0x7fea9c032ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b327c6cd2442c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.869+0000 7fea94c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c04a9a0 con 0x556ed921ed80 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fea9c04ab00 con 0x556ed921ed80 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ed917e160 con 0x556ed921ed80 2024-03-24T06:48:50.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fea9c04acc0 con 0x556ed921ed80 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 10 client.27421 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ed9184b20 con 0x556ed921ed80 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556ed917e160 con 0x556ed921ed80 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fea9c052040 con 0x556ed921ed80 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 10 client.27421 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 10 client.27421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:50.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 20 client.27421 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.870+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed92383c0 0x556ed923a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.871+0000 7feab4de9740 4 client.27421 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.871+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed8e857e0 con 0x556ed92383c0 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.871+0000 7feaaaa00640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed92383c0 0x556ed923a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.871+0000 7feaaaa00640 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed92383c0 0x556ed923a7b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56b6486a57c0cdd4 server_cookie=a614ebd6b6bda70d in_seq=0 out_seq=0 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.871+0000 7fea94c00640 10 client.27421 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:50.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7fea94c00640 1 -- 172.21.15.22:0/3777982127 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ed8e857e0 con 0x556ed92383c0 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7fea94c00640 10 client.27421 handle_command_reply: tid=0 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 client.27421 shutdown 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 msgr2=0x556ed9225140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fea9c000f80 tx=0x7fea9c032ae0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 shutdown_connections 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed92383c0 0x556ed923a7b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed92280f0 0x556ed9225bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921fa60 0x556ed9225680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 --2- 172.21.15.22:0/3777982127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921ed80 0x556ed9225140 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.872+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 >> 172.21.15.22:0/3777982127 conn(0x556ed91e5b30 msgr2=0x556ed921b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.873+0000 7fea94c00640 0 client.27421 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.873+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 shutdown_connections 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.873+0000 7feab4de9740 1 -- 172.21.15.22:0/3777982127 wait complete. 2024-03-24T06:48:50.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.875+0000 7feab4de9740 20 client.27421 trim_cache size 0 max 16384 2024-03-24T06:48:50.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 Processor -- start 2024-03-24T06:48:50.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 -- start start 2024-03-24T06:48:50.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 0x556ed92240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed9219300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed92280f0 0x556ed92339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556ed921e000 con 0x556ed921ed80 2024-03-24T06:48:50.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ed9213c30 con 0x556ed921fa60 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.898+0000 7feab4de9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556ed9233f20 con 0x556ed92280f0 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaab400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 0x556ed92240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed9219300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaab400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 0x556ed92240a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59890/0 (socket says 172.21.15.22:59890) 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaab400640 1 -- 172.21.15.22:0/2163060204 learned_addr learned my addr 172.21.15.22:0/2163060204 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaabe00640 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed92280f0 0x556ed92339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 -- 172.21.15.22:0/2163060204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed92280f0 msgr2=0x556ed92339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed92280f0 0x556ed92339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 -- 172.21.15.22:0/2163060204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 msgr2=0x556ed92240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 0x556ed92240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 -- 172.21.15.22:0/2163060204 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed91209a0 con 0x556ed921fa60 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.899+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed9219300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fea980cbb30 tx=0x7fea9809b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54040e6bdc8ecbff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7fea97400640 1 -- 172.21.15.22:0/2163060204 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9809ef20 con 0x556ed921fa60 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7fea97400640 1 -- 172.21.15.22:0/2163060204 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea980a3570 con 0x556ed921fa60 2024-03-24T06:48:50.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7fea97400640 1 -- 172.21.15.22:0/2163060204 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea980991f0 con 0x556ed921fa60 2024-03-24T06:48:50.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7feab4de9740 1 -- 172.21.15.22:0/2163060204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 msgr2=0x556ed9219300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7feab4de9740 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed9219300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fea980cbb30 tx=0x7fea9809b5b0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.900+0000 7feab4de9740 1 -- 172.21.15.22:0/2163060204 shutdown_connections 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed92280f0 0x556ed92339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921fa60 0x556ed9219300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 --2- 172.21.15.22:0/2163060204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed921ed80 0x556ed92240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 -- 172.21.15.22:0/2163060204 >> 172.21.15.22:0/2163060204 conn(0x556ed91e5b30 msgr2=0x556ed9218850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 -- 172.21.15.22:0/2163060204 shutdown_connections 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 -- 172.21.15.22:0/2163060204 wait complete. 2024-03-24T06:48:50.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 -- start start 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.901+0000 7feab4de9740 1 Processor -- start 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed9228e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556ed9230580 con 0x556ed92280f0 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ed92306f0 con 0x556ed921ed80 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556ed9230860 con 0x556ed921fa60 2024-03-24T06:48:50.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feab4de9740 10 client.0 _ll_register_callbacks cb 0x7feaa4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaaaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaabe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed9228e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaaaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37298/0 (socket says 172.21.15.22:37298) 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaaaa00640 1 -- 172.21.15.22:0/2322368373 learned_addr learned my addr 172.21.15.22:0/2322368373 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.902+0000 7feaab400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:58938/0 (socket says 172.21.15.22:58938) 2024-03-24T06:48:50.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 -- 172.21.15.22:0/2322368373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 msgr2=0x556ed9222670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 -- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 msgr2=0x556ed9228e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed9228e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 -- 172.21.15.22:0/2322368373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac1a6ce0 con 0x556ed921fa60 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaab400640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feaaaa00640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fea980cbbe0 tx=0x7fea9809ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84717c81db869554 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7fea94c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:50.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea980990d0 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fea980b91c0 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ed9219960 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fea980cbdb0 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feab4de9740 10 client.27427 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.903+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ed9146480 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556ed9219960 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fea9801f060 con 0x556ed921fa60 2024-03-24T06:48:50.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 10 client.27427 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 10 client.27427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 20 client.27427 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed929f100 0x556ed92a1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 4 client.27427 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fea9c003970 con 0x556ed929f100 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.904+0000 7feaab400640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed929f100 0x556ed92a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7feaab400640 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed929f100 0x556ed92a1520 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b0876dca545d044 server_cookie=128715ddfb3d40d4 in_seq=0 out_seq=0 2024-03-24T06:48:50.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7fea94c00640 10 client.27427 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7fea94c00640 1 -- 172.21.15.22:0/2322368373 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fea9c003970 con 0x556ed929f100 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7fea94c00640 10 client.27427 handle_command_reply: tid=0 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7feab4de9740 1 client.27427 shutdown 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 msgr2=0x556ed9222bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.905+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fea980cbbe0 tx=0x7fea9809ef20 comp rx=0 tx=0).stop 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 shutdown_connections 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556ed92280f0 0x556ed9228e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556ed921fa60 0x556ed9222bb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556ed929f100 0x556ed92a1520 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 --2- 172.21.15.22:0/2322368373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ed921ed80 0x556ed9222670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 >> 172.21.15.22:0/2322368373 conn(0x556ed91e5b30 msgr2=0x556ed921a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7fea94c00640 0 client.27427 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 shutdown_connections 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 1 -- 172.21.15.22:0/2322368373 wait complete. 2024-03-24T06:48:50.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab4de9740 20 client.27427 trim_cache size 0 max 16384 2024-03-24T06:48:50.915 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:50.915 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:48:50.915 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:48:50.915 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 msgr2=0x7fea900a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 0x7fea900a4600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fea9c02c790 tx=0x7fea9c041040 comp rx=0 tx=0).stop 2024-03-24T06:48:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 msgr2=0x7feaac19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.906+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fea9800e9f0 tx=0x7fea9800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 shutdown_connections 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fea900a2140 0x7fea900a4600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feaac10bd00 0x7feaac19e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feaac103200 0x7feaac19a520 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 --2- 172.21.15.22:0/219796789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaac1028e0 0x7feaac199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 >> 172.21.15.22:0/219796789 conn(0x7feaac0fc0e0 msgr2=0x7feaac100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 shutdown_connections 2024-03-24T06:48:50.917 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:50.907+0000 7feab2000640 1 -- 172.21.15.22:0/219796789 wait complete. 2024-03-24T06:48:50.926 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:48:50.927 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 -- 172.21.15.22:0/1081004156 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d81098b0 msgr2=0x7f05d8109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 --2- 172.21.15.22:0/1081004156 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d81098b0 0x7f05d8109c90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009930 tx=0x7f05c802bc20 comp rx=0 tx=0).stop 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 -- 172.21.15.22:0/1081004156 shutdown_connections 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 --2- 172.21.15.22:0/1081004156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d8104260 0x7f05d8110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 --2- 172.21.15.22:0/1081004156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81038a0 0x7f05d8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 --2- 172.21.15.22:0/1081004156 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d81098b0 0x7f05d8109c90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 -- 172.21.15.22:0/1081004156 >> 172.21.15.22:0/1081004156 conn(0x7f05d80ff370 msgr2=0x7f05d8101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 -- 172.21.15.22:0/1081004156 shutdown_connections 2024-03-24T06:48:51.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.153+0000 7f05dd800640 1 -- 172.21.15.22:0/1081004156 wait complete. 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.154+0000 7f05dd800640 1 Processor -- start 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 -- start start 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 0x7f05d819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 0x7f05d819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f05d819edf0 con 0x7f05d8104260 2024-03-24T06:48:51.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f05d819ef60 con 0x7f05d81098b0 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05dd800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05d819f120 con 0x7f05d81038a0 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05ce000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 0x7f05d819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05d6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05ce000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 0x7f05d819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59922/0 (socket says 172.21.15.22:59922) 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05d7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 0x7f05d819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05ce000640 1 -- 172.21.15.22:0/69786222 learned_addr learned my addr 172.21.15.22:0/69786222 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.155+0000 7f05d6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37314/0 (socket says 172.21.15.22:37314) 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 -- 172.21.15.22:0/69786222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 msgr2=0x7f05d819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 0x7f05d819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 -- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 msgr2=0x7f05d819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 0x7f05d819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 -- 172.21.15.22:0/69786222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c0009490 con 0x7f05d81038a0 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d7400640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 0x7f05d819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05d6a00640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f05c8036040 tx=0x7f05c802c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c802dea0 con 0x7f05d81038a0 2024-03-24T06:48:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05c80042d0 con 0x7f05d81038a0 2024-03-24T06:48:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c8009560 con 0x7f05d81038a0 2024-03-24T06:48:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.156+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8003b50 con 0x7f05d81038a0 2024-03-24T06:48:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.157+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d81a6aa0 con 0x7f05d81038a0 2024-03-24T06:48:51.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.158+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d8068950 con 0x7f05d81038a0 2024-03-24T06:48:51.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.160+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f05c8003cb0 con 0x7f05d81038a0 2024-03-24T06:48:51.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.162+0000 7f05cfe00640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 0x7f05bc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.162+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f05c80f0740 con 0x7f05d81038a0 2024-03-24T06:48:51.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.162+0000 7f05ce000640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 0x7f05bc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.163+0000 7f05ce000640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 0x7f05bc0a4770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05d8101990 tx=0x7f05c0005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.165+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f05c80b90d0 con 0x7f05d81038a0 2024-03-24T06:48:51.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.424+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05d810be10 con 0x7f05d81038a0 2024-03-24T06:48:51.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.425+0000 7f05cfe00640 1 -- 172.21.15.22:0/69786222 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f05c80bc790 con 0x7f05d81038a0 2024-03-24T06:48:51.425 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:48:51.425 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:48:51.426 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:48:51.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.428+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 msgr2=0x7f05bc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.428+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 0x7f05bc0a4770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05d8101990 tx=0x7f05c0005f90 comp rx=0 tx=0).stop 2024-03-24T06:48:51.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.428+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 msgr2=0x7f05d8199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.428+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f05c8036040 tx=0x7f05c802c630 comp rx=0 tx=0).stop 2024-03-24T06:48:51.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 shutdown_connections 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05bc0a22b0 0x7f05bc0a4770 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d81098b0 0x7f05d819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05d8104260 0x7f05d819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 --2- 172.21.15.22:0/69786222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05d81038a0 0x7f05d8199f30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 >> 172.21.15.22:0/69786222 conn(0x7f05d80ff370 msgr2=0x7f05d80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 shutdown_connections 2024-03-24T06:48:51.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.429+0000 7f05dd800640 1 -- 172.21.15.22:0/69786222 wait complete. 2024-03-24T06:48:51.454 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:48:51.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 -- 172.21.15.22:0/715423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc102840 msgr2=0x7ffbbc102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/715423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc102840 0x7ffbbc102c20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ffbac009900 tx=0x7ffbac02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 -- 172.21.15.22:0/715423 shutdown_connections 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/715423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc10bc10 0x7ffbbc10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/715423 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 0x7ffbbc10b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/715423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc102840 0x7ffbbc102c20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 -- 172.21.15.22:0/715423 >> 172.21.15.22:0/715423 conn(0x7ffbbc0fc020 msgr2=0x7ffbbc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 -- 172.21.15.22:0/715423 shutdown_connections 2024-03-24T06:48:51.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.678+0000 7ffbbbe00640 1 -- 172.21.15.22:0/715423 wait complete. 2024-03-24T06:48:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.679+0000 7ffbbbe00640 1 Processor -- start 2024-03-24T06:48:51.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.679+0000 7ffbbbe00640 1 -- start start 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 0x7ffbbc1009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc10bc10 0x7ffbbc100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1a6020 con 0x7ffbbc103160 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1a6190 con 0x7ffbbc10bc10 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbbc1a6350 con 0x7ffbbc102840 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37332/0 (socket says 172.21.15.22:37332) 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 -- 172.21.15.22:0/1344266441 learned_addr learned my addr 172.21.15.22:0/1344266441 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbba000640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 0x7ffbbc1009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbb400640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc10bc10 0x7ffbbc100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 -- 172.21.15.22:0/1344266441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc10bc10 msgr2=0x7ffbbc100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc10bc10 0x7ffbbc100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 -- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 msgr2=0x7ffbbc1009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 0x7ffbbc1009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.680+0000 7ffbbaa00640 1 -- 172.21.15.22:0/1344266441 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac009560 con 0x7ffbbc102840 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbbaa00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ffbac000c00 tx=0x7ffbac02c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbb3400640 1 -- 172.21.15.22:0/1344266441 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac02dc60 con 0x7ffbbc102840 2024-03-24T06:48:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbb3400640 1 -- 172.21.15.22:0/1344266441 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbac02ddc0 con 0x7ffbbc102840 2024-03-24T06:48:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbb3400640 1 -- 172.21.15.22:0/1344266441 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac003bf0 con 0x7ffbbc102840 2024-03-24T06:48:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbbc1a65d0 con 0x7ffbbc102840 2024-03-24T06:48:51.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.681+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbbc1a6b30 con 0x7ffbbc102840 2024-03-24T06:48:51.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.684+0000 7ffbb3400640 1 -- 172.21.15.22:0/1344266441 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7ffbac003d50 con 0x7ffbbc102840 2024-03-24T06:48:51.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.684+0000 7ffbc4449740 1 Processor -- start 2024-03-24T06:48:51.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 -- start start 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861d70 0x55e2ca86a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca86a950 0x55e2ca86cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca7cf6e0 con 0x55e2ca861d70 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca7ccc10 con 0x55e2ca86a950 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbc4449740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca7c83f0 con 0x55e2ca861450 2024-03-24T06:48:51.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbbaa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861d70 0x55e2ca86a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37348/0 (socket says 172.21.15.22:37348) 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbbb400640 1 -- 172.21.15.22:0/3250906742 learned_addr learned my addr 172.21.15.22:0/3250906742 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.685+0000 7ffbba000640 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca86a950 0x55e2ca86cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 -- 172.21.15.22:0/3250906742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca86a950 msgr2=0x55e2ca86cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca86a950 0x55e2ca86cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 -- 172.21.15.22:0/3250906742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861d70 msgr2=0x55e2ca86a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861d70 0x55e2ca86a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 -- 172.21.15.22:0/3250906742 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2ca789e40 con 0x55e2ca861450 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbb3400640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 0x7ffba00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbb3400640 1 -- 172.21.15.22:0/1344266441 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ffbac0f0940 con 0x7ffbbc102840 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbaa00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 0x7ffba00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.686+0000 7ffbbb400640 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009f50 tx=0x7ffbb40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a77d19b0ee51b928 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffba7e00640 1 -- 172.21.15.22:0/3250906742 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4004d20 con 0x55e2ca861450 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffba7e00640 1 -- 172.21.15.22:0/3250906742 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbb4004e80 con 0x55e2ca861450 2024-03-24T06:48:51.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffba7e00640 1 -- 172.21.15.22:0/3250906742 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4005280 con 0x55e2ca861450 2024-03-24T06:48:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 -- 172.21.15.22:0/3250906742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 msgr2=0x55e2ca861830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbbaa00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 0x7ffba00a4770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e2ca860f20 tx=0x7ffbac0f6530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009f50 tx=0x7ffbb40118d0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 -- 172.21.15.22:0/3250906742 shutdown_connections 2024-03-24T06:48:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca86a950 0x55e2ca86cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861d70 0x55e2ca86a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.687+0000 7ffbc4449740 1 --2- 172.21.15.22:0/3250906742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca861830 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.688+0000 7ffbc4449740 1 -- 172.21.15.22:0/3250906742 >> 172.21.15.22:0/3250906742 conn(0x55e2ca85bb80 msgr2=0x55e2ca85bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.688+0000 7ffbc4449740 1 -- 172.21.15.22:0/3250906742 shutdown_connections 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.688+0000 7ffbc4449740 1 -- 172.21.15.22:0/3250906742 wait complete. 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.688+0000 7ffbc4449740 1 -- start start 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.688+0000 7ffbc4449740 1 Processor -- start 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 0x55e2ca867a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 0x55e2ca8684b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca76d060 con 0x55e2ca861450 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca76cd00 con 0x55e2ca861d70 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca5361c0 con 0x55e2ca86a950 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbc4449740 10 client.0 _ll_register_callbacks cb 0x55e2ca4d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 0x55e2ca867a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbba000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 0x55e2ca867a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59950/0 (socket says 172.21.15.22:59950) 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbba000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59006/0 (socket says 172.21.15.22:59006) 2024-03-24T06:48:51.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbbb400640 1 -- 172.21.15.22:0/1414896527 learned_addr learned my addr 172.21.15.22:0/1414896527 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.689+0000 7ffbbb400640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 0x55e2ca8684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 -- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 msgr2=0x55e2ca8684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 0x55e2ca8684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 -- 172.21.15.22:0/1414896527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 msgr2=0x55e2ca867a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 0x55e2ca867a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 -- 172.21.15.22:0/1414896527 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb4009c00 con 0x55e2ca861d70 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbbb400640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 0x55e2ca8684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbba000640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc101f20 tx=0x7ffba800bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a172713b3b089ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8016070 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2ca504720 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ffba8003dc0 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ffba8005020 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbc4449740 10 client.27416 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.690+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e2ca7c6a70 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e2ca504720 con 0x55e2ca861d70 2024-03-24T06:48:51.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ffba801b470 con 0x55e2ca861d70 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 10 client.27416 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 10 client.27416 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 20 client.27416 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca87ae50 0x55e2ca87d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 4 client.27416 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.691+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2ca553890 con 0x55e2ca87ae50 2024-03-24T06:48:51.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.692+0000 7ffbbb400640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca87ae50 0x55e2ca87d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.692+0000 7ffbbb400640 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca87ae50 0x55e2ca87d270 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c8a8d28c64a3ec9 server_cookie=e48c77a371d6fc55 in_seq=0 out_seq=0 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.692+0000 7ffba4c00640 10 client.27416 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffba4c00640 1 -- 172.21.15.22:0/1414896527 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2ca553890 con 0x55e2ca87ae50 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffba4c00640 10 client.27416 handle_command_reply: tid=0 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 client.27416 shutdown 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 msgr2=0x55e2ca867f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc101f20 tx=0x7ffba800bf30 comp rx=0 tx=0).stop 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 shutdown_connections 2024-03-24T06:48:51.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca87ae50 0x55e2ca87d270 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca86a950 0x55e2ca8684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca867f70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffba4c00640 0 client.27416 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 --2- 172.21.15.22:0/1414896527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca861450 0x55e2ca867a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 >> 172.21.15.22:0/1414896527 conn(0x55e2ca85bb80 msgr2=0x55e2ca85d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.693+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 shutdown_connections 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.694+0000 7ffbc4449740 1 -- 172.21.15.22:0/1414896527 wait complete. 2024-03-24T06:48:51.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.694+0000 7ffbc4449740 20 client.27416 trim_cache size 0 max 16384 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.717+0000 7ffbc4449740 1 Processor -- start 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.717+0000 7ffbc4449740 1 -- start start 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.717+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca8761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca882b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbc4449740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca8574e0 con 0x55e2ca86a950 2024-03-24T06:48:51.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbc4449740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca877940 con 0x55e2ca861d70 2024-03-24T06:48:51.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbc4449740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca85cb80 con 0x55e2ca861450 2024-03-24T06:48:51.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca8761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca882b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca8761e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37358/0 (socket says 172.21.15.22:37358) 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbbb400640 1 -- 172.21.15.22:0/2543677293 learned_addr learned my addr 172.21.15.22:0/2543677293 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbbb400640 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 -- 172.21.15.22:0/2543677293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 msgr2=0x55e2ca8761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca8761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 -- 172.21.15.22:0/2543677293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 msgr2=0x55e2ca8679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.718+0000 7ffbba000640 1 -- 172.21.15.22:0/2543677293 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2ca762b00 con 0x55e2ca861d70 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.719+0000 7ffbba000640 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca882b80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffba801b9f0 tx=0x7ffba80043a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4abcd4e43b888fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.719+0000 7ffba7400640 1 -- 172.21.15.22:0/2543677293 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8016070 con 0x55e2ca861d70 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.719+0000 7ffba7400640 1 -- 172.21.15.22:0/2543677293 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba8017040 con 0x55e2ca861d70 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.719+0000 7ffba7400640 1 -- 172.21.15.22:0/2543677293 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8011920 con 0x55e2ca861d70 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- 172.21.15.22:0/2543677293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 msgr2=0x55e2ca882b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca882b80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffba801b9f0 tx=0x7ffba80043a0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.722 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- 172.21.15.22:0/2543677293 shutdown_connections 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861d70 0x55e2ca882b80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 --2- 172.21.15.22:0/2543677293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861450 0x55e2ca8761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- 172.21.15.22:0/2543677293 >> 172.21.15.22:0/2543677293 conn(0x55e2ca85bb80 msgr2=0x55e2ca862d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- 172.21.15.22:0/2543677293 shutdown_connections 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- 172.21.15.22:0/2543677293 wait complete. 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 -- start start 2024-03-24T06:48:51.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.720+0000 7ffbc4449740 1 Processor -- start 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8e01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca86f520 con 0x55e2ca86a950 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca8e0720 con 0x55e2ca861450 2024-03-24T06:48:51.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e2ca8e08e0 con 0x55e2ca861d70 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbc4449740 10 client.0 _ll_register_callbacks cb 0x7ffba8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbbb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59016/0 (socket says 172.21.15.22:59016) 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37366/0 (socket says 172.21.15.22:37366) 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbbb400640 1 -- 172.21.15.22:0/961578288 learned_addr learned my addr 172.21.15.22:0/961578288 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.721+0000 7ffbbb400640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8e01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 -- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 msgr2=0x55e2ca86ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 -- 172.21.15.22:0/961578288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 msgr2=0x55e2ca8e01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8e01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 -- 172.21.15.22:0/961578288 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2ca7c6a70 con 0x55e2ca861450 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbba000640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbbb400640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4000f80 tx=0x7ffbb40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b22f2051a1a911 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb402e070 con 0x55e2ca861450 2024-03-24T06:48:51.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7ffbb4003970 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e2ca4c77e0 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ffbb4003b30 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbc4449740 10 client.27422 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.722+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e2ca7886a0 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e2ca4c77e0 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ffbb4038040 con 0x55e2ca861450 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 10 client.27422 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:51.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 10 client.27422 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 20 client.27422 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca8e20d0 0x55e2ca8e44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 4 client.27422 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2ca473700 con 0x55e2ca8e20d0 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.723+0000 7ffbba000640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca8e20d0 0x55e2ca8e44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbba000640 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca8e20d0 0x55e2ca8e44c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=59ea1343c586c07c server_cookie=f1f7496cfc11cad7 in_seq=0 out_seq=0 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffba4c00640 10 client.27422 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffba4c00640 1 -- 172.21.15.22:0/961578288 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2ca473700 con 0x55e2ca8e20d0 2024-03-24T06:48:51.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffba4c00640 10 client.27422 handle_command_reply: tid=0 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 client.27422 shutdown 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 msgr2=0x55e2ca86e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4000f80 tx=0x7ffbb40118d0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 shutdown_connections 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55e2ca8e20d0 0x55e2ca8e44c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e2ca86a950 0x55e2ca8e01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e2ca861d70 0x55e2ca86ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffba4c00640 0 client.27422 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:48:51.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 --2- 172.21.15.22:0/961578288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2ca861450 0x55e2ca86e840 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 >> 172.21.15.22:0/961578288 conn(0x55e2ca85bb80 msgr2=0x55e2ca8679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 shutdown_connections 2024-03-24T06:48:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.724+0000 7ffbc4449740 1 -- 172.21.15.22:0/961578288 wait complete. 2024-03-24T06:48:51.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbc4449740 20 client.27422 trim_cache size 0 max 16384 2024-03-24T06:48:51.733 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:51.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 msgr2=0x7ffba00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 0x7ffba00a4770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e2ca860f20 tx=0x7ffbac0f6530 comp rx=0 tx=0).stop 2024-03-24T06:48:51.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 msgr2=0x7ffbbc100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ffbac000c00 tx=0x7ffbac02c740 comp rx=0 tx=0).stop 2024-03-24T06:48:51.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 shutdown_connections 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ffba00a22b0 0x7ffba00a4770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbbc10bc10 0x7ffbbc100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffbbc103160 0x7ffbbc1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 --2- 172.21.15.22:0/1344266441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbbc102840 0x7ffbbc100490 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 >> 172.21.15.22:0/1344266441 conn(0x7ffbbc0fc020 msgr2=0x7ffbbc0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 shutdown_connections 2024-03-24T06:48:51.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.725+0000 7ffbbbe00640 1 -- 172.21.15.22:0/1344266441 wait complete. 2024-03-24T06:48:51.743 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:48:51.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 -- 172.21.15.22:0/1208978305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101590 msgr2=0x7f93d0101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1208978305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101590 0x7f93d0101a10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009930 tx=0x7f93bc02bc30 comp rx=0 tx=0).stop 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 -- 172.21.15.22:0/1208978305 shutdown_connections 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1208978305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101f50 0x7f93d010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1208978305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101590 0x7f93d0101a10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1208978305 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 0x7f93d0107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 -- 172.21.15.22:0/1208978305 >> 172.21.15.22:0/1208978305 conn(0x7f93d00fd080 msgr2=0x7f93d00ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 -- 172.21.15.22:0/1208978305 shutdown_connections 2024-03-24T06:48:51.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.960+0000 7f93d6e00640 1 -- 172.21.15.22:0/1208978305 wait complete. 2024-03-24T06:48:51.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.961+0000 7f93d6e00640 1 Processor -- start 2024-03-24T06:48:51.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.961+0000 7f93d6e00640 1 -- start start 2024-03-24T06:48:51.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.961+0000 7f93d6e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.961+0000 7f93d6e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 0x7f93d0073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d6e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 0x7f93d0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d6e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f93d01a5f70 con 0x7f93d0107650 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d6e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f93d01a60e0 con 0x7f93d0101f50 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d6e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f93d01a62a0 con 0x7f93d0101590 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d6400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 0x7f93d0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37384/0 (socket says 172.21.15.22:37384) 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 -- 172.21.15.22:0/1832793474 learned_addr learned my addr 172.21.15.22:0/1832793474 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5000640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 0x7f93d0073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 -- 172.21.15.22:0/1832793474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 msgr2=0x7f93d0073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 0x7f93d0073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 -- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 msgr2=0x7f93d0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 0x7f93d0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5a00640 1 -- 172.21.15.22:0/1832793474 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93bc009560 con 0x7f93d0101590 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.962+0000 7f93d5000640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 0x7f93d0073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93d5a00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93c400d950 tx=0x7f93c400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93ce000640 1 -- 172.21.15.22:0/1832793474 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4009c30 con 0x7f93d0101590 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93ce000640 1 -- 172.21.15.22:0/1832793474 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c4009d90 con 0x7f93d0101590 2024-03-24T06:48:51.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93ce000640 1 -- 172.21.15.22:0/1832793474 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c400f7e0 con 0x7f93d0101590 2024-03-24T06:48:51.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d01a6580 con 0x7f93d0101590 2024-03-24T06:48:51.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.963+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d01a6a10 con 0x7f93d0101590 2024-03-24T06:48:51.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 Processor -- start 2024-03-24T06:48:51.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 -- start start 2024-03-24T06:48:51.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1cad70 0x55e54f1d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1d3950 0x55e54f1d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e54f1386e0 con 0x55e54f1cad70 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e54f135c10 con 0x55e54f1d3950 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.968+0000 7f93db03d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e54f1313f0 con 0x55e54f1ca450 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37390/0 (socket says 172.21.15.22:37390) 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 -- 172.21.15.22:0/386305567 learned_addr learned my addr 172.21.15.22:0/386305567 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93ce000640 1 -- 172.21.15.22:0/1832793474 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f93c4010ad0 con 0x7f93d0101590 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d5000640 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1d3950 0x55e54f1d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 -- 172.21.15.22:0/386305567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1d3950 msgr2=0x55e54f1d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1d3950 0x55e54f1d5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 -- 172.21.15.22:0/386305567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1cad70 msgr2=0x55e54f1d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1cad70 0x55e54f1d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 -- 172.21.15.22:0/386305567 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e54f0f2e40 con 0x55e54f1ca450 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.969+0000 7f93d6400640 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009fd0 tx=0x7f93c80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baeab659a32b90f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93c2a00640 1 -- 172.21.15.22:0/386305567 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8004d20 con 0x55e54f1ca450 2024-03-24T06:48:51.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93c2a00640 1 -- 172.21.15.22:0/386305567 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c8004e80 con 0x55e54f1ca450 2024-03-24T06:48:51.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93c2a00640 1 -- 172.21.15.22:0/386305567 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8005280 con 0x55e54f1ca450 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 -- 172.21.15.22:0/386305567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 msgr2=0x55e54f1ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009fd0 tx=0x7f93c80118d0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93ce000640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 0x7f93b80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 -- 172.21.15.22:0/386305567 shutdown_connections 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1d3950 0x55e54f1d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1cad70 0x55e54f1d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 --2- 172.21.15.22:0/386305567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1ca450 0x55e54f1ca830 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93db03d740 1 -- 172.21.15.22:0/386305567 >> 172.21.15.22:0/386305567 conn(0x55e54f1c4b80 msgr2=0x55e54f1c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.970+0000 7f93ce000640 1 -- 172.21.15.22:0/1832793474 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f93c40cdf80 con 0x7f93d0101590 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93d5a00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 0x7f93b80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93db03d740 1 -- 172.21.15.22:0/386305567 shutdown_connections 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93db03d740 1 -- 172.21.15.22:0/386305567 wait complete. 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93db03d740 1 -- start start 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93d5a00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 0x7f93b80a4770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55e54f1c9f20 tx=0x7f93c4005880 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.971+0000 7f93db03d740 1 Processor -- start 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1ca450 0x55e54f1d0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 0x55e54f1d1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e54f0d6060 con 0x55e54f1d3950 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e54f0d5d00 con 0x55e54f1ca450 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e54ee9f1c0 con 0x55e54f1cad70 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93db03d740 10 client.0 _ll_register_callbacks cb 0x55e54ee41be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93d5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93d6400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1ca450 0x55e54f1d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93d5000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37402/0 (socket says 172.21.15.22:37402) 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93d5000640 1 -- 172.21.15.22:0/1948968879 learned_addr learned my addr 172.21.15.22:0/1948968879 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.972+0000 7f93d6400640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 0x55e54f1d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 -- 172.21.15.22:0/1948968879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1ca450 msgr2=0x55e54f1d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1ca450 0x55e54f1d0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 -- 172.21.15.22:0/1948968879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 msgr2=0x55e54f1d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 0x55e54f1d1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 -- 172.21.15.22:0/1948968879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c8009c80 con 0x55e54f1cad70 2024-03-24T06:48:51.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d6400640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 0x55e54f1d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93d5000640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f93bc035000 tx=0x7f93bc02c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c311917260a60fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93b7400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.973+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc004410 con 0x55e54f1cad70 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f93bc046070 con 0x55e54f1cad70 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e54ee6d720 con 0x55e54f1cad70 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f93bc004df0 con 0x55e54f1cad70 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 10 client.27463 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e54f12fa70 con 0x55e54f1cad70 2024-03-24T06:48:51.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e54ee6d720 con 0x55e54f1cad70 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f93bc03c040 con 0x55e54f1cad70 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 10 client.27463 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 10 client.27463 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 20 client.27463 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f1e6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.974+0000 7f93db03d740 4 client.27463 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:51.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.975+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e54eebc890 con 0x55e54f1e3de0 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.975+0000 7f93d6400640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f1e6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.975+0000 7f93d6400640 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f1e6200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d8600acd314db657 server_cookie=a4e74a8205ffcc9 in_seq=0 out_seq=0 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.975+0000 7f93b7400640 10 client.27463 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.976+0000 7f93b7400640 1 -- 172.21.15.22:0/1948968879 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e54eebc890 con 0x55e54f1e3de0 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.976+0000 7f93b7400640 10 client.27463 handle_command_reply: tid=0 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.976+0000 7f93db03d740 1 client.27463 shutdown 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 msgr2=0x55e54f1d0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f93bc035000 tx=0x7f93bc02c680 comp rx=0 tx=0).stop 2024-03-24T06:48:51.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 shutdown_connections 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f1e6200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1d3950 0x55e54f1d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1cad70 0x55e54f1d0ec0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93b7400640 0 client.27463 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 --2- 172.21.15.22:0/1948968879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1ca450 0x55e54f1d0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 >> 172.21.15.22:0/1948968879 conn(0x55e54f1c4b80 msgr2=0x55e54f1c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 shutdown_connections 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 1 -- 172.21.15.22:0/1948968879 wait complete. 2024-03-24T06:48:51.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.977+0000 7f93db03d740 20 client.27463 trim_cache size 0 max 16384 2024-03-24T06:48:51.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.992+0000 7f93db03d740 1 Processor -- start 2024-03-24T06:48:51.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 -- start start 2024-03-24T06:48:51.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1d30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.994 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1bfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e54f19a190 con 0x55e54f1ca450 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e54f1d1950 con 0x55e54f1cad70 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93db03d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e54f1c04b0 con 0x55e54f1d3950 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93d5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93d5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59056/0 (socket says 172.21.15.22:59056) 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93d6400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1d30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93d5000640 1 -- 172.21.15.22:0/2547802994 learned_addr learned my addr 172.21.15.22:0/2547802994 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.993+0000 7f93d6400640 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1bfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 -- 172.21.15.22:0/2547802994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 msgr2=0x55e54f1bfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1bfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 -- 172.21.15.22:0/2547802994 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 msgr2=0x55e54f1d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1d30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 -- 172.21.15.22:0/2547802994 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e54f0cbb00 con 0x55e54f1cad70 2024-03-24T06:48:51.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d6400640 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1bfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:51.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93d5000640 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f10d6d168cb7d566 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93c2000640 1 -- 172.21.15.22:0/2547802994 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc02cca0 con 0x55e54f1cad70 2024-03-24T06:48:52.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93c2000640 1 -- 172.21.15.22:0/2547802994 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93bc044070 con 0x55e54f1cad70 2024-03-24T06:48:52.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.994+0000 7f93c2000640 1 -- 172.21.15.22:0/2547802994 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc005290 con 0x55e54f1cad70 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- 172.21.15.22:0/2547802994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 msgr2=0x55e54f1df1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc005b50 comp rx=0 tx=0).stop 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- 172.21.15.22:0/2547802994 shutdown_connections 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1bfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1df1e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 --2- 172.21.15.22:0/2547802994 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1d30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- 172.21.15.22:0/2547802994 >> 172.21.15.22:0/2547802994 conn(0x55e54f1c4b80 msgr2=0x55e54f1cf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- 172.21.15.22:0/2547802994 shutdown_connections 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- 172.21.15.22:0/2547802994 wait complete. 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.995+0000 7f93db03d740 1 -- start start 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 Processor -- start 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e54f249100 con 0x55e54f1ca450 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e54f249240 con 0x55e54f1cad70 2024-03-24T06:48:52.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e54f249380 con 0x55e54f1d3950 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93db03d740 10 client.0 _ll_register_callbacks cb 0x7f93bc02c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93d6400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93d5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93d6400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60036/0 (socket says 172.21.15.22:60036) 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93d5000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59062/0 (socket says 172.21.15.22:59062) 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.996+0000 7f93d6400640 1 -- 172.21.15.22:0/1845078797 learned_addr learned my addr 172.21.15.22:0/1845078797 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d6400640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 -- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 msgr2=0x55e54f1cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 -- 172.21.15.22:0/1845078797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 msgr2=0x55e54f1cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 -- 172.21.15.22:0/1845078797 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e54f12fa70 con 0x55e54f1cad70 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93d5000640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x55e54f1c52a0 tx=0x7f93bc005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d6ee60821da704 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93b7400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.997+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc003c30 con 0x55e54f1cad70 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f93bc046040 con 0x55e54f1cad70 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e54f1cc3f0 con 0x55e54f1cad70 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f93bc003d90 con 0x55e54f1cad70 2024-03-24T06:48:52.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 10 client.27440 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e54f0f16a0 con 0x55e54f1cad70 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e54f1cc3f0 con 0x55e54f1cad70 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f93bc05f070 con 0x55e54f1cad70 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 10 client.27440 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 10 client.27440 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 20 client.27440 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f24c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 4 client.27440 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.998+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e54ee307e0 con 0x55e54f1e3de0 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93d6400640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f24c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93d6400640 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f24c6a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9054a662899edf90 server_cookie=f1f3a67d9ea6b3a in_seq=0 out_seq=0 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93b7400640 10 client.27440 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93b7400640 1 -- 172.21.15.22:0/1845078797 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e54ee307e0 con 0x55e54f1e3de0 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93b7400640 10 client.27440 handle_command_reply: tid=0 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93db03d740 1 client.27440 shutdown 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 msgr2=0x55e54f1cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:51.999+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x55e54f1c52a0 tx=0x7f93bc005bc0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 shutdown_connections 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55e54f1e3de0 0x55e54f24c6a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e54f1d3950 0x55e54f1cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e54f1cad70 0x55e54f1cd9b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 --2- 172.21.15.22:0/1845078797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e54f1ca450 0x55e54f1cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 >> 172.21.15.22:0/1845078797 conn(0x55e54f1c4b80 msgr2=0x55e54f1cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93b7400640 0 client.27440 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 shutdown_connections 2024-03-24T06:48:52.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 1 -- 172.21.15.22:0/1845078797 wait complete. 2024-03-24T06:48:52.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.000+0000 7f93db03d740 20 client.27440 trim_cache size 0 max 16384 2024-03-24T06:48:52.009 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 msgr2=0x7f93b80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 0x7f93b80a4770 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55e54f1c9f20 tx=0x7f93c4005880 comp rx=0 tx=0).stop 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 msgr2=0x7f93d0073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93c400d950 tx=0x7f93c400de10 comp rx=0 tx=0).stop 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 shutdown_connections 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f93b80a22b0 0x7f93b80a4770 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f93d0107650 0x7f93d0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93d0101f50 0x7f93d0073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 --2- 172.21.15.22:0/1832793474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93d0101590 0x7f93d0073a10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 >> 172.21.15.22:0/1832793474 conn(0x7f93d00fd080 msgr2=0x7f93d00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 shutdown_connections 2024-03-24T06:48:52.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.001+0000 7f93d6e00640 1 -- 172.21.15.22:0/1832793474 wait complete. 2024-03-24T06:48:52.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:48:52.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 -- 172.21.15.22:0/198190448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 msgr2=0x7fd0bc107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 --2- 172.21.15.22:0/198190448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc107a40 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009930 tx=0x7fd0b002bc30 comp rx=0 tx=0).stop 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 -- 172.21.15.22:0/198190448 shutdown_connections 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 --2- 172.21.15.22:0/198190448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 --2- 172.21.15.22:0/198190448 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 0x7fd0bc101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 --2- 172.21.15.22:0/198190448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc107a40 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 -- 172.21.15.22:0/198190448 >> 172.21.15.22:0/198190448 conn(0x7fd0bc0fd0b0 msgr2=0x7fd0bc0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 -- 172.21.15.22:0/198190448 shutdown_connections 2024-03-24T06:48:52.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.239+0000 7fd0c2200640 1 -- 172.21.15.22:0/198190448 wait complete. 2024-03-24T06:48:52.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.240+0000 7fd0c2200640 1 Processor -- start 2024-03-24T06:48:52.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 -- start start 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 0x7fd0bc199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd0bc19ec50 con 0x7fd0bc1015a0 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd0bc19edc0 con 0x7fd0bc107660 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c2200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0bc19ef80 con 0x7fd0bc101f60 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.241+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 0x7fd0bc199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37440/0 (socket says 172.21.15.22:37440) 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 -- 172.21.15.22:0/3304958798 learned_addr learned my addr 172.21.15.22:0/3304958798 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0c1800640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 -- 172.21.15.22:0/3304958798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 msgr2=0x7fd0bc19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 -- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 msgr2=0x7fd0bc199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 0x7fd0bc199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 -- 172.21.15.22:0/3304958798 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b0009560 con 0x7fd0bc101f60 2024-03-24T06:48:52.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.242+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800d950 tx=0x7fd0a800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.243+0000 7fd0b9600640 1 -- 172.21.15.22:0/3304958798 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8009c30 con 0x7fd0bc101f60 2024-03-24T06:48:52.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.243+0000 7fd0b9600640 1 -- 172.21.15.22:0/3304958798 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a8009d90 con 0x7fd0bc101f60 2024-03-24T06:48:52.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.243+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0bc19f260 con 0x7fd0bc101f60 2024-03-24T06:48:52.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.243+0000 7fd0b9600640 1 -- 172.21.15.22:0/3304958798 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800f7e0 con 0x7fd0bc101f60 2024-03-24T06:48:52.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.243+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0bc1a6ae0 con 0x7fd0bc101f60 2024-03-24T06:48:52.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.246+0000 7fd0c65c6740 1 Processor -- start 2024-03-24T06:48:52.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.246+0000 7fd0c65c6740 1 -- start start 2024-03-24T06:48:52.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.246+0000 7fd0b9600640 1 -- 172.21.15.22:0/3304958798 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd0a8010ad0 con 0x7fd0bc101f60 2024-03-24T06:48:52.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.246+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b8544410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8546d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d7b84a96e0 con 0x55d7b8544950 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7b84a6c10 con 0x55d7b853b450 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d7b84a23f0 con 0x55d7b853bd70 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59080/0 (socket says 172.21.15.22:59080) 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b8544410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0bbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8546d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 -- 172.21.15.22:0/2845719510 learned_addr learned my addr 172.21.15.22:0/2845719510 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 -- 172.21.15.22:0/2845719510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 msgr2=0x55d7b8544410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b8544410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 -- 172.21.15.22:0/2845719510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 msgr2=0x55d7b8546d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8546d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.247+0000 7fd0c1800640 1 -- 172.21.15.22:0/2845719510 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7b8463e40 con 0x55d7b853b450 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0c1800640 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009fd0 tx=0x7fd0b40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c96dcbf9693857c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0b9600640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 0x7fd0a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.250 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0ae000640 1 -- 172.21.15.22:0/2845719510 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4004d20 con 0x55d7b853b450 2024-03-24T06:48:52.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0ae000640 1 -- 172.21.15.22:0/2845719510 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0b4004e80 con 0x55d7b853b450 2024-03-24T06:48:52.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0ae000640 1 -- 172.21.15.22:0/2845719510 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4005280 con 0x55d7b853b450 2024-03-24T06:48:52.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0b9600640 1 -- 172.21.15.22:0/3304958798 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fd0a80ce060 con 0x7fd0bc101f60 2024-03-24T06:48:52.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.248+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 0x7fd0a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2845719510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 msgr2=0x55d7b853b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009fd0 tx=0x7fd0b40118d0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2845719510 shutdown_connections 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8546d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b8544410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2845719510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b853b830 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2845719510 >> 172.21.15.22:0/2845719510 conn(0x55d7b8535b80 msgr2=0x55d7b8535f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2845719510 shutdown_connections 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2845719510 wait complete. 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.249+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 0x7fd0a40a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009900 tx=0x7fd0b0005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 -- start start 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 Processor -- start 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b8541f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 0x55d7b8542470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d7b8447060 con 0x55d7b853bd70 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7b8446d00 con 0x55d7b853b450 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d7b82101c0 con 0x55d7b8544950 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.250+0000 7fd0c65c6740 10 client.0 _ll_register_callbacks cb 0x55d7b81b2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 0x55d7b8542470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59090/0 (socket says 172.21.15.22:59090) 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c0e00640 1 -- 172.21.15.22:0/301843710 learned_addr learned my addr 172.21.15.22:0/301843710 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b8541f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 -- 172.21.15.22:0/301843710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 msgr2=0x55d7b85419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 -- 172.21.15.22:0/301843710 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 msgr2=0x55d7b8541f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b8541f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 -- 172.21.15.22:0/301843710 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b4009c80 con 0x55d7b8544950 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0c1800640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 0x55d7b8542470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4011dd0 tx=0x7fd0b4004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab38528550c0a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.251+0000 7fd0a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4004890 con 0x55d7b8544950 2024-03-24T06:48:52.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fd0b400f4e0 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d7b81de720 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0b40044b0 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0c65c6740 10 client.27481 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7b84a0a70 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.252+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d7b81de720 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd0b4021040 con 0x55d7b8544950 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 10 client.27481 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 10 client.27481 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 20 client.27481 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b8554e50 0x55d7b8557240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 4 client.27481 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7b822d890 con 0x55d7b8554e50 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.253+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b8554e50 0x55d7b8557240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.254+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b8554e50 0x55d7b8557240 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9797427238bc33a3 server_cookie=91e53c1c7a0856ef in_seq=0 out_seq=0 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.254+0000 7fd0a2a00640 10 client.27481 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0a2a00640 1 -- 172.21.15.22:0/301843710 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d7b822d890 con 0x55d7b8554e50 2024-03-24T06:48:52.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0a2a00640 10 client.27481 handle_command_reply: tid=0 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 client.27481 shutdown 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 msgr2=0x55d7b8542470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 0x55d7b8542470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4011dd0 tx=0x7fd0b4004cd0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 shutdown_connections 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b8554e50 0x55d7b8557240 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b8544950 0x55d7b8542470 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b8541f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0a2a00640 0 client.27481 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:52.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/301843710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 >> 172.21.15.22:0/301843710 conn(0x55d7b8535b80 msgr2=0x55d7b85378d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 shutdown_connections 2024-03-24T06:48:52.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.255+0000 7fd0c65c6740 1 -- 172.21.15.22:0/301843710 wait complete. 2024-03-24T06:48:52.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.256+0000 7fd0c65c6740 20 client.27481 trim_cache size 0 max 16384 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 Processor -- start 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 -- start start 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b85501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b8544950 0x55d7b855cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d7b853c830 con 0x55d7b853bd70 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7b8531090 con 0x55d7b8544950 2024-03-24T06:48:52.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d7b85578a0 con 0x55d7b853b450 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.272+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37470/0 (socket says 172.21.15.22:37470) 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 -- 172.21.15.22:0/4132241703 learned_addr learned my addr 172.21.15.22:0/4132241703 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b85501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c1800640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b8544950 0x55d7b855cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 -- 172.21.15.22:0/4132241703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b8544950 msgr2=0x55d7b855cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b8544950 0x55d7b855cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 -- 172.21.15.22:0/4132241703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 msgr2=0x55d7b85501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b85501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 -- 172.21.15.22:0/4132241703 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7b843cb00 con 0x55d7b853b450 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b85501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009230 tx=0x7fd0b0042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70d1b42a6bd18fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0ad600640 1 -- 172.21.15.22:0/4132241703 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0051040 con 0x55d7b853b450 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0ad600640 1 -- 172.21.15.22:0/4132241703 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0b0044070 con 0x55d7b853b450 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.273+0000 7fd0ad600640 1 -- 172.21.15.22:0/4132241703 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00035f0 con 0x55d7b853b450 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 -- 172.21.15.22:0/4132241703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 msgr2=0x55d7b8530b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.317 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009230 tx=0x7fd0b0042000 comp rx=0 tx=0).stop 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 -- 172.21.15.22:0/4132241703 shutdown_connections 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b8544950 0x55d7b855cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b853bd70 0x55d7b85501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/4132241703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853b450 0x55d7b8530b50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 -- 172.21.15.22:0/4132241703 >> 172.21.15.22:0/4132241703 conn(0x55d7b8535b80 msgr2=0x55d7b8534cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 -- 172.21.15.22:0/4132241703 shutdown_connections 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.274+0000 7fd0c65c6740 1 -- 172.21.15.22:0/4132241703 wait complete. 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 -- start start 2024-03-24T06:48:52.318 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 Processor -- start 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b85488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8548e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d7b85ba280 con 0x55d7b8544950 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7b85ba3f0 con 0x55d7b853b450 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d7b85ba5b0 con 0x55d7b853bd70 2024-03-24T06:48:52.319 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.275+0000 7fd0c65c6740 10 client.0 _ll_register_callbacks cb 0x7fd0a80d19a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59118/0 (socket says 172.21.15.22:59118) 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0bbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b85488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 -- 172.21.15.22:0/2777518092 learned_addr learned my addr 172.21.15.22:0/2777518092 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c1800640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8548e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 -- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 msgr2=0x55d7b85488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.320 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b85488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 -- 172.21.15.22:0/2777518092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 msgr2=0x55d7b8548e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8548e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 -- 172.21.15.22:0/2777518092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0bc10ed20 con 0x55d7b853b450 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b85488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0c0e00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0b005b040 tx=0x7fd0b0058700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63e1eaad7a8d47f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.276+0000 7fd0a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0051040 con 0x55d7b853b450 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fd0b0004040 con 0x55d7b853b450 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d7b81a17e0 con 0x55d7b853b450 2024-03-24T06:48:52.321 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd0b0004200 con 0x55d7b853b450 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 10 client.27458 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d7b84626a0 con 0x55d7b853b450 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d7b81a17e0 con 0x55d7b853b450 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fd0b005f070 con 0x55d7b853b450 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 10 client.27458 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 10 client.27458 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 20 client.27458 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.322 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b85bb9b0 0x55d7b85bdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 4 client.27458 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.277+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7b814d700 con 0x55d7b85bb9b0 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b85bb9b0 0x55d7b85bdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0bbe00640 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b85bb9b0 0x55d7b85bdda0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8352affe45f82bab server_cookie=98d2bef384a43530 in_seq=0 out_seq=0 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0a2a00640 10 client.27458 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0a2a00640 1 -- 172.21.15.22:0/2777518092 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7b814d700 con 0x55d7b85bb9b0 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0a2a00640 10 client.27458 handle_command_reply: tid=0 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0c65c6740 1 client.27458 shutdown 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 msgr2=0x55d7b85483a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.323 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.278+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd0b005b040 tx=0x7fd0b0058700 comp rx=0 tx=0).stop 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 shutdown_connections 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55d7b85bb9b0 0x55d7b85bdda0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d7b8544950 0x55d7b8548e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d7b853bd70 0x55d7b85488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0a2a00640 0 client.27458 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 --2- 172.21.15.22:0/2777518092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7b853b450 0x55d7b85483a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 >> 172.21.15.22:0/2777518092 conn(0x55d7b8535b80 msgr2=0x55d7b8536d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 shutdown_connections 2024-03-24T06:48:52.324 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 1 -- 172.21.15.22:0/2777518092 wait complete. 2024-03-24T06:48:52.325 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.279+0000 7fd0c65c6740 20 client.27458 trim_cache size 0 max 16384 2024-03-24T06:48:52.325 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 msgr2=0x7fd0a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 0x7fd0a40a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009900 tx=0x7fd0b0005a20 comp rx=0 tx=0).stop 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 msgr2=0x7fd0bc19a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800d950 tx=0x7fd0a800de10 comp rx=0 tx=0).stop 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 shutdown_connections 2024-03-24T06:48:52.326 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd0a40a2230 0x7fd0a40a46f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0bc107660 0x7fd0bc19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0bc101f60 0x7fd0bc19a2d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 --2- 172.21.15.22:0/3304958798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd0bc1015a0 0x7fd0bc199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 >> 172.21.15.22:0/3304958798 conn(0x7fd0bc0fd0b0 msgr2=0x7fd0bc10d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 shutdown_connections 2024-03-24T06:48:52.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.280+0000 7fd0c2200640 1 -- 172.21.15.22:0/3304958798 wait complete. 2024-03-24T06:48:52.327 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:52 smithi022 ceph-mon[32003]: pgmap v274: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:52.327 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:52 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/69786222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:52.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:48:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:52 smithi143 ceph-mon[36520]: pgmap v274: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:52 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/69786222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:52 smithi155 ceph-mon[36262]: pgmap v274: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:48:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:52 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/69786222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 -- 172.21.15.22:0/4228380510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb8103100 msgr2=0x7fcfb810b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 --2- 172.21.15.22:0/4228380510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb8103100 0x7fcfb810b5e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009900 tx=0x7fcfa002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 -- 172.21.15.22:0/4228380510 shutdown_connections 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 --2- 172.21.15.22:0/4228380510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb810bbb0 0x7fcfb810e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 --2- 172.21.15.22:0/4228380510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb8103100 0x7fcfb810b5e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 --2- 172.21.15.22:0/4228380510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb81027e0 0x7fcfb8102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.558+0000 7fcfb7200640 1 -- 172.21.15.22:0/4228380510 >> 172.21.15.22:0/4228380510 conn(0x7fcfb80fbf60 msgr2=0x7fcfb80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.559+0000 7fcfb7200640 1 -- 172.21.15.22:0/4228380510 shutdown_connections 2024-03-24T06:48:52.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.559+0000 7fcfb7200640 1 -- 172.21.15.22:0/4228380510 wait complete. 2024-03-24T06:48:52.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 Processor -- start 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 -- start start 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb8103100 0x7fcfb819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 0x7fcfb819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb819ec90 con 0x7fcfb810bbb0 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb819ee00 con 0x7fcfb81027e0 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb7200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb819efc0 con 0x7fcfb8103100 2024-03-24T06:48:52.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 0x7fcfb819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59130/0 (socket says 172.21.15.22:59130) 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.560+0000 7fcfb5e00640 1 -- 172.21.15.22:0/714063368 learned_addr learned my addr 172.21.15.22:0/714063368 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfafe00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb8103100 0x7fcfb819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 -- 172.21.15.22:0/714063368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb8103100 msgr2=0x7fcfb819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb8103100 0x7fcfb819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 -- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 msgr2=0x7fcfb819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 0x7fcfb819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 -- 172.21.15.22:0/714063368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa0009560 con 0x7fcfb81027e0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb6800640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 0x7fcfb819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800e9f0 tx=0x7fcfa800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.561+0000 7fcfaea00640 1 -- 172.21.15.22:0/714063368 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8009700 con 0x7fcfb81027e0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.562+0000 7fcfaea00640 1 -- 172.21.15.22:0/714063368 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa800cd80 con 0x7fcfb81027e0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.562+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb819f2a0 con 0x7fcfb81027e0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.562+0000 7fcfaea00640 1 -- 172.21.15.22:0/714063368 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8018920 con 0x7fcfb81027e0 2024-03-24T06:48:52.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.562+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb81a6b20 con 0x7fcfb81027e0 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfaea00640 1 -- 172.21.15.22:0/714063368 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fcfa8018a80 con 0x7fcfb81027e0 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 Processor -- start 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 -- start start 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f26986c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f26986c950 0x55f26986ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.572 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f2697d16e0 con 0x55f26986c950 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2697cec10 con 0x55f269863450 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.565+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2697ca3f0 con 0x55f269863d70 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f26986c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59132/0 (socket says 172.21.15.22:59132) 2024-03-24T06:48:52.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 -- 172.21.15.22:0/2319068340 learned_addr learned my addr 172.21.15.22:0/2319068340 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfafe00640 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f26986c950 0x55f26986ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 -- 172.21.15.22:0/2319068340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 msgr2=0x55f26986c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f26986c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 -- 172.21.15.22:0/2319068340 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f26986c950 msgr2=0x55f26986ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f26986c950 0x55f26986ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb6800640 1 -- 172.21.15.22:0/2319068340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26978be40 con 0x55f269863450 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.566+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f26986c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfb6800640 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0011790 tx=0x7fcfb0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fed066a9ccca82a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfaea00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 0x7fcf9c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfa7400640 1 -- 172.21.15.22:0/2319068340 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0011ea0 con 0x55f269863450 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfaea00640 1 -- 172.21.15.22:0/714063368 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fcfa8014070 con 0x7fcfb81027e0 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfa7400640 1 -- 172.21.15.22:0/2319068340 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb000fe80 con 0x55f269863450 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfa7400640 1 -- 172.21.15.22:0/2319068340 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004800 con 0x55f269863450 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.567+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 0x7fcf9c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- 172.21.15.22:0/2319068340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 msgr2=0x55f269863830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0011790 tx=0x7fcfb0011c50 comp rx=0 tx=0).stop 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- 172.21.15.22:0/2319068340 shutdown_connections 2024-03-24T06:48:52.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f26986c950 0x55f26986ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f26986c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/2319068340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269863830 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- 172.21.15.22:0/2319068340 >> 172.21.15.22:0/2319068340 conn(0x55f26985db80 msgr2=0x55f26985df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfb5e00640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 0x7fcf9c0a4770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55f269862f20 tx=0x7fcfa80d40e0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- 172.21.15.22:0/2319068340 shutdown_connections 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- 172.21.15.22:0/2319068340 wait complete. 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 -- start start 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.568+0000 7fcfbfc01740 1 Processor -- start 2024-03-24T06:48:52.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 0x55f269869ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f26986a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f26976f060 con 0x55f269863d70 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f26976ed00 con 0x55f269863450 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2695381c0 con 0x55f26986c950 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfbfc01740 10 client.0 _ll_register_callbacks cb 0x55f2694dabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59146/0 (socket says 172.21.15.22:59146) 2024-03-24T06:48:52.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfb6800640 1 -- 172.21.15.22:0/4184937990 learned_addr learned my addr 172.21.15.22:0/4184937990 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfafe00640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 0x55f269869ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.569+0000 7fcfb6800640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f26986a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 -- 172.21.15.22:0/4184937990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 msgr2=0x55f26986a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f26986a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 -- 172.21.15.22:0/4184937990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 msgr2=0x55f269869ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 0x55f269869ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 -- 172.21.15.22:0/4184937990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb0011440 con 0x55f269863450 2024-03-24T06:48:52.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfafe00640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 0x55f269869ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfb6800640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0010010 tx=0x7fcfb00043a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e13bb222a78db1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcf9be00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0003e70 con 0x55f269863450 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f2694c97e0 con 0x55f269863450 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.570+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fcfb0003fd0 con 0x55f269863450 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcfb0031810 con 0x55f269863450 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 10 client.27470 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f2697c8a70 con 0x55f269863450 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f2694c97e0 con 0x55f269863450 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcfb001a440 con 0x55f269863450 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 10 client.27470 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 10 client.27470 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 20 client.27470 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f26987cd90 0x55f26987f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 4 client.27470 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:52.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.571+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f269506720 con 0x55f26987cd90 2024-03-24T06:48:52.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.572+0000 7fcfafe00640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f26987cd90 0x55f26987f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.572+0000 7fcfafe00640 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f26987cd90 0x55f26987f1b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a60a80dfdefe569c server_cookie=71e013c294e3fac6 in_seq=0 out_seq=0 2024-03-24T06:48:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.572+0000 7fcf9be00640 10 client.27470 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:52.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcf9be00640 1 -- 172.21.15.22:0/4184937990 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f269506720 con 0x55f26987cd90 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcf9be00640 10 client.27470 handle_command_reply: tid=0 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 client.27470 shutdown 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 msgr2=0x55f269869960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0010010 tx=0x7fcfb00043a0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 shutdown_connections 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f26986a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863d70 0x55f269869ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f26987cd90 0x55f26987f1b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/4184937990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863450 0x55f269869960 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcf9be00640 0 client.27470 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.573+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 >> 172.21.15.22:0/4184937990 conn(0x55f26985db80 msgr2=0x55f26985f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.574+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 shutdown_connections 2024-03-24T06:48:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.574+0000 7fcfbfc01740 1 -- 172.21.15.22:0/4184937990 wait complete. 2024-03-24T06:48:52.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.574+0000 7fcfbfc01740 20 client.27470 trim_cache size 0 max 16384 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.594+0000 7fcfbfc01740 1 Processor -- start 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.594+0000 7fcfbfc01740 1 -- start start 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.594+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269868ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f269884a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f269865580 con 0x55f269863450 2024-03-24T06:48:52.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f26985eb80 con 0x55f269863d70 2024-03-24T06:48:52.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f26985fe90 con 0x55f26986c950 2024-03-24T06:48:52.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59160/0 (socket says 172.21.15.22:59160) 2024-03-24T06:48:52.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 -- 172.21.15.22:0/500193782 learned_addr learned my addr 172.21.15.22:0/500193782 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfb6800640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269868ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.598 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfb6800640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f269884a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 -- 172.21.15.22:0/500193782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 msgr2=0x55f269884a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f269884a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 -- 172.21.15.22:0/500193782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 msgr2=0x55f269868ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269868ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfafe00640 1 -- 172.21.15.22:0/500193782 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f269764b00 con 0x55f269863d70 2024-03-24T06:48:52.599 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.595+0000 7fcfb6800640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269868ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.596+0000 7fcfafe00640 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009900 tx=0x7fcfa002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c322b06928501200 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.596+0000 7fcfa6a00640 1 -- 172.21.15.22:0/500193782 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0046070 con 0x55f269863d70 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.596+0000 7fcfa6a00640 1 -- 172.21.15.22:0/500193782 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa004c070 con 0x55f269863d70 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.596+0000 7fcfa6a00640 1 -- 172.21.15.22:0/500193782 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0048690 con 0x55f269863d70 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- 172.21.15.22:0/500193782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 msgr2=0x55f2698781e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009900 tx=0x7fcfa002e040 comp rx=0 tx=0).stop 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- 172.21.15.22:0/500193782 shutdown_connections 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f26986c950 0x55f269884a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f269863d70 0x55f2698781e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/500193782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269868ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- 172.21.15.22:0/500193782 >> 172.21.15.22:0/500193782 conn(0x55f26985db80 msgr2=0x55f26985c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- 172.21.15.22:0/500193782 shutdown_connections 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- 172.21.15.22:0/500193782 wait complete. 2024-03-24T06:48:52.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.597+0000 7fcfbfc01740 1 -- start start 2024-03-24T06:48:52.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 Processor -- start 2024-03-24T06:48:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269874810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 0x55f2698661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f269874e20 con 0x55f269863450 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f269874f90 con 0x55f26986c950 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.598+0000 7fcfbfc01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f2698666f0 con 0x55f269863d70 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfbfc01740 10 client.0 _ll_register_callbacks cb 0x7fcfb00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfb6800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269874810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37514/0 (socket says 172.21.15.22:37514) 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 -- 172.21.15.22:0/3213502408 learned_addr learned my addr 172.21.15.22:0/3213502408 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfb6800640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 0x55f2698661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 -- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 msgr2=0x55f2698661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 0x55f2698661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 -- 172.21.15.22:0/3213502408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 msgr2=0x55f269874810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269874810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfafe00640 1 -- 172.21.15.22:0/3213502408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb810ecd0 con 0x55f269863d70 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.599+0000 7fcfb6800640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 0x55f2698661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcfafe00640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcfa002c210 tx=0x7fcfa00042a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12a58bb01a55c5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0046070 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb000fe80 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7fcfa002c880 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fcfa002ca40 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcfbfc01740 10 client.27502 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f26978a6a0 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb000fe80 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.600+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fcfa003c040 con 0x55f269863d70 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 10 client.27502 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 10 client.27502 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 20 client.27502 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f2698eb890 0x55f2698edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 4 client.27502 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f269555890 con 0x55f2698eb890 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfb6800640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f2698eb890 0x55f2698edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcfb6800640 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f2698eb890 0x55f2698edcb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68c510fda0aa557d server_cookie=ac1794ee5009d628 in_seq=0 out_seq=0 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.601+0000 7fcf9be00640 10 client.27502 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:52.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcf9be00640 1 -- 172.21.15.22:0/3213502408 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f269555890 con 0x55f2698eb890 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcf9be00640 10 client.27502 handle_command_reply: tid=0 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 client.27502 shutdown 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 msgr2=0x55f269865c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcfa002c210 tx=0x7fcfa00042a0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 shutdown_connections 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55f2698eb890 0x55f2698edcb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f26986c950 0x55f2698661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f269863d70 0x55f269865c70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 --2- 172.21.15.22:0/3213502408 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f269863450 0x55f269874810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 >> 172.21.15.22:0/3213502408 conn(0x55f26985db80 msgr2=0x55f26985f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcf9be00640 0 client.27502 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 shutdown_connections 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 1 -- 172.21.15.22:0/3213502408 wait complete. 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.602+0000 7fcfbfc01740 20 client.27502 trim_cache size 0 max 16384 2024-03-24T06:48:52.611 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:52.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 msgr2=0x7fcf9c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 0x7fcf9c0a4770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55f269862f20 tx=0x7fcfa80d40e0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 msgr2=0x7fcfb8199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcfa800e9f0 tx=0x7fcfa800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 shutdown_connections 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fcf9c0a22b0 0x7fcf9c0a4770 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfb810bbb0 0x7fcfb819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfb8103100 0x7fcfb819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 --2- 172.21.15.22:0/714063368 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfb81027e0 0x7fcfb8199dd0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 >> 172.21.15.22:0/714063368 conn(0x7fcfb80fbf60 msgr2=0x7fcfb810de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 shutdown_connections 2024-03-24T06:48:52.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.603+0000 7fcfb7200640 1 -- 172.21.15.22:0/714063368 wait complete. 2024-03-24T06:48:52.623 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:48:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 -- 172.21.15.22:0/293035364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681058c0 msgr2=0x7f9e68105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 --2- 172.21.15.22:0/293035364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681058c0 0x7f9e68105d70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009cd0 tx=0x7f9e6002c170 comp rx=0 tx=0).stop 2024-03-24T06:48:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 -- 172.21.15.22:0/293035364 shutdown_connections 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 --2- 172.21.15.22:0/293035364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681058c0 0x7f9e68105d70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 --2- 172.21.15.22:0/293035364 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 0x7f9e68105380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 --2- 172.21.15.22:0/293035364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681002c0 0x7f9e681006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 -- 172.21.15.22:0/293035364 >> 172.21.15.22:0/293035364 conn(0x7f9e680fbf60 msgr2=0x7f9e680fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 -- 172.21.15.22:0/293035364 shutdown_connections 2024-03-24T06:48:52.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.842+0000 7f9e6e600640 1 -- 172.21.15.22:0/293035364 wait complete. 2024-03-24T06:48:52.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.843+0000 7f9e6e600640 1 Processor -- start 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.843+0000 7f9e6e600640 1 -- start start 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.843+0000 7f9e6e600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6e600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 0x7f9e6819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6e600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681058c0 0x7f9e681a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6e600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681a3240 con 0x7f9e68100be0 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6e600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681a33b0 con 0x7f9e681058c0 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6e600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681a3570 con 0x7f9e681002c0 2024-03-24T06:48:52.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37542/0 (socket says 172.21.15.22:37542) 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 -- 172.21.15.22:0/4024714047 learned_addr learned my addr 172.21.15.22:0/4024714047 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 0x7f9e6819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681058c0 0x7f9e681a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 -- 172.21.15.22:0/4024714047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681058c0 msgr2=0x7f9e681a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681058c0 0x7f9e681a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 -- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 msgr2=0x7f9e6819e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 0x7f9e6819e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.844+0000 7f9e6d200640 1 -- 172.21.15.22:0/4024714047 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e60009900 con 0x7f9e681002c0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d950 tx=0x7f9e5800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e65600640 1 -- 172.21.15.22:0/4024714047 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58009c30 con 0x7f9e681002c0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e65600640 1 -- 172.21.15.22:0/4024714047 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58009d90 con 0x7f9e681002c0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e65600640 1 -- 172.21.15.22:0/4024714047 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5800f7e0 con 0x7f9e681002c0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e681a3850 con 0x7f9e681002c0 2024-03-24T06:48:52.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.845+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e681ab0d0 con 0x7f9e681002c0 2024-03-24T06:48:52.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.848+0000 7f9e72991740 1 Processor -- start 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.848+0000 7f9e72991740 1 -- start start 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b14410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b14950 0x5627d0b16d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5627d0a796e0 con 0x5627d0b0b450 2024-03-24T06:48:52.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5627d0a76c10 con 0x5627d0b14950 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e72991740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627d0a723f0 con 0x5627d0b0bd70 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60190/0 (socket says 172.21.15.22:60190) 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 -- 172.21.15.22:0/1780513747 learned_addr learned my addr 172.21.15.22:0/1780513747 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b14950 0x5627d0b16d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 -- 172.21.15.22:0/1780513747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 msgr2=0x5627d0b14410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:48:52.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b14410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 -- 172.21.15.22:0/1780513747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b14950 msgr2=0x5627d0b16d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b14950 0x5627d0b16d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.849+0000 7f9e6dc00640 1 -- 172.21.15.22:0/1780513747 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d0a33e40 con 0x5627d0b0b450 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.850+0000 7f9e65600640 1 -- 172.21.15.22:0/4024714047 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f9e58010ad0 con 0x7f9e681002c0 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.850+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9e60004af0 tx=0x7f9e600093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b431b499867923 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.850+0000 7f9e5e000640 1 -- 172.21.15.22:0/1780513747 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60034440 con 0x5627d0b0b450 2024-03-24T06:48:52.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.850+0000 7f9e5e000640 1 -- 172.21.15.22:0/1780513747 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e60003970 con 0x5627d0b0b450 2024-03-24T06:48:52.854 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.850+0000 7f9e5e000640 1 -- 172.21.15.22:0/1780513747 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60032ce0 con 0x5627d0b0b450 2024-03-24T06:48:52.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 -- 172.21.15.22:0/1780513747 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 msgr2=0x5627d0b0b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9e60004af0 tx=0x7f9e600093e0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 -- 172.21.15.22:0/1780513747 shutdown_connections 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b14950 0x5627d0b16d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b14410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 --2- 172.21.15.22:0/1780513747 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b0b450 0x5627d0b0b830 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 -- 172.21.15.22:0/1780513747 >> 172.21.15.22:0/1780513747 conn(0x5627d0b05b80 msgr2=0x5627d0b05f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 -- 172.21.15.22:0/1780513747 shutdown_connections 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e65600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 0x7f9e4c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e72991740 1 -- 172.21.15.22:0/1780513747 wait complete. 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.851+0000 7f9e65600640 1 -- 172.21.15.22:0/4024714047 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f9e580cdff0 con 0x7f9e681002c0 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.852+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 0x7f9e4c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.852+0000 7f9e72991740 1 -- start start 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.852+0000 7f9e72991740 1 Processor -- start 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.852+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 0x7f9e4c0a4720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5627d0b06d10 tx=0x7f9e5000b390 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b11950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5627d0a17060 con 0x5627d0b14950 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5627d0a16d00 con 0x5627d0b0b450 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627d07e01c0 con 0x5627d0b0bd70 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e72991740 10 client.0 _ll_register_callbacks cb 0x5627d0782be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e6d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b11950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37544/0 (socket says 172.21.15.22:37544) 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 -- 172.21.15.22:0/4224050043 learned_addr learned my addr 172.21.15.22:0/4224050043 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 -- 172.21.15.22:0/4224050043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 msgr2=0x5627d0b11950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b11950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 -- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 msgr2=0x5627d0b123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e67e00640 1 -- 172.21.15.22:0/4224050043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e68115440 con 0x5627d0b0bd70 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.853+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b11950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e67e00640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9e50003b50 tx=0x7f9e50004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c22b229034d11f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e56a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:52.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50016070 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f9e5001fd10 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5627d07717e0 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e50027410 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e72991740 10 client.27511 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.854+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5627d0a70a70 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5627d07717e0 con 0x5627d0b0bd70 2024-03-24T06:48:52.863 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9e5001c560 con 0x5627d0b0bd70 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 10 client.27511 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 10 client.27511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 20 client.27511 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b24d60 0x5627d0b27180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 4 client.27511 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627d07ae720 con 0x5627d0b24d60 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.855+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b24d60 0x5627d0b27180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.864 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.856+0000 7f9e6d200640 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b24d60 0x5627d0b27180 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a24994e71079f36e server_cookie=cb09e610c19b9b5 in_seq=0 out_seq=0 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.856+0000 7f9e56a00640 10 client.27511 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.856+0000 7f9e56a00640 1 -- 172.21.15.22:0/4224050043 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5627d07ae720 con 0x5627d0b24d60 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.856+0000 7f9e56a00640 10 client.27511 handle_command_reply: tid=0 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.856+0000 7f9e72991740 1 client.27511 shutdown 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 msgr2=0x5627d0b11e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9e50003b50 tx=0x7f9e50004730 comp rx=0 tx=0).stop 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 shutdown_connections 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b11e90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b24d60 0x5627d0b27180 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 --2- 172.21.15.22:0/4224050043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b11950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 >> 172.21.15.22:0/4224050043 conn(0x5627d0b05b80 msgr2=0x5627d0b078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e56a00640 0 client.27511 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 shutdown_connections 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 1 -- 172.21.15.22:0/4224050043 wait complete. 2024-03-24T06:48:52.866 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.857+0000 7f9e72991740 20 client.27511 trim_cache size 0 max 16384 2024-03-24T06:48:52.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 Processor -- start 2024-03-24T06:48:52.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 -- start start 2024-03-24T06:48:52.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0b450 0x5627d0b14080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b2ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b235b0 con 0x5627d0b14950 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b059f0 con 0x5627d0b0bd70 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.872+0000 7f9e72991740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b06bb0 con 0x5627d0b0b450 2024-03-24T06:48:52.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e6dc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b2ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59214/0 (socket says 172.21.15.22:59214) 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 -- 172.21.15.22:0/1426179069 learned_addr learned my addr 172.21.15.22:0/1426179069 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e6d200640 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0b450 0x5627d0b14080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 -- 172.21.15.22:0/1426179069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0b450 msgr2=0x5627d0b14080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0b450 0x5627d0b14080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 -- 172.21.15.22:0/1426179069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 msgr2=0x5627d0b2ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b2ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 -- 172.21.15.22:0/1426179069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d0a0cb00 con 0x5627d0b0bd70 2024-03-24T06:48:52.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9e5001f800 tx=0x7f9e5001f830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cd31152d6e36ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e5d600640 1 -- 172.21.15.22:0/1426179069 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50003910 con 0x5627d0b0bd70 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.873+0000 7f9e5d600640 1 -- 172.21.15.22:0/1426179069 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e50004c80 con 0x5627d0b0bd70 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e5d600640 1 -- 172.21.15.22:0/1426179069 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50005030 con 0x5627d0b0bd70 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 -- 172.21.15.22:0/1426179069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 msgr2=0x5627d0b201b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9e5001f800 tx=0x7f9e5001f830 comp rx=0 tx=0).stop 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 -- 172.21.15.22:0/1426179069 shutdown_connections 2024-03-24T06:48:52.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b2ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0bd70 0x5627d0b201b0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 --2- 172.21.15.22:0/1426179069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0b450 0x5627d0b14080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 -- 172.21.15.22:0/1426179069 >> 172.21.15.22:0/1426179069 conn(0x5627d0b05b80 msgr2=0x5627d0b05f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 -- 172.21.15.22:0/1426179069 shutdown_connections 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.874+0000 7f9e72991740 1 -- 172.21.15.22:0/1426179069 wait complete. 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 -- start start 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 Processor -- start 2024-03-24T06:48:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b0ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b0f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b8a170 con 0x5627d0b14950 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b8a2e0 con 0x5627d0b0b450 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627d0b8a4a0 con 0x5627d0b0bd70 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e72991740 10 client.0 _ll_register_callbacks cb 0x7f9e500042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.875+0000 7f9e6d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b0ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e6d200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59220/0 (socket says 172.21.15.22:59220) 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e6d200640 1 -- 172.21.15.22:0/1444701507 learned_addr learned my addr 172.21.15.22:0/1444701507 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b0f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 -- 172.21.15.22:0/1444701507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 msgr2=0x5627d0b0e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 -- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 msgr2=0x5627d0b0f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b0f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 -- 172.21.15.22:0/1444701507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d0a70a70 con 0x5627d0b0bd70 2024-03-24T06:48:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e6d200640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e6dc00640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b0f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e67e00640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b0ec30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x5627d0771920 tx=0x7f9e5001cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd7c3356142f8e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.876+0000 7f9e56a00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50004680 con 0x5627d0b0bd70 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e60003970 con 0x5627d0b0bd70 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f9e500047e0 con 0x5627d0b0bd70 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9e500049c0 con 0x5627d0b0bd70 2024-03-24T06:48:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 10 client.27517 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5627d0a326a0 con 0x5627d0b0bd70 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e60003970 con 0x5627d0b0bd70 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f9e5002f040 con 0x5627d0b0bd70 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 10 client.27517 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 10 client.27517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 20 client.27517 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b8b970 0x5627d0b8dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 4 client.27517 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.877+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627d07fd890 con 0x5627d0b8b970 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.878+0000 7f9e6d200640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b8b970 0x5627d0b8dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.878+0000 7f9e6d200640 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b8b970 0x5627d0b8dd60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1a0d881274c6246 server_cookie=1f78a06952f4bcee in_seq=0 out_seq=0 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.878+0000 7f9e56a00640 10 client.27517 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e56a00640 1 -- 172.21.15.22:0/1444701507 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627d07fd890 con 0x5627d0b8b970 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e56a00640 10 client.27517 handle_command_reply: tid=0 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 client.27517 shutdown 2024-03-24T06:48:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 msgr2=0x5627d0b0ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b0ec30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x5627d0771920 tx=0x7f9e5001cd10 comp rx=0 tx=0).stop 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 shutdown_connections 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5627d0b14950 0x5627d0b0f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627d0b0bd70 0x5627d0b0ec30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5627d0b8b970 0x5627d0b8dd60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 --2- 172.21.15.22:0/1444701507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5627d0b0b450 0x5627d0b0e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e56a00640 0 client.27517 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:52.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 >> 172.21.15.22:0/1444701507 conn(0x5627d0b05b80 msgr2=0x5627d0b118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 shutdown_connections 2024-03-24T06:48:52.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 1 -- 172.21.15.22:0/1444701507 wait complete. 2024-03-24T06:48:52.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.879+0000 7f9e72991740 20 client.27517 trim_cache size 0 max 16384 2024-03-24T06:48:52.889 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:48:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 msgr2=0x7f9e4c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 0x7f9e4c0a4720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5627d0b06d10 tx=0x7f9e5000b390 comp rx=0 tx=0).stop 2024-03-24T06:48:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 msgr2=0x7f9e6819e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800d950 tx=0x7f9e5800de10 comp rx=0 tx=0).stop 2024-03-24T06:48:52.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 shutdown_connections 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9e4c0a2260 0x7f9e4c0a4720 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e681058c0 0x7f9e681a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e68100be0 0x7f9e6819e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 --2- 172.21.15.22:0/4024714047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e681002c0 0x7f9e6819e380 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 >> 172.21.15.22:0/4024714047 conn(0x7f9e680fbf60 msgr2=0x7f9e680fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 shutdown_connections 2024-03-24T06:48:52.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:52.880+0000 7f9e6e600640 1 -- 172.21.15.22:0/4024714047 wait complete. 2024-03-24T06:48:53.898 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:48:53.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:48:53.899 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:48:54.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 -- 172.21.15.22:0/2895315345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 msgr2=0x7f766810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 --2- 172.21.15.22:0/2895315345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766810e570 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7660009d00 tx=0x7f766002c0e0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 -- 172.21.15.22:0/2895315345 shutdown_connections 2024-03-24T06:48:54.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 --2- 172.21.15.22:0/2895315345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766810e570 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 --2- 172.21.15.22:0/2895315345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668068910 0x7f7668068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 --2- 172.21.15.22:0/2895315345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668107760 0x7f7668107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.133+0000 7f766e200640 1 -- 172.21.15.22:0/2895315345 >> 172.21.15.22:0/2895315345 conn(0x7f7668077710 msgr2=0x7f7668077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.134+0000 7f766e200640 1 -- 172.21.15.22:0/2895315345 shutdown_connections 2024-03-24T06:48:54.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.134+0000 7f766e200640 1 -- 172.21.15.22:0/2895315345 wait complete. 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 Processor -- start 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 -- start start 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 0x7f766819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.137 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f766819edb0 con 0x7f7668107760 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f766819ef20 con 0x7f76680692d0 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f766e200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f766819f0e0 con 0x7f7668068910 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7667e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 0x7f766819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7667400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7666a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7666a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766819a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59230/0 (socket says 172.21.15.22:59230) 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7667400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37590/0 (socket says 172.21.15.22:37590) 2024-03-24T06:48:54.138 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7666a00640 1 -- 172.21.15.22:0/1720570499 learned_addr learned my addr 172.21.15.22:0/1720570499 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.135+0000 7f7667e00640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 0x7f766819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60244/0 (socket says 172.21.15.22:60244) 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 -- 172.21.15.22:0/1720570499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 msgr2=0x7f766819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 -- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 msgr2=0x7f766819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 0x7f766819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 -- 172.21.15.22:0/1720570499 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7660009900 con 0x7f7668068910 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.136+0000 7f7667400640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f765800e9f0 tx=0x7f765800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.137+0000 7f765fe00640 1 -- 172.21.15.22:0/1720570499 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765800ccc0 con 0x7f7668068910 2024-03-24T06:48:54.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.137+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766819f3c0 con 0x7f7668068910 2024-03-24T06:48:54.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.137+0000 7f765fe00640 1 -- 172.21.15.22:0/1720570499 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f765800ce20 con 0x7f7668068910 2024-03-24T06:48:54.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.137+0000 7f765fe00640 1 -- 172.21.15.22:0/1720570499 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658018920 con 0x7f7668068910 2024-03-24T06:48:54.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.137+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76681a6c40 con 0x7f7668068910 2024-03-24T06:48:54.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.140+0000 7f7670ded740 1 Processor -- start 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.140+0000 7f7670ded740 1 -- start start 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.140+0000 7f7670ded740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.140+0000 7f7670ded740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de6ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.140+0000 7f7670ded740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de6e330 0x55bb4de70720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7670ded740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bb4ddd32d0 con 0x55bb4de6e330 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7670ded740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bb4ddd0800 con 0x55bb4de65750 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7670ded740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bb4ddcbfe0 con 0x55bb4de64e30 2024-03-24T06:48:54.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7666a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de6e330 0x55bb4de70720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37594/0 (socket says 172.21.15.22:37594) 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 -- 172.21.15.22:0/2848116630 learned_addr learned my addr 172.21.15.22:0/2848116630 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 -- 172.21.15.22:0/2848116630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 msgr2=0x55bb4de6ddf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de6ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 -- 172.21.15.22:0/2848116630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de6e330 msgr2=0x55bb4de70720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de6e330 0x55bb4de70720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f7667e00640 1 -- 172.21.15.22:0/2848116630 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb4dd8de90 con 0x55bb4de64e30 2024-03-24T06:48:54.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.141+0000 7f765fe00640 1 -- 172.21.15.22:0/1720570499 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7658018a80 con 0x7f7668068910 2024-03-24T06:48:54.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7667e00640 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7660004af0 tx=0x7f7660032d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42f70f0cc81f0a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7653e00640 1 -- 172.21.15.22:0/2848116630 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660034440 con 0x55bb4de64e30 2024-03-24T06:48:54.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7653e00640 1 -- 172.21.15.22:0/2848116630 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76600345a0 con 0x55bb4de64e30 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7653e00640 1 -- 172.21.15.22:0/2848116630 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660032b70 con 0x55bb4de64e30 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7670ded740 1 -- 172.21.15.22:0/2848116630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 msgr2=0x55bb4de65210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.142+0000 7f7670ded740 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7660004af0 tx=0x7f7660032d70 comp rx=0 tx=0).stop 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 -- 172.21.15.22:0/2848116630 shutdown_connections 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de6e330 0x55bb4de70720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de6ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 --2- 172.21.15.22:0/2848116630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de64e30 0x55bb4de65210 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 -- 172.21.15.22:0/2848116630 >> 172.21.15.22:0/2848116630 conn(0x55bb4de5f560 msgr2=0x55bb4de5f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 -- 172.21.15.22:0/2848116630 shutdown_connections 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 -- 172.21.15.22:0/2848116630 wait complete. 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 -- start start 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.143+0000 7f7670ded740 1 Processor -- start 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f765fe00640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 0x7f764c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f765fe00640 1 -- 172.21.15.22:0/1720570499 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f7658014070 con 0x7f7668068910 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7667e00640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 0x7f764c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de6e330 0x55bb4de6be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bb4dd70fc0 con 0x55bb4de64e30 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bb4dd70c60 con 0x55bb4de6e330 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bb4db140c0 con 0x55bb4de65750 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7670ded740 10 client.0 _ll_register_callbacks cb 0x55bb4dd88e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7666a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7666a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37610/0 (socket says 172.21.15.22:37610) 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7666a00640 1 -- 172.21.15.22:0/3802113544 learned_addr learned my addr 172.21.15.22:0/3802113544 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7667e00640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de6e330 0x55bb4de6be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.144+0000 7f7667400640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 -- 172.21.15.22:0/3802113544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de6e330 msgr2=0x55bb4de6be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de6e330 0x55bb4de6be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 -- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 msgr2=0x55bb4de6b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 -- 172.21.15.22:0/3802113544 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766810ee70 con 0x55bb4de65750 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7667400640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7666a00640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7654012780 tx=0x7f7654012c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51cd35be27d31d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7650c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7667e00640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 0x7f764c0a4600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7660033000 tx=0x7f76600329b0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654017070 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb4ddf0e90 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.145+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f7654004050 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7654004210 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 10 client.27529 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb4ddca660 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bb4ddf0e90 con 0x55bb4de65750 2024-03-24T06:48:54.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f76540039d0 con 0x55bb4de65750 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 10 client.27529 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 10 client.27529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 20 client.27529 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4de7e8a0 0x55bb4de80cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 4 client.27529 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.146+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb4ddc2210 con 0x55bb4de7e8a0 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.147+0000 7f7667400640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4de7e8a0 0x55bb4de80cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.147+0000 7f7667400640 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4de7e8a0 0x55bb4de80cc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d348e1935e83185a server_cookie=fc94d316ab0d9edf in_seq=0 out_seq=0 2024-03-24T06:48:54.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.147+0000 7f7650c00640 10 client.27529 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.148+0000 7f7650c00640 1 -- 172.21.15.22:0/3802113544 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb4ddc2210 con 0x55bb4de7e8a0 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.148+0000 7f7650c00640 10 client.27529 handle_command_reply: tid=0 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.148+0000 7f7670ded740 1 client.27529 shutdown 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.148+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 msgr2=0x55bb4de6b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.148+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7654012780 tx=0x7f7654012c40 comp rx=0 tx=0).stop 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 shutdown_connections 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4de7e8a0 0x55bb4de80cc0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de6e330 0x55bb4de6be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.155 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de65750 0x55bb4de6b8f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 --2- 172.21.15.22:0/3802113544 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 >> 172.21.15.22:0/3802113544 conn(0x55bb4de5f560 msgr2=0x55bb4de612b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7650c00640 0 client.27529 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 shutdown_connections 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 1 -- 172.21.15.22:0/3802113544 wait complete. 2024-03-24T06:48:54.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.149+0000 7f7670ded740 20 client.27529 trim_cache size 0 max 16384 2024-03-24T06:48:54.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.168+0000 7f7670ded740 1 Processor -- start 2024-03-24T06:48:54.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 -- start start 2024-03-24T06:48:54.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de79b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.170 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bb4de5ac80 con 0x55bb4de64e30 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bb4de7a0a0 con 0x55bb4de65750 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7670ded740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bb4de6c730 con 0x55bb4de6e330 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37620/0 (socket says 172.21.15.22:37620) 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7667e00640 1 -- 172.21.15.22:0/2939810158 learned_addr learned my addr 172.21.15.22:0/2939810158 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7666a00640 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de79b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.169+0000 7f7667400640 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 -- 172.21.15.22:0/2939810158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 msgr2=0x55bb4de79b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de79b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 -- 172.21.15.22:0/2939810158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 msgr2=0x55bb4de6dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 -- 172.21.15.22:0/2939810158 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb4dd66ae0 con 0x55bb4de6e330 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7667e00640 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7660047000 tx=0x7f7660051000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e516c4355eff9371 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7653400640 1 -- 172.21.15.22:0/2939810158 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660032340 con 0x55bb4de6e330 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7653400640 1 -- 172.21.15.22:0/2939810158 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7660034e90 con 0x55bb4de6e330 2024-03-24T06:48:54.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.170+0000 7f7653400640 1 -- 172.21.15.22:0/2939810158 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660004710 con 0x55bb4de6e330 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 -- 172.21.15.22:0/2939810158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 msgr2=0x55bb4de61040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7660047000 tx=0x7f7660051000 comp rx=0 tx=0).stop 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 -- 172.21.15.22:0/2939810158 shutdown_connections 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4de61040 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de79b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 --2- 172.21.15.22:0/2939810158 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de6dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 -- 172.21.15.22:0/2939810158 >> 172.21.15.22:0/2939810158 conn(0x55bb4de5f560 msgr2=0x55bb4de60930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 -- 172.21.15.22:0/2939810158 shutdown_connections 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.171+0000 7f7670ded740 1 -- 172.21.15.22:0/2939810158 wait complete. 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 -- start start 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 Processor -- start 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de68280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4dee3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55bb4de68f20 con 0x55bb4de64e30 2024-03-24T06:48:54.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bb4de69090 con 0x55bb4de65750 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.172+0000 7f7670ded740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bb4dee4100 con 0x55bb4de6e330 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7670ded740 10 client.0 _ll_register_callbacks cb 0x7f76540099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7667400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de68280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7667e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4dee3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7667400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de68280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60276/0 (socket says 172.21.15.22:60276) 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59266/0 (socket says 172.21.15.22:59266) 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7667400640 1 -- 172.21.15.22:0/630805511 learned_addr learned my addr 172.21.15.22:0/630805511 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 -- 172.21.15.22:0/630805511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 msgr2=0x55bb4dee3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4dee3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 -- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 msgr2=0x55bb4de68280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de68280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 -- 172.21.15.22:0/630805511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb4dd8de90 con 0x55bb4de65750 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7666a00640 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7654003d60 tx=0x7f765400fbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27a0e95ccbee762f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.173+0000 7f7650c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654017070 con 0x55bb4de65750 2024-03-24T06:48:54.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76600345a0 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f76540035f0 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7654005d30 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7670ded740 10 client.27509 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb4dd8c6f0 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.174+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76600345a0 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7654010430 con 0x55bb4de65750 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 10 client.27509 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 10 client.27509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:54.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 20 client.27509 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4dee5a00 0x55bb4dee7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 4 client.27509 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bb4dacb860 con 0x55bb4dee5a00 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7667400640 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4dee5a00 0x55bb4dee7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7667400640 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4dee5a00 0x55bb4dee7e20 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bc5ec5bcf29c4bc server_cookie=b037a953ebbfb03a in_seq=0 out_seq=0 2024-03-24T06:48:54.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.175+0000 7f7650c00640 10 client.27509 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7650c00640 1 -- 172.21.15.22:0/630805511 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bb4dacb860 con 0x55bb4dee5a00 2024-03-24T06:48:54.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7650c00640 10 client.27509 handle_command_reply: tid=0 2024-03-24T06:48:54.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 client.27509 shutdown 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 msgr2=0x55bb4de687c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7654003d60 tx=0x7f765400fbd0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 shutdown_connections 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bb4de6e330 0x55bb4dee3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55bb4dee5a00 0x55bb4dee7e20 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bb4de65750 0x55bb4de687c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7670ded740 1 --2- 172.21.15.22:0/630805511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55bb4de64e30 0x55bb4de68280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.177+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 >> 172.21.15.22:0/630805511 conn(0x55bb4de5f560 msgr2=0x55bb4de6bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.176+0000 7f7650c00640 0 client.27509 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.177+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 shutdown_connections 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.177+0000 7f7670ded740 1 -- 172.21.15.22:0/630805511 wait complete. 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.177+0000 7f7670ded740 20 client.27509 trim_cache size 0 max 16384 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "b7529afd-5144-4c68-9b77-0801d1adffe0", 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:48:54.184 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:54.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 msgr2=0x7f764c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 0x7f764c0a4600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7660033000 tx=0x7f76600329b0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 msgr2=0x7f7668199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f765800e9f0 tx=0x7f765800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 shutdown_connections 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f764c0a2140 0x7f764c0a4600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7668107760 0x7f766819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76680692d0 0x7f766819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 --2- 172.21.15.22:0/1720570499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7668068910 0x7f7668199ef0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.178+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 >> 172.21.15.22:0/1720570499 conn(0x7f7668077710 msgr2=0x7f76680fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.179+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 shutdown_connections 2024-03-24T06:48:54.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.179+0000 7f766e200640 1 -- 172.21.15.22:0/1720570499 wait complete. 2024-03-24T06:48:54.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:48:54.203 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:48:54.266 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:54 smithi022 ceph-mon[32003]: pgmap v275: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:54.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 -- 172.21.15.22:0/1084812921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1180100500 msgr2=0x7f11801008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 --2- 172.21.15.22:0/1084812921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1180100500 0x7f11801008e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1170009900 tx=0x7f117002bc30 comp rx=0 tx=0).stop 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 -- 172.21.15.22:0/1084812921 shutdown_connections 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 --2- 172.21.15.22:0/1084812921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f118010e0b0 0x7f11801104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 --2- 172.21.15.22:0/1084812921 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100eb0 0x7f118010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 --2- 172.21.15.22:0/1084812921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1180100500 0x7f11801008e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 -- 172.21.15.22:0/1084812921 >> 172.21.15.22:0/1084812921 conn(0x7f1180077770 msgr2=0x7f1180077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 -- 172.21.15.22:0/1084812921 shutdown_connections 2024-03-24T06:48:54.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.398+0000 7f1185400640 1 -- 172.21.15.22:0/1084812921 wait complete. 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.399+0000 7f1185400640 1 Processor -- start 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.399+0000 7f1185400640 1 -- start start 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.399+0000 7f1185400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 0x7f118019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1185400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1185400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 0x7f118019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1185400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f118019eef0 con 0x7f1180100500 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1185400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f118019f060 con 0x7f118010e0b0 2024-03-24T06:48:54.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1185400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f118019f220 con 0x7f1180100eb0 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f117e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 0x7f118019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37650/0 (socket says 172.21.15.22:37650) 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f117e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 0x7f118019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60294/0 (socket says 172.21.15.22:60294) 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 -- 172.21.15.22:0/2442734533 learned_addr learned my addr 172.21.15.22:0/2442734533 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f117f200640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 0x7f118019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 -- 172.21.15.22:0/2442734533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 msgr2=0x7f118019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 0x7f118019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 -- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 msgr2=0x7f118019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 0x7f118019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f1175e00640 1 -- 172.21.15.22:0/2442734533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1170009560 con 0x7f1180100eb0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.400+0000 7f117f200640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 0x7f118019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1175e00640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f116800d950 tx=0x7f116800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1177e00640 1 -- 172.21.15.22:0/2442734533 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168009a30 con 0x7f1180100eb0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118019f500 con 0x7f1180100eb0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1177e00640 1 -- 172.21.15.22:0/2442734533 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1168009b90 con 0x7f1180100eb0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1177e00640 1 -- 172.21.15.22:0/2442734533 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168004a00 con 0x7f1180100eb0 2024-03-24T06:48:54.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.401+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11801a6d80 con 0x7f1180100eb0 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.404+0000 7f1188180740 1 Processor -- start 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.404+0000 7f1188180740 1 -- start start 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1177e00640 1 -- 172.21.15.22:0/2442734533 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1168009d00 con 0x7f1180100eb0 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dc9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5650e7d37790 con 0x5650e7dd20f0 2024-03-24T06:48:54.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5650e7d34cc0 con 0x5650e7dc8d80 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1188180740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5650e7d304a0 con 0x5650e7dc9a60 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f117f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f117e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dc9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f117f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37658/0 (socket says 172.21.15.22:37658) 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f117f200640 1 -- 172.21.15.22:0/1292845284 learned_addr learned my addr 172.21.15.22:0/1292845284 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f117e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dc9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59304/0 (socket says 172.21.15.22:59304) 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.405+0000 7f1175e00640 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 -- 172.21.15.22:0/1292845284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 msgr2=0x5650e7dc9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dc9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 -- 172.21.15.22:0/1292845284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 msgr2=0x5650e7dd44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 -- 172.21.15.22:0/1292845284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650e7cf1c20 con 0x5650e7dc9a60 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f1175e00640 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f117f200640 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1178009f80 tx=0x7f1178010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47f5917af66f4ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f116f400640 1 -- 172.21.15.22:0/1292845284 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178011700 con 0x5650e7dc9a60 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f116f400640 1 -- 172.21.15.22:0/1292845284 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1178015070 con 0x5650e7dc9a60 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.406+0000 7f116f400640 1 -- 172.21.15.22:0/1292845284 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117801cdc0 con 0x5650e7dc9a60 2024-03-24T06:48:54.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- 172.21.15.22:0/1292845284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 msgr2=0x5650e7dc87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1178009f80 tx=0x7f1178010860 comp rx=0 tx=0).stop 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- 172.21.15.22:0/1292845284 shutdown_connections 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dc9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 --2- 172.21.15.22:0/1292845284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dc87b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- 172.21.15.22:0/1292845284 >> 172.21.15.22:0/1292845284 conn(0x5650e7d8fb30 msgr2=0x5650e7dc37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- 172.21.15.22:0/1292845284 shutdown_connections 2024-03-24T06:48:54.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- 172.21.15.22:0/1292845284 wait complete. 2024-03-24T06:48:54.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1177e00640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 0x7f11640a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.407+0000 7f1188180740 1 -- start start 2024-03-24T06:48:54.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.408+0000 7f1177e00640 1 -- 172.21.15.22:0/2442734533 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f11680cde50 con 0x7f1180100eb0 2024-03-24T06:48:54.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.408+0000 7f117f200640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 0x7f11640a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.408+0000 7f1188180740 1 Processor -- start 2024-03-24T06:48:54.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.408+0000 7f1188180740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 0x5650e7dcf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.408+0000 7f1188180740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1188180740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7dcfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1188180740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5650e7cd4f20 con 0x5650e7dc8d80 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1188180740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5650e7cd4bc0 con 0x5650e7dc9a60 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1188180740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5650e7a9e1c0 con 0x5650e7dd20f0 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1188180740 10 client.0 _ll_register_callbacks cb 0x5650e7a40be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f117f200640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 0x7f11640a4680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1178010d60 tx=0x7f1178013040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f117e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 0x5650e7dcf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59308/0 (socket says 172.21.15.22:59308) 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f117f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7dcfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f117e800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 0x5650e7dcf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60308/0 (socket says 172.21.15.22:60308) 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 -- 172.21.15.22:0/1689371118 learned_addr learned my addr 172.21.15.22:0/1689371118 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 -- 172.21.15.22:0/1689371118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 msgr2=0x5650e7dcfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7dcfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 -- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 msgr2=0x5650e7dcf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 0x5650e7dcf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.409+0000 7f1175e00640 1 -- 172.21.15.22:0/1689371118 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1178009bb0 con 0x5650e7dc9a60 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1175e00640 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1168009610 tx=0x7f116809a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c18094acbb62a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680cb610 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f116809ab30 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5650e7d28160 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f116809a430 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1188180740 10 client.27521 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5650e7d2eb20 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5650e7d28160 con 0x5650e7dc9a60 2024-03-24T06:48:54.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f116809e220 con 0x5650e7dc9a60 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.410+0000 7f1188180740 10 client.27521 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f1188180740 10 client.27521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f1188180740 20 client.27521 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7de2680 0x5650e7de4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f1188180740 4 client.27521 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650e7a2f7e0 con 0x5650e7de2680 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.411+0000 7f117e800640 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7de2680 0x5650e7de4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.412+0000 7f117e800640 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7de2680 0x5650e7de4aa0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c072302e8fad0ed5 server_cookie=d804481da227a0be in_seq=0 out_seq=0 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.412+0000 7f1163e00640 10 client.27521 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.412+0000 7f1163e00640 1 -- 172.21.15.22:0/1689371118 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650e7a2f7e0 con 0x5650e7de2680 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.412+0000 7f1163e00640 10 client.27521 handle_command_reply: tid=0 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.412+0000 7f1188180740 1 client.27521 shutdown 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 msgr2=0x5650e7dcf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1168009610 tx=0x7f116809a200 comp rx=0 tx=0).stop 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 shutdown_connections 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7dcfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7de2680 0x5650e7de4aa0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc9a60 0x5650e7dcf620 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 --2- 172.21.15.22:0/1689371118 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc8d80 0x5650e7dcf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1163e00640 0 client.27521 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 >> 172.21.15.22:0/1689371118 conn(0x5650e7d8fb30 msgr2=0x5650e7dc5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 shutdown_connections 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 1 -- 172.21.15.22:0/1689371118 wait complete. 2024-03-24T06:48:54.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.413+0000 7f1188180740 20 client.27521 trim_cache size 0 max 16384 2024-03-24T06:48:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:54 smithi143 ceph-mon[36520]: pgmap v275: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.435+0000 7f1188180740 1 Processor -- start 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.435+0000 7f1188180740 1 -- start start 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc9a60 0x5650e7dc3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7ddda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.436 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5650e7dc5b30 con 0x5650e7dc9a60 2024-03-24T06:48:54.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5650e7de2490 con 0x5650e7dc8d80 2024-03-24T06:48:54.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1188180740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5650e7dc47d0 con 0x5650e7dd20f0 2024-03-24T06:48:54.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7ddda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f1175e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc9a60 0x5650e7dc3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117f200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7ddda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37670/0 (socket says 172.21.15.22:37670) 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117f200640 1 -- 172.21.15.22:0/788814285 learned_addr learned my addr 172.21.15.22:0/788814285 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 -- 172.21.15.22:0/788814285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 msgr2=0x5650e7ddda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7ddda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 -- 172.21.15.22:0/788814285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc9a60 msgr2=0x5650e7dc3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc9a60 0x5650e7dc3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.436+0000 7f117e800640 1 -- 172.21.15.22:0/788814285 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650e7cca9a0 con 0x5650e7dc8d80 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.437+0000 7f117e800640 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dce040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f117002c170 tx=0x7f1170005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c399cc0abde1b3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.437+0000 7f116ea00640 1 -- 172.21.15.22:0/788814285 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170045040 con 0x5650e7dc8d80 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.437+0000 7f116ea00640 1 -- 172.21.15.22:0/788814285 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1170047040 con 0x5650e7dc8d80 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.437+0000 7f116ea00640 1 -- 172.21.15.22:0/788814285 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117002cca0 con 0x5650e7dc8d80 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- 172.21.15.22:0/788814285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 msgr2=0x5650e7dce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dce040 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f117002c170 tx=0x7f1170005a20 comp rx=0 tx=0).stop 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- 172.21.15.22:0/788814285 shutdown_connections 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dd20f0 0x5650e7ddda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dc9a60 0x5650e7dc3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 --2- 172.21.15.22:0/788814285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dce040 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- 172.21.15.22:0/788814285 >> 172.21.15.22:0/788814285 conn(0x5650e7d8fb30 msgr2=0x5650e7dc5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- 172.21.15.22:0/788814285 shutdown_connections 2024-03-24T06:48:54.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- 172.21.15.22:0/788814285 wait complete. 2024-03-24T06:48:54.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.438+0000 7f1188180740 1 -- start start 2024-03-24T06:48:54.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 Processor -- start 2024-03-24T06:48:54.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dd3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5650e7dd3fe0 con 0x5650e7dd20f0 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5650e7dd8b40 con 0x5650e7dc8d80 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5650e7dd8d00 con 0x5650e7dc9a60 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1188180740 10 client.0 _ll_register_callbacks cb 0x7f11680046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f117e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f117e800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:59318/0 (socket says 172.21.15.22:59318) 2024-03-24T06:48:54.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f117e800640 1 -- 172.21.15.22:0/459270404 learned_addr learned my addr 172.21.15.22:0/459270404 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f1175e00640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dd3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.439+0000 7f117f200640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 -- 172.21.15.22:0/459270404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 msgr2=0x5650e7dd3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dd3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 -- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 msgr2=0x5650e7dd85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 -- 172.21.15.22:0/459270404 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11801110d0 con 0x5650e7dc8d80 2024-03-24T06:48:54.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f117e800640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1170006010 tx=0x7f117004dba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad1dbd43c274fec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f1163e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170045070 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9467+0+0 (secure 0 0 0) 0x7f117002c920 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.440+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5650e7dc3960 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f117002cb00 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 10 client.27527 fetch_fsmap learned FSMap version 37 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5650e7cf0480 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5650e7dc3960 con 0x5650e7dc8d80 2024-03-24T06:48:54.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f117005c030 con 0x5650e7dc8d80 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 10 client.27527 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 10 client.27527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 20 client.27527 populate_metadata read hostname 'smithi022' 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7e49670 0x5650e7e4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 4 client.27527 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f1178015070 con 0x5650e7e49670 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.441+0000 7f1175e00640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7e49670 0x5650e7e4ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.442+0000 7f1175e00640 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7e49670 0x5650e7e4ba60 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f8c581f8faa4bad server_cookie=d056d9932a7483af in_seq=0 out_seq=0 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.442+0000 7f1163e00640 10 client.27527 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.442+0000 7f1163e00640 1 -- 172.21.15.22:0/459270404 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f1178015070 con 0x5650e7e49670 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.442+0000 7f1163e00640 10 client.27527 handle_command_reply: tid=0 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.442+0000 7f1188180740 1 client.27527 shutdown 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 msgr2=0x5650e7dd3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1170006010 tx=0x7f117004dba0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 shutdown_connections 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5650e7dd20f0 0x5650e7dd85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5650e7dc9a60 0x5650e7dd3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5650e7e49670 0x5650e7e4ba60 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 --2- 172.21.15.22:0/459270404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5650e7dc8d80 0x5650e7dd3340 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 >> 172.21.15.22:0/459270404 conn(0x5650e7d8fb30 msgr2=0x5650e7dc1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1163e00640 0 client.27527 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 shutdown_connections 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 1 -- 172.21.15.22:0/459270404 wait complete. 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.443+0000 7f1188180740 20 client.27527 trim_cache size 0 max 16384 2024-03-24T06:48:54.450 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "b7529afd-5144-4c68-9b77-0801d1adffe0": { 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "b7529afd-5144-4c68-9b77-0801d1adffe0", 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:48:54.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:48:54.452 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:48:54.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 msgr2=0x7f11640a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 0x7f11640a4680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1178010d60 tx=0x7f1178013040 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 msgr2=0x7f118019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f116800d950 tx=0x7f116800de10 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 shutdown_connections 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f11640a21c0 0x7f11640a4680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f118010e0b0 0x7f118019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1180100eb0 0x7f118019a5c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 --2- 172.21.15.22:0/2442734533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1180100500 0x7f118019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:48:54.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 >> 172.21.15.22:0/2442734533 conn(0x7f1180077770 msgr2=0x7f118010f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:48:54.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 shutdown_connections 2024-03-24T06:48:54.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:48:54.444+0000 7f1185400640 1 -- 172.21.15.22:0/2442734533 wait complete. 2024-03-24T06:48:54.462 INFO:tasks.cephfs.filesystem:scrub status for tag:b7529afd-5144-4c68-9b77-0801d1adffe0 - {'path': '/', 'tag': 'b7529afd-5144-4c68-9b77-0801d1adffe0', 'options': 'recursive,force'} 2024-03-24T06:48:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:54 smithi155 ceph-mon[36262]: pgmap v275: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:48:55.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:48:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.401 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:55 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:48:55.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:55 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:48:55.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:48:55.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:55.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:55 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:48:56.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:56 smithi022 ceph-mon[32003]: pgmap v276: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:48:56.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:56 smithi143 ceph-mon[36520]: pgmap v276: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:48:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:56 smithi155 ceph-mon[36262]: pgmap v276: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-24T06:48:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:48:58 smithi022 ceph-mon[32003]: pgmap v277: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:48:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:48:58 smithi143 ceph-mon[36520]: pgmap v277: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:48:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:48:58 smithi155 ceph-mon[36262]: pgmap v277: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:00.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:00 smithi022 ceph-mon[32003]: pgmap v278: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:00 smithi143 ceph-mon[36520]: pgmap v278: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:00 smithi155 ceph-mon[36262]: pgmap v278: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:02.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:02 smithi022 ceph-mon[32003]: pgmap v279: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:02.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:02 smithi143 ceph-mon[36520]: pgmap v279: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:02 smithi155 ceph-mon[36262]: pgmap v279: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:04.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:04 smithi022 ceph-mon[32003]: pgmap v280: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:49:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:04 smithi143 ceph-mon[36520]: pgmap v280: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:49:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:04 smithi155 ceph-mon[36262]: pgmap v280: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-24T06:49:06.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:06 smithi022 ceph-mon[32003]: pgmap v281: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:49:06.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:06 smithi143 ceph-mon[36520]: pgmap v281: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:49:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:06 smithi155 ceph-mon[36262]: pgmap v281: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 26 op/s 2024-03-24T06:49:08.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:08 smithi022 ceph-mon[32003]: pgmap v282: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:49:08.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:08 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:08.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:08 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:08.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:08 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:08.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:08 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:08 smithi143 ceph-mon[36520]: pgmap v282: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:49:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:08 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:08 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:08 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:08.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:08 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:08 smithi155 ceph-mon[36262]: pgmap v282: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 45 KiB/s wr, 28 op/s 2024-03-24T06:49:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:08 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:08 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:08.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:08 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:08.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:08 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:09 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:09.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:09 smithi022 ceph-mon[32003]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T06:49:09.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:09 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:09.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:09 smithi143 ceph-mon[36520]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T06:49:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:09 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:09 smithi155 ceph-mon[36262]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T06:49:10.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:10 smithi022 ceph-mon[32003]: pgmap v284: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:10.446 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:10 smithi143 ceph-mon[36520]: pgmap v284: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:10.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:10 smithi155 ceph-mon[36262]: pgmap v284: 129 pgs: 129 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:12.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:12 smithi022 ceph-mon[32003]: pgmap v285: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:12.435 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:12 smithi143 ceph-mon[36520]: pgmap v285: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:12 smithi155 ceph-mon[36262]: pgmap v285: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-24T06:49:14.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:14 smithi143 ceph-mon[36520]: pgmap v286: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:49:14.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:14 smithi155 ceph-mon[36262]: pgmap v286: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:49:14.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:14 smithi022 ceph-mon[32003]: pgmap v286: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 14 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:49:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:16 smithi143 ceph-mon[36520]: pgmap v287: 129 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 115 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:16 smithi155 ceph-mon[36262]: pgmap v287: 129 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 115 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:16.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:16 smithi022 ceph-mon[32003]: pgmap v287: 129 pgs: 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 115 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:18 smithi143 ceph-mon[36520]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:18 smithi155 ceph-mon[36262]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:18.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:18 smithi022 ceph-mon[32003]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:20 smithi143 ceph-mon[36520]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 187 B/s wr, 9 op/s 2024-03-24T06:49:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:20 smithi143 ceph-mon[36520]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T06:49:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:20 smithi155 ceph-mon[36262]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 187 B/s wr, 9 op/s 2024-03-24T06:49:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:20 smithi155 ceph-mon[36262]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T06:49:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:20 smithi022 ceph-mon[32003]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 187 B/s wr, 9 op/s 2024-03-24T06:49:20.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:20 smithi022 ceph-mon[32003]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T06:49:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:22 smithi143 ceph-mon[36520]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:22 smithi155 ceph-mon[36262]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:22.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:22 smithi022 ceph-mon[32003]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:49:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:24 smithi143 ceph-mon[36520]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:24.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:49:24.463 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:49:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:24 smithi155 ceph-mon[36262]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:24.551 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:24 smithi022 ceph-mon[32003]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:24.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.695+0000 7fc159a00640 1 -- 172.21.15.22:0/896372849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc154102780 msgr2=0x7fc154102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.695+0000 7fc159a00640 1 --2- 172.21.15.22:0/896372849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc154102780 0x7fc154102b60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc144009900 tx=0x7fc14402bcc0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.695+0000 7fc159a00640 1 -- 172.21.15.22:0/896372849 shutdown_connections 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.695+0000 7fc159a00640 1 --2- 172.21.15.22:0/896372849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc15410bb50 0x7fc15410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.695+0000 7fc159a00640 1 --2- 172.21.15.22:0/896372849 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 0x7fc15410b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.696+0000 7fc159a00640 1 --2- 172.21.15.22:0/896372849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc154102780 0x7fc154102b60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.696+0000 7fc159a00640 1 -- 172.21.15.22:0/896372849 >> 172.21.15.22:0/896372849 conn(0x7fc1540fbf60 msgr2=0x7fc1540fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.696+0000 7fc159a00640 1 -- 172.21.15.22:0/896372849 shutdown_connections 2024-03-24T06:49:24.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.696+0000 7fc159a00640 1 -- 172.21.15.22:0/896372849 wait complete. 2024-03-24T06:49:24.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.697+0000 7fc159a00640 1 Processor -- start 2024-03-24T06:49:24.697 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.697+0000 7fc159a00640 1 -- start start 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 0x7fc154100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 0x7fc154100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc1541a5f90 con 0x7fc1541030a0 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc1541a6100 con 0x7fc15410bb50 2024-03-24T06:49:24.698 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc1541a62c0 con 0x7fc154102780 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc153e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc153e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:35798/0 (socket says 172.21.15.22:35798) 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc153e00640 1 -- 172.21.15.22:0/832087730 learned_addr learned my addr 172.21.15.22:0/832087730 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc159000640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 0x7fc154100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.698+0000 7fc153400640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 0x7fc154100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 -- 172.21.15.22:0/832087730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 msgr2=0x7fc154100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 0x7fc154100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 -- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 msgr2=0x7fc154100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 0x7fc154100910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 -- 172.21.15.22:0/832087730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c009840 con 0x7fc154102780 2024-03-24T06:49:24.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc159000640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 0x7fc154100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:24.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.699+0000 7fc153e00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc14402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.700+0000 7fc150c00640 1 -- 172.21.15.22:0/832087730 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14402db10 con 0x7fc154102780 2024-03-24T06:49:24.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.700+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc144009560 con 0x7fc154102780 2024-03-24T06:49:24.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.700+0000 7fc150c00640 1 -- 172.21.15.22:0/832087730 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14402dc70 con 0x7fc154102780 2024-03-24T06:49:24.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.700+0000 7fc150c00640 1 -- 172.21.15.22:0/832087730 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc144003ab0 con 0x7fc154102780 2024-03-24T06:49:24.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.700+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1541a68c0 con 0x7fc154102780 2024-03-24T06:49:24.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.703+0000 7fc150c00640 1 -- 172.21.15.22:0/832087730 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc1440053d0 con 0x7fc154102780 2024-03-24T06:49:24.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.703+0000 7fc15de35740 1 Processor -- start 2024-03-24T06:49:24.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.703+0000 7fc15de35740 1 -- start start 2024-03-24T06:49:24.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bb93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff46b27790 con 0x55ff46bb8f60 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff46b24cc0 con 0x55ff46bb9c40 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc15de35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff46b204a0 con 0x55ff46bc22d0 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc153e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bb93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35874/0 (socket says 172.21.15.22:35874) 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 -- 172.21.15.22:0/2378285752 learned_addr learned my addr 172.21.15.22:0/2378285752 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc153400640 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 -- 172.21.15.22:0/2378285752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 msgr2=0x55ff46bc46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 -- 172.21.15.22:0/2378285752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 msgr2=0x55ff46bb93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bb93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.704+0000 7fc159000640 1 -- 172.21.15.22:0/2378285752 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff46ae1c20 con 0x55ff46bb9c40 2024-03-24T06:49:24.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc159000640 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00e890 tx=0x7fc14c00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ec9fcab42c84ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc149600640 1 -- 172.21.15.22:0/2378285752 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c011ce0 con 0x55ff46bb9c40 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc150c00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 0x7fc1380a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc149600640 1 -- 172.21.15.22:0/2378285752 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14c003e50 con 0x55ff46bb9c40 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc149600640 1 -- 172.21.15.22:0/2378285752 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c019430 con 0x55ff46bb9c40 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc150c00640 1 -- 172.21.15.22:0/832087730 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc1440f0870 con 0x7fc154102780 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.705+0000 7fc153e00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 0x7fc1380a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 -- 172.21.15.22:0/2378285752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 msgr2=0x55ff46bb8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00e890 tx=0x7fc14c00ed50 comp rx=0 tx=0).stop 2024-03-24T06:49:24.707 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 -- 172.21.15.22:0/2378285752 shutdown_connections 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bb93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 --2- 172.21.15.22:0/2378285752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bb8990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 -- 172.21.15.22:0/2378285752 >> 172.21.15.22:0/2378285752 conn(0x55ff46b7fb30 msgr2=0x55ff46bb39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc153e00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 0x7fc1380a4720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ff46bba190 tx=0x7fc1440f6530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 -- 172.21.15.22:0/2378285752 shutdown_connections 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.706+0000 7fc15de35740 1 -- 172.21.15.22:0/2378285752 wait complete. 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 -- start start 2024-03-24T06:49:24.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 Processor -- start 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb8f60 0x55ff46bbf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 0x55ff46bbfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff46ac4f20 con 0x55ff46bb9c40 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff46ac4bc0 con 0x55ff46bc22d0 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff4688e1c0 con 0x55ff46bb8f60 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.707+0000 7fc15de35740 10 client.0 _ll_register_callbacks cb 0x55ff46830be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc159000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb8f60 0x55ff46bbf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56890/0 (socket says 172.21.15.22:56890) 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 -- 172.21.15.22:0/20022704 learned_addr learned my addr 172.21.15.22:0/20022704 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc159000640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 0x55ff46bbfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 -- 172.21.15.22:0/20022704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb8f60 msgr2=0x55ff46bbf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb8f60 0x55ff46bbf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 -- 172.21.15.22:0/20022704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 msgr2=0x55ff46bbfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 0x55ff46bbfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 -- 172.21.15.22:0/20022704 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14c00e540 con 0x55ff46bb9c40 2024-03-24T06:49:24.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc159000640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 0x55ff46bbfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc153400640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc140012710 tx=0x7fc140012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e56c0bd29c15f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.708+0000 7fc13e000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140017070 con 0x55ff46bb9c40 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc14000f920 con 0x55ff46bb9c40 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ff46b18160 con 0x55ff46bb9c40 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc14000fae0 con 0x55ff46bb9c40 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc15de35740 10 client.17178 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ff46b1eb20 con 0x55ff46bb9c40 2024-03-24T06:49:24.714 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.709+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ff46b18160 con 0x55ff46bb9c40 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc14001c420 con 0x55ff46bb9c40 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 10 client.17178 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 20 client.17178 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46bd2930 0x55ff46bd4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 4 client.17178 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff4681f7e0 con 0x55ff46bd2930 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.710+0000 7fc159000640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46bd2930 0x55ff46bd4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.711+0000 7fc159000640 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46bd2930 0x55ff46bd4d50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f0d93a98cb12dc server_cookie=5ce4aa895d5f0e21 in_seq=0 out_seq=0 2024-03-24T06:49:24.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.711+0000 7fc13e000640 10 client.17178 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.711+0000 7fc13e000640 1 -- 172.21.15.22:0/20022704 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff4681f7e0 con 0x55ff46bd2930 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.711+0000 7fc13e000640 10 client.17178 handle_command_reply: tid=0 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 client.17178 shutdown 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 msgr2=0x55ff46bbf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc140012710 tx=0x7fc140012bd0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 shutdown_connections 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46bd2930 0x55ff46bd4d50 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bc22d0 0x55ff46bbfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb9c40 0x55ff46bbf7b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 --2- 172.21.15.22:0/20022704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb8f60 0x55ff46bbf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc13e000640 0 client.17178 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 >> 172.21.15.22:0/20022704 conn(0x55ff46b7fb30 msgr2=0x55ff46bb5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 shutdown_connections 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 1 -- 172.21.15.22:0/20022704 wait complete. 2024-03-24T06:49:24.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.712+0000 7fc15de35740 20 client.17178 trim_cache size 0 max 16384 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.731+0000 7fc15de35740 1 Processor -- start 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.731+0000 7fc15de35740 1 -- start start 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.731+0000 7fc15de35740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc15de35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb9c40 0x55ff46bba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc15de35740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bc22d0 0x55ff46bcdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc15de35740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff46bce160 con 0x55ff46bc22d0 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc15de35740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff46bc1d40 con 0x55ff46bb8f60 2024-03-24T06:49:24.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc15de35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff46bb81e0 con 0x55ff46bb9c40 2024-03-24T06:49:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35892/0 (socket says 172.21.15.22:35892) 2024-03-24T06:49:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 -- 172.21.15.22:0/420746039 learned_addr learned my addr 172.21.15.22:0/420746039 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc153400640 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb9c40 0x55ff46bba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bc22d0 0x55ff46bcdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 -- 172.21.15.22:0/420746039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb9c40 msgr2=0x55ff46bba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb9c40 0x55ff46bba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 -- 172.21.15.22:0/420746039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bc22d0 msgr2=0x55ff46bcdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bc22d0 0x55ff46bcdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.732+0000 7fc159000640 1 -- 172.21.15.22:0/420746039 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff46aba9a0 con 0x55ff46bb8f60 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc159000640 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc14c01b040 tx=0x7fc14c00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d9e334a63fb4940 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc148c00640 1 -- 172.21.15.22:0/420746039 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c005bd0 con 0x55ff46bb8f60 2024-03-24T06:49:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc148c00640 1 -- 172.21.15.22:0/420746039 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14c02e070 con 0x55ff46bb8f60 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc148c00640 1 -- 172.21.15.22:0/420746039 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c004b90 con 0x55ff46bb8f60 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 -- 172.21.15.22:0/420746039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 msgr2=0x55ff46bbe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc14c01b040 tx=0x7fc14c00ed50 comp rx=0 tx=0).stop 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 -- 172.21.15.22:0/420746039 shutdown_connections 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bc22d0 0x55ff46bcdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bb9c40 0x55ff46bba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 --2- 172.21.15.22:0/420746039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb8f60 0x55ff46bbe1d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.733+0000 7fc15de35740 1 -- 172.21.15.22:0/420746039 >> 172.21.15.22:0/420746039 conn(0x55ff46b7fb30 msgr2=0x55ff46bb2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.734+0000 7fc15de35740 1 -- 172.21.15.22:0/420746039 shutdown_connections 2024-03-24T06:49:24.735 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.734+0000 7fc15de35740 1 -- 172.21.15.22:0/420746039 wait complete. 2024-03-24T06:49:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.734+0000 7fc15de35740 1 -- start start 2024-03-24T06:49:24.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.734+0000 7fc15de35740 1 Processor -- start 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55ff46bc7e60 con 0x55ff46bb8f60 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ff46bc7fd0 con 0x55ff46bb9c40 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ff46c38c30 con 0x55ff46bc22d0 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc15de35740 10 client.0 _ll_register_callbacks cb 0x7fc140009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:24.739 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc159000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:35904/0 (socket says 172.21.15.22:35904) 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 -- 172.21.15.22:0/3302850959 learned_addr learned my addr 172.21.15.22:0/3302850959 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc159000640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 -- 172.21.15.22:0/3302850959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 msgr2=0x55ff46bc7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 -- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 msgr2=0x55ff46bca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc153400640 1 -- 172.21.15.22:0/3302850959 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff46ae1c20 con 0x55ff46bb9c40 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.735+0000 7fc159000640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc153400640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc140009900 tx=0x7fc14001c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31a081fdfaeb43a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc13e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140017040 con 0x55ff46bb9c40 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc14000fb20 con 0x55ff46bb9c40 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ff46bbadf0 con 0x55ff46bb9c40 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc14000fd00 con 0x55ff46bb9c40 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc15de35740 10 client.27545 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:24.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ff46ae0480 con 0x55ff46bb9c40 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.736+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ff46bbadf0 con 0x55ff46bb9c40 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc140003780 con 0x55ff46bb9c40 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 10 client.27545 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 10 client.27545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 20 client.27545 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46c3a3b0 0x55ff46c3c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 4 client.27545 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fc14c003e50 con 0x55ff46c3a3b0 2024-03-24T06:49:24.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.737+0000 7fc159000640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46c3a3b0 0x55ff46c3c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc159000640 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46c3a3b0 0x55ff46c3c7d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe6694a987e4c8d server_cookie=605396821e2fb2a1 in_seq=0 out_seq=0 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc13e000640 10 client.27545 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc13e000640 1 -- 172.21.15.22:0/3302850959 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fc14c003e50 con 0x55ff46c3a3b0 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc13e000640 10 client.27545 handle_command_reply: tid=0 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 client.27545 shutdown 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 msgr2=0x55ff46bc7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc140009900 tx=0x7fc14001c730 comp rx=0 tx=0).stop 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 shutdown_connections 2024-03-24T06:49:24.743 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ff46bc22d0 0x55ff46bc7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55ff46c3a3b0 0x55ff46c3c7d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ff46bb9c40 0x55ff46bc7220 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 --2- 172.21.15.22:0/3302850959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55ff46bb8f60 0x55ff46bca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc13e000640 0 client.27545 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 >> 172.21.15.22:0/3302850959 conn(0x55ff46b7fb30 msgr2=0x55ff46bc0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.738+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 shutdown_connections 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc15de35740 1 -- 172.21.15.22:0/3302850959 wait complete. 2024-03-24T06:49:24.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc15de35740 20 client.27545 trim_cache size 0 max 16384 2024-03-24T06:49:24.745 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:49:24.745 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:49:24.745 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:49:24.745 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:49:24.745 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 msgr2=0x7fc1380a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 0x7fc1380a4720 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ff46bba190 tx=0x7fc1440f6530 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 msgr2=0x7fc1541003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc14402c6c0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 shutdown_connections 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc1380a2260 0x7fc1380a4720 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc15410bb50 0x7fc154100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1541030a0 0x7fc154100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 --2- 172.21.15.22:0/832087730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc154102780 0x7fc1541003d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.746 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 >> 172.21.15.22:0/832087730 conn(0x7fc1540fbf60 msgr2=0x7fc1540fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.739+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 shutdown_connections 2024-03-24T06:49:24.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.740+0000 7fc159a00640 1 -- 172.21.15.22:0/832087730 wait complete. 2024-03-24T06:49:24.755 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:49:24.755 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:49:24.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 -- 172.21.15.22:0/2718875437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 msgr2=0x7f59dc073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 --2- 172.21.15.22:0/2718875437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc073ea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009900 tx=0x7f59cc02bcd0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 -- 172.21.15.22:0/2718875437 shutdown_connections 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 --2- 172.21.15.22:0/2718875437 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 --2- 172.21.15.22:0/2718875437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 0x7f59dc10da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 --2- 172.21.15.22:0/2718875437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc073ea0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 -- 172.21.15.22:0/2718875437 >> 172.21.15.22:0/2718875437 conn(0x7f59dc0fd1b0 msgr2=0x7f59dc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 -- 172.21.15.22:0/2718875437 shutdown_connections 2024-03-24T06:49:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.959+0000 7f59e3c00640 1 -- 172.21.15.22:0/2718875437 wait complete. 2024-03-24T06:49:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.960+0000 7f59e3c00640 1 Processor -- start 2024-03-24T06:49:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 -- start start 2024-03-24T06:49:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 0x7f59dc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc19edd0 con 0x7f59dc10df60 2024-03-24T06:49:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc19ef40 con 0x7f59dc0743e0 2024-03-24T06:49:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e3c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc19f100 con 0x7f59dc073ac0 2024-03-24T06:49:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e1200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44388/0 (socket says 172.21.15.22:44388) 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.961+0000 7f59e1200640 1 -- 172.21.15.22:0/629877380 learned_addr learned my addr 172.21.15.22:0/629877380 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59d3e00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 0x7f59dc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 -- 172.21.15.22:0/629877380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 msgr2=0x7f59dc19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 0x7f59dc19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 -- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 msgr2=0x7f59dc19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 -- 172.21.15.22:0/629877380 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc009560 con 0x7f59dc073ac0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.962+0000 7f59e1200640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f59cc000c00 tx=0x7f59cc02c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.963+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc02dbe0 con 0x7f59dc073ac0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.963+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59cc02dd40 con 0x7f59dc073ac0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.963+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc19f380 con 0x7f59dc073ac0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.963+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc003b20 con 0x7f59dc073ac0 2024-03-24T06:49:24.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.963+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1a6c60 con 0x7f59dc073ac0 2024-03-24T06:49:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.964+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59dc068950 con 0x7f59dc073ac0 2024-03-24T06:49:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.966+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f59cc003c80 con 0x7f59dc073ac0 2024-03-24T06:49:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.967+0000 7f59da000640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 0x7f59c00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.968+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f59cc0f0cb0 con 0x7f59dc073ac0 2024-03-24T06:49:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.968+0000 7f59d3e00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 0x7f59c00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.969+0000 7f59d3e00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 0x7f59c00a4770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0ff7d0 tx=0x7f59c4005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:24.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:24.974+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f59cc0b96d0 con 0x7f59dc073ac0 2024-03-24T06:49:25.271 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.272+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59dc19b260 con 0x7f59dc073ac0 2024-03-24T06:49:25.272 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.273+0000 7f59da000640 1 -- 172.21.15.22:0/629877380 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f59cc0bcd90 con 0x7f59dc073ac0 2024-03-24T06:49:25.273 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:49:25.273 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:49:25.273 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:49:25.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 msgr2=0x7f59c00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 0x7f59c00a4770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0ff7d0 tx=0x7f59c4005f90 comp rx=0 tx=0).stop 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 msgr2=0x7f59dc199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f59cc000c00 tx=0x7f59cc02c6d0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 shutdown_connections 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f59c00a22b0 0x7f59c00a4770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59dc10df60 0x7f59dc19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59dc0743e0 0x7f59dc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 --2- 172.21.15.22:0/629877380 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59dc073ac0 0x7f59dc199f10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 >> 172.21.15.22:0/629877380 conn(0x7f59dc0fd1b0 msgr2=0x7f59dc0fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 shutdown_connections 2024-03-24T06:49:25.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.277+0000 7f59e3c00640 1 -- 172.21.15.22:0/629877380 wait complete. 2024-03-24T06:49:25.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:49:25.558 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 -- 172.21.15.22:0/3420770684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc100630 msgr2=0x7f32fc100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 --2- 172.21.15.22:0/3420770684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc100630 0x7f32fc100ab0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009900 tx=0x7f32e802bc30 comp rx=0 tx=0).stop 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 -- 172.21.15.22:0/3420770684 shutdown_connections 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 --2- 172.21.15.22:0/3420770684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 --2- 172.21.15.22:0/3420770684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc100630 0x7f32fc100ab0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 --2- 172.21.15.22:0/3420770684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc106640 0x7f32fc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.559 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 -- 172.21.15.22:0/3420770684 >> 172.21.15.22:0/3420770684 conn(0x7f32fc077770 msgr2=0x7f32fc077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 -- 172.21.15.22:0/3420770684 shutdown_connections 2024-03-24T06:49:25.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.558+0000 7f3301a00640 1 -- 172.21.15.22:0/3420770684 wait complete. 2024-03-24T06:49:25.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.560+0000 7f3301a00640 1 Processor -- start 2024-03-24T06:49:25.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.560+0000 7f3301a00640 1 -- start start 2024-03-24T06:49:25.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.560+0000 7f3301a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.560+0000 7f3301a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc0fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.560+0000 7f3301a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc106640 0x7f32fc0fe030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f3301a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc1a61b0 con 0x7f32fc100630 2024-03-24T06:49:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f3301a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc1a6320 con 0x7f32fc106640 2024-03-24T06:49:25.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f3301a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f32fc1a64e0 con 0x7f32fc100ff0 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50742/0 (socket says 172.21.15.22:50742) 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 -- 172.21.15.22:0/3232101110 learned_addr learned my addr 172.21.15.22:0/3232101110 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32fa000640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc0fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32fb400640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc106640 0x7f32fc0fe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 -- 172.21.15.22:0/3232101110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 msgr2=0x7f32fc0fdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc0fdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 -- 172.21.15.22:0/3232101110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc106640 msgr2=0x7f32fc0fe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc106640 0x7f32fc0fe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32faa00640 1 -- 172.21.15.22:0/3232101110 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f4009840 con 0x7f32fc100630 2024-03-24T06:49:25.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.561+0000 7f32fa000640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc0fdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f32faa00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00e9f0 tx=0x7f32ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f32f3400640 1 -- 172.21.15.22:0/3232101110 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec009700 con 0x7f32fc100630 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f32f3400640 1 -- 172.21.15.22:0/3232101110 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec00ccc0 con 0x7f32fc100630 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f32f3400640 1 -- 172.21.15.22:0/3232101110 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec018820 con 0x7f32fc100630 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e8009560 con 0x7f32fc100630 2024-03-24T06:49:25.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.562+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc1a6b10 con 0x7f32fc100630 2024-03-24T06:49:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.565+0000 7f32f3400640 1 -- 172.21.15.22:0/3232101110 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f32ec010430 con 0x7f32fc100630 2024-03-24T06:49:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.565+0000 7f330482a740 1 Processor -- start 2024-03-24T06:49:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 -- start start 2024-03-24T06:49:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 0x55fa31434660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434ba0 0x55fa3143d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fa313a26e0 con 0x55fa31434ba0 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fa3139fc10 con 0x55fa31434280 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f330482a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fa3139b3f0 con 0x55fa3143d780 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32fb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 0x55fa31434660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32fa000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434ba0 0x55fa3143d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32fa000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44410/0 (socket says 172.21.15.22:44410) 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32fb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 0x55fa31434660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34250/0 (socket says 172.21.15.22:34250) 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.566+0000 7f32fa000640 1 -- 172.21.15.22:0/1398320780 learned_addr learned my addr 172.21.15.22:0/1398320780 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 -- 172.21.15.22:0/1398320780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 msgr2=0x55fa31434660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 0x55fa31434660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 -- 172.21.15.22:0/1398320780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434ba0 msgr2=0x55fa3143d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434ba0 0x55fa3143d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 -- 172.21.15.22:0/1398320780 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3135ce40 con 0x55fa3143d780 2024-03-24T06:49:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32fa000640 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009900 tx=0x7f32e802c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fc3bb87778c468e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.567+0000 7f32f3400640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 0x7f32e00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f32f3400640 1 -- 172.21.15.22:0/3232101110 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f32ec014070 con 0x7f32fc100630 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f32e7e00640 1 -- 172.21.15.22:0/1398320780 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e802dea0 con 0x55fa3143d780 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f32fb400640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 0x7f32e00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f32e7e00640 1 -- 172.21.15.22:0/1398320780 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e8004af0 con 0x55fa3143d780 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f32e7e00640 1 -- 172.21.15.22:0/1398320780 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e80052b0 con 0x55fa3143d780 2024-03-24T06:49:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f330482a740 1 -- 172.21.15.22:0/1398320780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 msgr2=0x55fa3143fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.568+0000 7f330482a740 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009900 tx=0x7f32e802c670 comp rx=0 tx=0).stop 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 -- 172.21.15.22:0/1398320780 shutdown_connections 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa3143fb70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434ba0 0x55fa3143d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 --2- 172.21.15.22:0/1398320780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434280 0x55fa31434660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 -- 172.21.15.22:0/1398320780 >> 172.21.15.22:0/1398320780 conn(0x55fa3142e9b0 msgr2=0x55fa3142ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f32fb400640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 0x7f32e00a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f32f4005fd0 tx=0x7f32f4009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 -- 172.21.15.22:0/1398320780 shutdown_connections 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 -- 172.21.15.22:0/1398320780 wait complete. 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.569+0000 7f330482a740 1 -- start start 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 Processor -- start 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa3143a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa3143d780 0x55fa3143b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fa31340060 con 0x55fa31434280 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fa3133fd00 con 0x55fa3143d780 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fa311091c0 con 0x55fa31434ba0 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f330482a740 10 client.0 _ll_register_callbacks cb 0x55fa310abbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa3143a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f32fa000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f32fb400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa3143d780 0x55fa3143b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f32fa000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44414/0 (socket says 172.21.15.22:44414) 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.570+0000 7f32fa000640 1 -- 172.21.15.22:0/1462524386 learned_addr learned my addr 172.21.15.22:0/1462524386 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 -- 172.21.15.22:0/1462524386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa3143d780 msgr2=0x55fa3143b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa3143d780 0x55fa3143b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 -- 172.21.15.22:0/1462524386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 msgr2=0x55fa3143a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa3143a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 -- 172.21.15.22:0/1462524386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32fc10de00 con 0x55fa31434ba0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32fa000640 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f32e802c240 tx=0x7f32e8004df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=301d6bd2f6aed8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8005600 con 0x55fa31434ba0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fa3109a7e0 con 0x55fa31434ba0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.571+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f32e804a450 con 0x55fa31434ba0 2024-03-24T06:49:25.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f32e804a610 con 0x55fa31434ba0 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 10 client.27589 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fa31399a70 con 0x55fa31434ba0 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fa3109a7e0 con 0x55fa31434ba0 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f32e803c050 con 0x55fa31434ba0 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 10 client.27589 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 10 client.27589 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:49:25.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 20 client.27589 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa3144d7f0 0x55fa3144fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 4 client.27589 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.572+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa310d7720 con 0x55fa3144d7f0 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.573+0000 7f32faa00640 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa3144d7f0 0x55fa3144fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.573+0000 7f32faa00640 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa3144d7f0 0x55fa3144fc10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=358a722cd272c23a server_cookie=5ee80037272dde7e in_seq=0 out_seq=0 2024-03-24T06:49:25.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.573+0000 7f32e4c00640 10 client.27589 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f32e4c00640 1 -- 172.21.15.22:0/1462524386 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa310d7720 con 0x55fa3144d7f0 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f32e4c00640 10 client.27589 handle_command_reply: tid=0 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 client.27589 shutdown 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 msgr2=0x55fa3143ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f32e802c240 tx=0x7f32e8004df0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 shutdown_connections 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa3144d7f0 0x55fa3144fc10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa3143d780 0x55fa3143b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434ba0 0x55fa3143ace0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 --2- 172.21.15.22:0/1462524386 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa3143a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f32e4c00640 0 client.27589 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.574+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 >> 172.21.15.22:0/1462524386 conn(0x55fa3142e9b0 msgr2=0x55fa31430700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.588 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.575+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 shutdown_connections 2024-03-24T06:49:25.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.575+0000 7f330482a740 1 -- 172.21.15.22:0/1462524386 wait complete. 2024-03-24T06:49:25.589 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.575+0000 7f330482a740 20 client.27589 trim_cache size 0 max 16384 2024-03-24T06:49:25.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.589+0000 7f330482a740 1 Processor -- start 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 -- start start 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa3143d780 0x55fa31455500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fa31435620 con 0x55fa3143d780 2024-03-24T06:49:25.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fa3142f9b0 con 0x55fa31434ba0 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f330482a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fa314363b0 con 0x55fa31434280 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32faa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32fa000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32fb400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa3143d780 0x55fa31455500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32faa00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44422/0 (socket says 172.21.15.22:44422) 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32fa000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34258/0 (socket says 172.21.15.22:34258) 2024-03-24T06:49:25.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32faa00640 1 -- 172.21.15.22:0/3827716959 learned_addr learned my addr 172.21.15.22:0/3827716959 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.590+0000 7f32faa00640 1 -- 172.21.15.22:0/3827716959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 msgr2=0x55fa3143b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32faa00640 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32faa00640 1 -- 172.21.15.22:0/3827716959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa3143d780 msgr2=0x55fa31455500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32faa00640 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa3143d780 0x55fa31455500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32faa00640 1 -- 172.21.15.22:0/3827716959 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa31335b00 con 0x55fa31434280 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32fa000640 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32faa00640 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00cf50 tx=0x7f32ec0d3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a3186e72a5b3b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32e7400640 1 -- 172.21.15.22:0/3827716959 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec010a10 con 0x55fa31434280 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32e7400640 1 -- 172.21.15.22:0/3827716959 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec09b580 con 0x55fa31434280 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.591+0000 7f32e7400640 1 -- 172.21.15.22:0/3827716959 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec0992a0 con 0x55fa31434280 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- 172.21.15.22:0/3827716959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 msgr2=0x55fa3143ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00cf50 tx=0x7f32ec0d3040 comp rx=0 tx=0).stop 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- 172.21.15.22:0/3827716959 shutdown_connections 2024-03-24T06:49:25.595 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa3143d780 0x55fa31455500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa3143b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 --2- 172.21.15.22:0/3827716959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa31434280 0x55fa3143ced0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- 172.21.15.22:0/3827716959 >> 172.21.15.22:0/3827716959 conn(0x55fa3142e9b0 msgr2=0x55fa3142fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- 172.21.15.22:0/3827716959 shutdown_connections 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- 172.21.15.22:0/3827716959 wait complete. 2024-03-24T06:49:25.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 -- start start 2024-03-24T06:49:25.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.592+0000 7f330482a740 1 Processor -- start 2024-03-24T06:49:25.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa314425e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa31442b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55fa31443060 con 0x55fa31434280 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fa314431d0 con 0x55fa31434ba0 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fa31443390 con 0x55fa3143d780 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f330482a740 10 client.0 _ll_register_callbacks cb 0x7f32ec0d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32fb400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa31442b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50768/0 (socket says 172.21.15.22:50768) 2024-03-24T06:49:25.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 -- 172.21.15.22:0/3265323133 learned_addr learned my addr 172.21.15.22:0/3265323133 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32fa000640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa314425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 -- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 msgr2=0x55fa31442b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa31442b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.593+0000 7f32faa00640 1 -- 172.21.15.22:0/3265323133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 msgr2=0x55fa314425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32faa00640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa314425e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32faa00640 1 -- 172.21.15.22:0/3265323133 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa31399a70 con 0x55fa31434280 2024-03-24T06:49:25.602 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32fa000640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa314425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32faa00640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0d3010 tx=0x7f32ec010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae0c6d54ab91770 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec099280 con 0x55fa31434280 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f32ec010a10 con 0x55fa31434280 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e8004af0 con 0x55fa31434280 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f32ec09ae80 con 0x55fa31434280 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f330482a740 10 client.17208 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:25.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.594+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x7f32fc1a6b10 con 0x55fa31434280 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e8004af0 con 0x55fa31434280 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f32ec0ce440 con 0x55fa31434280 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 10 client.17208 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 10 client.17208 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 20 client.17208 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa314b4cb0 0x55fa314b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 4 client.17208 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa31126890 con 0x55fa314b4cb0 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.595+0000 7f32fa000640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa314b4cb0 0x55fa314b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f32fa000640 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa314b4cb0 0x55fa314b70d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=716a528a30fa3678 server_cookie=4d8ad4af74f89f5e in_seq=0 out_seq=0 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f32e4c00640 10 client.17208 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f32e4c00640 1 -- 172.21.15.22:0/3265323133 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa31126890 con 0x55fa314b4cb0 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f32e4c00640 10 client.17208 handle_command_reply: tid=0 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 client.17208 shutdown 2024-03-24T06:49:25.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 msgr2=0x55fa31445610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0d3010 tx=0x7f32ec010e10 comp rx=0 tx=0).stop 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 shutdown_connections 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x55fa314b4cb0 0x55fa314b70d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fa3143d780 0x55fa31442b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fa31434ba0 0x55fa314425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 --2- 172.21.15.22:0/3265323133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55fa31434280 0x55fa31445610 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f32e4c00640 0 client.17208 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 >> 172.21.15.22:0/3265323133 conn(0x55fa3142e9b0 msgr2=0x55fa31430400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 shutdown_connections 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 1 -- 172.21.15.22:0/3265323133 wait complete. 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.596+0000 7f330482a740 20 client.17208 trim_cache size 0 max 16384 2024-03-24T06:49:25.607 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:25.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 msgr2=0x7f32e00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 0x7f32e00a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f32f4005fd0 tx=0x7f32f4009500 comp rx=0 tx=0).stop 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 msgr2=0x7f32fc0fd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f32ec00e9f0 tx=0x7f32ec00edf0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 shutdown_connections 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f32e00a2230 0x7f32e00a46f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32fc106640 0x7f32fc0fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32fc100ff0 0x7f32fc0fdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 --2- 172.21.15.22:0/3232101110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32fc100630 0x7f32fc0fd5b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 >> 172.21.15.22:0/3232101110 conn(0x7f32fc077770 msgr2=0x7f32fc06a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 shutdown_connections 2024-03-24T06:49:25.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.597+0000 7f3301a00640 1 -- 172.21.15.22:0/3232101110 wait complete. 2024-03-24T06:49:25.621 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.838+0000 7fc9c8a00640 1 -- 172.21.15.22:0/2473737417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41098c0 msgr2=0x7fc9c4109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.838+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/2473737417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41098c0 0x7fc9c4109ca0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009930 tx=0x7fc9b402bc30 comp rx=0 tx=0).stop 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.838+0000 7fc9c8a00640 1 -- 172.21.15.22:0/2473737417 shutdown_connections 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.838+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/2473737417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c4104270 0x7fc9c4110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.838+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/2473737417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c41038b0 0x7fc9c4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.839+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/2473737417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41098c0 0x7fc9c4109ca0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.839+0000 7fc9c8a00640 1 -- 172.21.15.22:0/2473737417 >> 172.21.15.22:0/2473737417 conn(0x7fc9c40ff360 msgr2=0x7fc9c4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.839+0000 7fc9c8a00640 1 -- 172.21.15.22:0/2473737417 shutdown_connections 2024-03-24T06:49:25.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.839+0000 7fc9c8a00640 1 -- 172.21.15.22:0/2473737417 wait complete. 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 Processor -- start 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 -- start start 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c41098c0 0x7fc9c419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c419ee70 con 0x7fc9c4104270 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c419efe0 con 0x7fc9c41098c0 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c419f1a0 con 0x7fc9c41038b0 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44440/0 (socket says 172.21.15.22:44440) 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c1400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50794/0 (socket says 172.21.15.22:50794) 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c1e00640 1 -- 172.21.15.22:0/997640014 learned_addr learned my addr 172.21.15.22:0/997640014 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.840+0000 7fc9c2800640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c41098c0 0x7fc9c419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 -- 172.21.15.22:0/997640014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c41098c0 msgr2=0x7fc9c419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c41098c0 0x7fc9c419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 -- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 msgr2=0x7fc9c419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 -- 172.21.15.22:0/997640014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b4009560 con 0x7fc9c41038b0 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1400640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.841+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000c00 tx=0x7fc9b402c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.842+0000 7fc9baa00640 1 -- 172.21.15.22:0/997640014 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40042d0 con 0x7fc9c41038b0 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.842+0000 7fc9baa00640 1 -- 172.21.15.22:0/997640014 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b4004430 con 0x7fc9c41038b0 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.842+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c419f420 con 0x7fc9c41038b0 2024-03-24T06:49:25.843 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.842+0000 7fc9baa00640 1 -- 172.21.15.22:0/997640014 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4003c50 con 0x7fc9c41038b0 2024-03-24T06:49:25.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.842+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c41a6d00 con 0x7fc9c41038b0 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.845+0000 7fc9baa00640 1 -- 172.21.15.22:0/997640014 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc9b4004df0 con 0x7fc9c41038b0 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.845+0000 7fc9cb5ff740 1 Processor -- start 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.845+0000 7fc9cb5ff740 1 -- start start 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.845+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.845+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d21240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d23b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559303c866e0 con 0x559303d18280 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559303c83c10 con 0x559303d21780 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559303c7f3f0 con 0x559303d18ba0 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c2800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c2800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50804/0 (socket says 172.21.15.22:50804) 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c2800640 1 -- 172.21.15.22:0/463886329 learned_addr learned my addr 172.21.15.22:0/463886329 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d23b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d21240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 -- 172.21.15.22:0/463886329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 msgr2=0x559303d21240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d21240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 -- 172.21.15.22:0/463886329 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 msgr2=0x559303d18660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c1400640 1 -- 172.21.15.22:0/463886329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559303c40e40 con 0x559303d21780 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9c2800640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.846+0000 7fc9baa00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 0x7fc9a80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9c1400640 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d23b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009fa0 tx=0x7fc9b00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58d77b51f29a4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9baa00640 1 -- 172.21.15.22:0/997640014 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc9b40f09d0 con 0x7fc9c41038b0 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9c2800640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 0x7fc9a80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9af000640 1 -- 172.21.15.22:0/463886329 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0004030 con 0x559303d21780 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9af000640 1 -- 172.21.15.22:0/463886329 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b0004190 con 0x559303d21780 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.847+0000 7fc9af000640 1 -- 172.21.15.22:0/463886329 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000f800 con 0x559303d21780 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/463886329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 msgr2=0x559303d23b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d23b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc9b0009fa0 tx=0x7fc9b00118d0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9c2800640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 0x7fc9a80a4770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009fd0 tx=0x7fc9bc009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/463886329 shutdown_connections 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d23b70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d21240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/463886329 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d18660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/463886329 >> 172.21.15.22:0/463886329 conn(0x559303d129b0 msgr2=0x559303d12d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/463886329 shutdown_connections 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/463886329 wait complete. 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.848+0000 7fc9cb5ff740 1 -- start start 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 Processor -- start 2024-03-24T06:49:25.851 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 0x559303d1ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559303c24060 con 0x559303d18280 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559303c23d00 con 0x559303d18ba0 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5593039ed1c0 con 0x559303d21780 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9cb5ff740 10 client.0 _ll_register_callbacks cb 0x55930398fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.849+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44458/0 (socket says 172.21.15.22:44458) 2024-03-24T06:49:25.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 -- 172.21.15.22:0/1662705639 learned_addr learned my addr 172.21.15.22:0/1662705639 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50812/0 (socket says 172.21.15.22:50812) 2024-03-24T06:49:25.853 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c1400640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 0x559303d1ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 -- 172.21.15.22:0/1662705639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 msgr2=0x559303d1ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 0x559303d1ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.855 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 -- 172.21.15.22:0/1662705639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 msgr2=0x559303d1e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 -- 172.21.15.22:0/1662705639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b0009c00 con 0x559303d21780 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c1400640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 0x559303d1ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00fae0 tx=0x7fc9bc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=624eb40230debba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.850+0000 7fc9a3400640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc022070 con 0x559303d21780 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55930397e7e0 con 0x559303d21780 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc9bc003970 con 0x559303d21780 2024-03-24T06:49:25.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc9bc016070 con 0x559303d21780 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9cb5ff740 10 client.27607 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559303c7da70 con 0x559303d21780 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.851+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55930397e7e0 con 0x559303d21780 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc9bc01d5e0 con 0x559303d21780 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 10 client.27607 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 10 client.27607 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 20 client.27607 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d31d10 0x559303d34130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 4 client.27607 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593039bb720 con 0x559303d31d10 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.852+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d31d10 0x559303d34130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.853+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d31d10 0x559303d34130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce071fb3334b2690 server_cookie=c8d7576dbfffaca7 in_seq=0 out_seq=0 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.853+0000 7fc9a3400640 10 client.27607 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.853+0000 7fc9a3400640 1 -- 172.21.15.22:0/1662705639 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593039bb720 con 0x559303d31d10 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.853+0000 7fc9a3400640 10 client.27607 handle_command_reply: tid=0 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.853+0000 7fc9cb5ff740 1 client.27607 shutdown 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 msgr2=0x559303d1f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.858 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00fae0 tx=0x7fc9bc004ab0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 shutdown_connections 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d31d10 0x559303d34130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d21780 0x559303d1f210 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18ba0 0x559303d1ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9a3400640 0 client.27607 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1662705639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18280 0x559303d1e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 >> 172.21.15.22:0/1662705639 conn(0x559303d129b0 msgr2=0x559303d14700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 shutdown_connections 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1662705639 wait complete. 2024-03-24T06:49:25.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.854+0000 7fc9cb5ff740 20 client.27607 trim_cache size 0 max 16384 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 Processor -- start 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 -- start start 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d2d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d21780 0x559303d118f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559303d19620 con 0x559303d21780 2024-03-24T06:49:25.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559303d24450 con 0x559303d18280 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.869+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559303d0e2e0 con 0x559303d18ba0 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34302/0 (socket says 172.21.15.22:34302) 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 -- 172.21.15.22:0/1584872512 learned_addr learned my addr 172.21.15.22:0/1584872512 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1400640 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d2d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.870 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d21780 0x559303d118f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 -- 172.21.15.22:0/1584872512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 msgr2=0x559303d2d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d2d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 -- 172.21.15.22:0/1584872512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d21780 msgr2=0x559303d118f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d21780 0x559303d118f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 -- 172.21.15.22:0/1584872512 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559303c19b00 con 0x559303d18280 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.870+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40b9a90 tx=0x7fc9b40f2de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52000f293bc7f935 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9ae600640 1 -- 172.21.15.22:0/1584872512 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40bcc00 con 0x559303d18280 2024-03-24T06:49:25.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9ae600640 1 -- 172.21.15.22:0/1584872512 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b40bca20 con 0x559303d18280 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9ae600640 1 -- 172.21.15.22:0/1584872512 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40bb1f0 con 0x559303d18280 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1584872512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 msgr2=0x559303d20ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40b9a90 tx=0x7fc9b40f2de0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1584872512 shutdown_connections 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d21780 0x559303d118f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18ba0 0x559303d2d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1584872512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d18280 0x559303d20ec0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1584872512 >> 172.21.15.22:0/1584872512 conn(0x559303d129b0 msgr2=0x559303d111e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.874 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1584872512 shutdown_connections 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.871+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1584872512 wait complete. 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 -- start start 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 Processor -- start 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18ba0 0x559303d24d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d25280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x559303d25980 con 0x559303d18ba0 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559303d25af0 con 0x559303d21780 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559303d97070 con 0x559303d18280 2024-03-24T06:49:25.875 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.872+0000 7fc9cb5ff740 10 client.0 _ll_register_callbacks cb 0x7fc9bc011de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d25280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44472/0 (socket says 172.21.15.22:44472) 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c1e00640 1 -- 172.21.15.22:0/1147142247 learned_addr learned my addr 172.21.15.22:0/1147142247 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c1400640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18ba0 0x559303d24d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 -- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 msgr2=0x559303d299b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 -- 172.21.15.22:0/1147142247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18ba0 msgr2=0x559303d24d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:25.876 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18ba0 0x559303d24d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 -- 172.21.15.22:0/1147142247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c41a6d00 con 0x559303d21780 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9c2800640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d25280 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009fa0 tx=0x7fc9bc01d5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b3175256b65865 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.873+0000 7fc9a3400640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc022070 con 0x559303d21780 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc9bc01d250 con 0x559303d21780 2024-03-24T06:49:25.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9b0004190 con 0x559303d21780 2024-03-24T06:49:25.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc9bc016040 con 0x559303d21780 2024-03-24T06:49:25.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9cb5ff740 10 client.27587 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:25.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559303c3f6a0 con 0x559303d21780 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b0004190 con 0x559303d21780 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.874+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc9bc004530 con 0x559303d21780 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 10 client.27587 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 10 client.27587 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 20 client.27587 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d98860 0x559303d9ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 4 client.27587 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559303a0a890 con 0x559303d98860 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d98860 0x559303d9ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:25.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9c1e00640 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d98860 0x559303d9ac80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4bf04c73ea96ca06 server_cookie=fec82a721398b414 in_seq=0 out_seq=0 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.875+0000 7fc9a3400640 10 client.27587 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9a3400640 1 -- 172.21.15.22:0/1147142247 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559303a0a890 con 0x559303d98860 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9a3400640 10 client.27587 handle_command_reply: tid=0 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 client.27587 shutdown 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 msgr2=0x559303d25280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d25280 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009fa0 tx=0x7fc9bc01d5e0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 shutdown_connections 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x559303d98860 0x559303d9ac80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559303d21780 0x559303d25280 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559303d18ba0 0x559303d24d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9a3400640 0 client.27587 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 --2- 172.21.15.22:0/1147142247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559303d18280 0x559303d299b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 >> 172.21.15.22:0/1147142247 conn(0x559303d129b0 msgr2=0x559303d11a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 shutdown_connections 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 1 -- 172.21.15.22:0/1147142247 wait complete. 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.876+0000 7fc9cb5ff740 20 client.27587 trim_cache size 0 max 16384 2024-03-24T06:49:25.882 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:25.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 msgr2=0x7fc9a80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 0x7fc9a80a4770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009fd0 tx=0x7fc9bc009500 comp rx=0 tx=0).stop 2024-03-24T06:49:25.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 msgr2=0x7fc9c419a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000c00 tx=0x7fc9b402c5f0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 shutdown_connections 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc9a80a22b0 0x7fc9a80a4770 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9c41098c0 0x7fc9c419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9c4104270 0x7fc9c419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 --2- 172.21.15.22:0/997640014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c41038b0 0x7fc9c419a000 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 >> 172.21.15.22:0/997640014 conn(0x7fc9c40ff360 msgr2=0x7fc9c40ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 shutdown_connections 2024-03-24T06:49:25.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:25.877+0000 7fc9c8a00640 1 -- 172.21.15.22:0/997640014 wait complete. 2024-03-24T06:49:25.897 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 -- 172.21.15.22:0/197757999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4106550 msgr2=0x7f03a4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 --2- 172.21.15.22:0/197757999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4106550 0x7f03a4106930 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0394009930 tx=0x7f039402bc30 comp rx=0 tx=0).stop 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 -- 172.21.15.22:0/197757999 shutdown_connections 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 --2- 172.21.15.22:0/197757999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100e50 0x7f03a410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 --2- 172.21.15.22:0/197757999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4100490 0x7f03a4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 --2- 172.21.15.22:0/197757999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4106550 0x7f03a4106930 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.126 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 -- 172.21.15.22:0/197757999 >> 172.21.15.22:0/197757999 conn(0x7f03a40fbf60 msgr2=0x7f03a40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 -- 172.21.15.22:0/197757999 shutdown_connections 2024-03-24T06:49:26.127 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.126+0000 7f03a3200640 1 -- 172.21.15.22:0/197757999 wait complete. 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.127+0000 7f03a3200640 1 Processor -- start 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 -- start start 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 0x7f03a419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4106550 0x7f03a419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f03a419eca0 con 0x7f03a4106550 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f03a419ee10 con 0x7f03a4100490 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03a419efd0 con 0x7f03a4100e50 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 0x7f03a419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34330/0 (socket says 172.21.15.22:34330) 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a1e00640 1 -- 172.21.15.22:0/1660033745 learned_addr learned my addr 172.21.15.22:0/1660033745 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.128+0000 7f03a2800640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4106550 0x7f03a419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 -- 172.21.15.22:0/1660033745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 msgr2=0x7f03a419a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 0x7f03a419a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 -- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4106550 msgr2=0x7f03a419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4106550 0x7f03a419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 -- 172.21.15.22:0/1660033745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0394009560 con 0x7f03a4100490 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1400640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 0x7f03a419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f03a1e00640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0394000c00 tx=0x7f039402c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.129+0000 7f039aa00640 1 -- 172.21.15.22:0/1660033745 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03940042d0 con 0x7f03a4100490 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.130+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a419f250 con 0x7f03a4100490 2024-03-24T06:49:26.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.130+0000 7f039aa00640 1 -- 172.21.15.22:0/1660033745 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0394004430 con 0x7f03a4100490 2024-03-24T06:49:26.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.130+0000 7f039aa00640 1 -- 172.21.15.22:0/1660033745 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394003c50 con 0x7f03a4100490 2024-03-24T06:49:26.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.130+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41a6b30 con 0x7f03a4100490 2024-03-24T06:49:26.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.133+0000 7f039aa00640 1 -- 172.21.15.22:0/1660033745 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0394004df0 con 0x7f03a4100490 2024-03-24T06:49:26.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 Processor -- start 2024-03-24T06:49:26.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 -- start start 2024-03-24T06:49:26.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f039aa00640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 0x7f03880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f039aa00640 1 -- 172.21.15.22:0/1660033745 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f03940f0a20 con 0x7f03a4100490 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f29830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f32410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f34d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a250e976e0 con 0x55a250f29450 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a250e94c10 con 0x55a250f29d70 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.134+0000 7f03abdcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a250e903f0 con 0x55a250f32950 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a2800640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 0x7f03880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f34d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f29830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f34d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44506/0 (socket says 172.21.15.22:44506) 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a2800640 1 -- 172.21.15.22:0/3272496184 learned_addr learned my addr 172.21.15.22:0/3272496184 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1400640 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f32410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 -- 172.21.15.22:0/3272496184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 msgr2=0x55a250f34d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f34d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 -- 172.21.15.22:0/3272496184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 msgr2=0x55a250f32410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f32410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.135+0000 7f03a1e00640 1 -- 172.21.15.22:0/3272496184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a250e51e40 con 0x55a250f29450 2024-03-24T06:49:26.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.136+0000 7f03a1e00640 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f29830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03940ede70 tx=0x7f03940bca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853263e349b05cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.136+0000 7f03a2800640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 0x7f03880a46f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f039c013c90 tx=0x7f039c009e20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.136+0000 7f038f400640 1 -- 172.21.15.22:0/3272496184 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03940c1020 con 0x55a250f29450 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.136+0000 7f038f400640 1 -- 172.21.15.22:0/3272496184 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03940c1260 con 0x55a250f29450 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.136+0000 7f038f400640 1 -- 172.21.15.22:0/3272496184 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03940b9780 con 0x55a250f29450 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- 172.21.15.22:0/3272496184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 msgr2=0x55a250f29830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f29830 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f03940ede70 tx=0x7f03940bca70 comp rx=0 tx=0).stop 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- 172.21.15.22:0/3272496184 shutdown_connections 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f34d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f32410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3272496184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f29830 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- 172.21.15.22:0/3272496184 >> 172.21.15.22:0/3272496184 conn(0x55a250f23b80 msgr2=0x55a250f23f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- 172.21.15.22:0/3272496184 shutdown_connections 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- 172.21.15.22:0/3272496184 wait complete. 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.137+0000 7f03abdcb740 1 -- start start 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 Processor -- start 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f2f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f30430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a250e35060 con 0x55a250f29450 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a250e34d00 con 0x55a250f29d70 2024-03-24T06:49:26.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a250bfe1c0 con 0x55a250f32950 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03abdcb740 10 client.0 _ll_register_callbacks cb 0x55a250ba0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03a1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f2f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03a1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34342/0 (socket says 172.21.15.22:34342) 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03a1400640 1 -- 172.21.15.22:0/3110617540 learned_addr learned my addr 172.21.15.22:0/3110617540 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.138+0000 7f03a2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f30430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 -- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 msgr2=0x55a250f30430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f30430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 -- 172.21.15.22:0/3110617540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 msgr2=0x55a250f2f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f2f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 -- 172.21.15.22:0/3110617540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a41a6b30 con 0x55a250f29d70 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a2800640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f30430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03a1400640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0390012d30 tx=0x7f039000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a93995a762569a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f0383e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0390017070 con 0x55a250f29d70 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f0390012110 con 0x55a250f29d70 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.139+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a250b8f7e0 con 0x55a250f29d70 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0390004050 con 0x55a250f29d70 2024-03-24T06:49:26.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 10 client.27599 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a250e8ea70 con 0x55a250f29d70 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a250b8f7e0 con 0x55a250f29d70 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f0390010430 con 0x55a250f29d70 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 10 client.27599 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 10 client.27599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 20 client.27599 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.140+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250f42bf0 0x55a250f45030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.141+0000 7f03abdcb740 4 client.27599 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:26.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.141+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a250bcc720 con 0x55a250f42bf0 2024-03-24T06:49:26.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.141+0000 7f03a1e00640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250f42bf0 0x55a250f45030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.141+0000 7f03a1e00640 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250f42bf0 0x55a250f45030 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23017cf4615fa1bc server_cookie=38fc2e430492661e in_seq=0 out_seq=0 2024-03-24T06:49:26.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.141+0000 7f0383e00640 10 client.27599 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f0383e00640 1 -- 172.21.15.22:0/3110617540 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a250bcc720 con 0x55a250f42bf0 2024-03-24T06:49:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f0383e00640 10 client.27599 handle_command_reply: tid=0 2024-03-24T06:49:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 client.27599 shutdown 2024-03-24T06:49:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 msgr2=0x55a250f2fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0390012d30 tx=0x7f039000f790 comp rx=0 tx=0).stop 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 shutdown_connections 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250f42bf0 0x55a250f45030 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250f30430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29d70 0x55a250f2fef0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f0383e00640 0 client.27599 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 --2- 172.21.15.22:0/3110617540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f2f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.142+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 >> 172.21.15.22:0/3110617540 conn(0x55a250f23b80 msgr2=0x55a250f258d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.143+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 shutdown_connections 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.143+0000 7f03abdcb740 1 -- 172.21.15.22:0/3110617540 wait complete. 2024-03-24T06:49:26.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.143+0000 7f03abdcb740 20 client.27599 trim_cache size 0 max 16384 2024-03-24T06:49:26.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.162+0000 7f03abdcb740 1 Processor -- start 2024-03-24T06:49:26.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.162+0000 7f03abdcb740 1 -- start start 2024-03-24T06:49:26.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f29d70 0x55a250f3dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a250f239f0 con 0x55a250f29450 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a250f24b80 con 0x55a250f32950 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03abdcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a250f28e90 con 0x55a250f29d70 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03a2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03a2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34356/0 (socket says 172.21.15.22:34356) 2024-03-24T06:49:26.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03a2800640 1 -- 172.21.15.22:0/2058893826 learned_addr learned my addr 172.21.15.22:0/2058893826 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.164 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.163+0000 7f03a1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f29d70 0x55a250f3dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 -- 172.21.15.22:0/2058893826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f29d70 msgr2=0x55a250f3dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f29d70 0x55a250f3dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 -- 172.21.15.22:0/2058893826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 msgr2=0x55a250f320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 -- 172.21.15.22:0/2058893826 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a250e2ab00 con 0x55a250f32950 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f03a2800640 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f039c013710 tx=0x7f039c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f6afbb5f3d1e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f038ea00640 1 -- 172.21.15.22:0/2058893826 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c003d40 con 0x55a250f32950 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f038ea00640 1 -- 172.21.15.22:0/2058893826 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c013170 con 0x55a250f32950 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.164+0000 7f038ea00640 1 -- 172.21.15.22:0/2058893826 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c02acd0 con 0x55a250f32950 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 -- 172.21.15.22:0/2058893826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 msgr2=0x55a250f4a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f039c013710 tx=0x7f039c009500 comp rx=0 tx=0).stop 2024-03-24T06:49:26.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 -- 172.21.15.22:0/2058893826 shutdown_connections 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f32950 0x55a250f4a8f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f29d70 0x55a250f3dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2058893826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29450 0x55a250f320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 -- 172.21.15.22:0/2058893826 >> 172.21.15.22:0/2058893826 conn(0x55a250f23b80 msgr2=0x55a250f22cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 -- 172.21.15.22:0/2058893826 shutdown_connections 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.165+0000 7f03abdcb740 1 -- 172.21.15.22:0/2058893826 wait complete. 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.166+0000 7f03abdcb740 1 -- start start 2024-03-24T06:49:26.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.166+0000 7f03abdcb740 1 Processor -- start 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.166+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 0x55a250f36fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250fa0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a250fa1590 con 0x55a250f29d70 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a250fa1700 con 0x55a250f29450 2024-03-24T06:49:26.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a250fa18c0 con 0x55a250f32950 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03abdcb740 10 client.0 _ll_register_callbacks cb 0x7f0390009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a2800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250fa0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34368/0 (socket says 172.21.15.22:34368) 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 -- 172.21.15.22:0/2305583036 learned_addr learned my addr 172.21.15.22:0/2305583036 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1400640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 0x55a250f36fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 -- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 msgr2=0x55a250fa0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250fa0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 -- 172.21.15.22:0/2305583036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 msgr2=0x55a250f36fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 0x55a250f36fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1e00640 1 -- 172.21.15.22:0/2305583036 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f039c01b040 con 0x55a250f29450 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a2800640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250fa0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.167+0000 7f03a1400640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 0x55a250f36fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03a1e00640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f039402c780 tx=0x7f0394009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a8e69460d0baa50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03940bd890 con 0x55a250f29450 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a250f2f7e0 con 0x55a250f29450 2024-03-24T06:49:26.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f03940d5e40 con 0x55a250f29450 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f03940c58b0 con 0x55a250f29450 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 10 client.27605 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a250e506a0 con 0x55a250f29450 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a250f2f7e0 con 0x55a250f29450 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f039403c070 con 0x55a250f29450 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 10 client.27605 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 10 client.27605 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 20 client.27605 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.168+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250fa9c00 0x55a250fabff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f03abdcb740 4 client.27605 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f03940c1260 con 0x55a250fa9c00 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f03a1400640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250fa9c00 0x55a250fabff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f03a1400640 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250fa9c00 0x55a250fabff0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52eddca030a901e9 server_cookie=be823552b2efb399 in_seq=0 out_seq=0 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f0383e00640 10 client.27605 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f0383e00640 1 -- 172.21.15.22:0/2305583036 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f03940c1260 con 0x55a250fa9c00 2024-03-24T06:49:26.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.169+0000 7f0383e00640 10 client.27605 handle_command_reply: tid=0 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 client.27605 shutdown 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 msgr2=0x55a250f36a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f039402c780 tx=0x7f0394009260 comp rx=0 tx=0).stop 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 shutdown_connections 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55a250fa9c00 0x55a250fabff0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a250f32950 0x55a250fa0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a250f29d70 0x55a250f36fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 --2- 172.21.15.22:0/2305583036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a250f29450 0x55a250f36a80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 >> 172.21.15.22:0/2305583036 conn(0x55a250f23b80 msgr2=0x55a250f22430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f0383e00640 0 client.27605 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:26.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 shutdown_connections 2024-03-24T06:49:26.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 1 -- 172.21.15.22:0/2305583036 wait complete. 2024-03-24T06:49:26.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.170+0000 7f03abdcb740 20 client.27605 trim_cache size 0 max 16384 2024-03-24T06:49:26.177 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 msgr2=0x7f03880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 0x7f03880a46f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f039c013c90 tx=0x7f039c009e20 comp rx=0 tx=0).stop 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 msgr2=0x7f03a4199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0394000c00 tx=0x7f039402c5f0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 shutdown_connections 2024-03-24T06:49:26.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f03880a2230 0x7f03880a46f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a4106550 0x7f03a419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a4100e50 0x7f03a419a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 --2- 172.21.15.22:0/1660033745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03a4100490 0x7f03a4199e30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 >> 172.21.15.22:0/1660033745 conn(0x7f03a40fbf60 msgr2=0x7f03a40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 shutdown_connections 2024-03-24T06:49:26.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.171+0000 7f03a3200640 1 -- 172.21.15.22:0/1660033745 wait complete. 2024-03-24T06:49:26.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 -- 172.21.15.22:0/565040161 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4106550 msgr2=0x7f8be4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 --2- 172.21.15.22:0/565040161 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4106550 0x7f8be4106930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009900 tx=0x7f8bd402bc40 comp rx=0 tx=0).stop 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 -- 172.21.15.22:0/565040161 shutdown_connections 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 --2- 172.21.15.22:0/565040161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100e50 0x7f8be410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 --2- 172.21.15.22:0/565040161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4100490 0x7f8be4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 --2- 172.21.15.22:0/565040161 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4106550 0x7f8be4106930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 -- 172.21.15.22:0/565040161 >> 172.21.15.22:0/565040161 conn(0x7f8be40fbf60 msgr2=0x7f8be40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.407+0000 7f8be8a00640 1 -- 172.21.15.22:0/565040161 shutdown_connections 2024-03-24T06:49:26.408 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.408+0000 7f8be8a00640 1 -- 172.21.15.22:0/565040161 wait complete. 2024-03-24T06:49:26.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 Processor -- start 2024-03-24T06:49:26.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 -- start start 2024-03-24T06:49:26.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4100e50 0x7f8be419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4106550 0x7f8be4197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f8be4197fc0 con 0x7f8be4100e50 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8be4198130 con 0x7f8be4100490 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be8a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8be41982f0 con 0x7f8be4106550 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4100e50 0x7f8be419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4106550 0x7f8be4197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.409+0000 7f8be3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34390/0 (socket says 172.21.15.22:34390) 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 -- 172.21.15.22:0/2897997459 learned_addr learned my addr 172.21.15.22:0/2897997459 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 -- 172.21.15.22:0/2897997459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4106550 msgr2=0x7f8be4197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4106550 0x7f8be4197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 -- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4100e50 msgr2=0x7f8be419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4100e50 0x7f8be419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 -- 172.21.15.22:0/2897997459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd4009560 con 0x7f8be4100490 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.410+0000 7f8be3400640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4000c00 tx=0x7f8bd402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.411+0000 7f8be0c00640 1 -- 172.21.15.22:0/2897997459 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4004410 con 0x7f8be4100490 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.411+0000 7f8be0c00640 1 -- 172.21.15.22:0/2897997459 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bd40052d0 con 0x7f8be4100490 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.411+0000 7f8be0c00640 1 -- 172.21.15.22:0/2897997459 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4003d40 con 0x7f8be4100490 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.411+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be40697f0 con 0x7f8be4100490 2024-03-24T06:49:26.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.411+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be41a4680 con 0x7f8be4100490 2024-03-24T06:49:26.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.414+0000 7f8be0c00640 1 -- 172.21.15.22:0/2897997459 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f8bd4005440 con 0x7f8be4100490 2024-03-24T06:49:26.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.414+0000 7f8becbec740 1 Processor -- start 2024-03-24T06:49:26.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.414+0000 7f8becbec740 1 -- start start 2024-03-24T06:49:26.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.414+0000 7f8becbec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.414+0000 7f8becbec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8becbec740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1cf950 0x55d4bf1d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8becbec740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf1346e0 con 0x55d4bf1cf950 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8becbec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf131c10 con 0x55d4bf1c6450 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8becbec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf12d3f0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34400/0 (socket says 172.21.15.22:34400) 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1cf950 0x55d4bf1d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 -- 172.21.15.22:0/3349869106 learned_addr learned my addr 172.21.15.22:0/3349869106 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 -- 172.21.15.22:0/3349869106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 msgr2=0x55d4bf1cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 -- 172.21.15.22:0/3349869106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1cf950 msgr2=0x55d4bf1d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1cf950 0x55d4bf1d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3e00640 1 -- 172.21.15.22:0/3349869106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4bf0eee40 con 0x55d4bf1c6450 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.415+0000 7f8be3400640 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8be3e00640 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc009fd0 tx=0x7f8bdc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c38ff63d34d3dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8be0c00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 0x7f8bc80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8bd3e00640 1 -- 172.21.15.22:0/3349869106 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc004d20 con 0x55d4bf1c6450 2024-03-24T06:49:26.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8bd3e00640 1 -- 172.21.15.22:0/3349869106 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bdc004e80 con 0x55d4bf1c6450 2024-03-24T06:49:26.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8bd3e00640 1 -- 172.21.15.22:0/3349869106 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc005280 con 0x55d4bf1c6450 2024-03-24T06:49:26.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8be0c00640 1 -- 172.21.15.22:0/2897997459 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f8bd40f0920 con 0x7f8be4100490 2024-03-24T06:49:26.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.416+0000 7f8be3400640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 0x7f8bc80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- 172.21.15.22:0/3349869106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 msgr2=0x55d4bf1c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc009fd0 tx=0x7f8bdc0118d0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- 172.21.15.22:0/3349869106 shutdown_connections 2024-03-24T06:49:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1cf950 0x55d4bf1d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 --2- 172.21.15.22:0/3349869106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1c6830 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- 172.21.15.22:0/3349869106 >> 172.21.15.22:0/3349869106 conn(0x55d4bf1c0b80 msgr2=0x55d4bf1c0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8be3400640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 0x7f8bc80a4770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55d4bf1c5f20 tx=0x7f8bd40f6530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- 172.21.15.22:0/3349869106 shutdown_connections 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- 172.21.15.22:0/3349869106 wait complete. 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.417+0000 7f8becbec740 1 -- start start 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 Processor -- start 2024-03-24T06:49:26.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1cf950 0x55d4bf1cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf0d2060 con 0x55d4bf1c6450 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf0d1d00 con 0x55d4bf1cf950 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d4bee9b1c0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8becbec740 10 client.0 _ll_register_callbacks cb 0x55d4bee3dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.418+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44576/0 (socket says 172.21.15.22:44576) 2024-03-24T06:49:26.423 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 -- 172.21.15.22:0/1849156645 learned_addr learned my addr 172.21.15.22:0/1849156645 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8be3e00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1cf950 0x55d4bf1cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 -- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1cf950 msgr2=0x55d4bf1cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1cf950 0x55d4bf1cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 -- 172.21.15.22:0/1849156645 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 msgr2=0x55d4bf1cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 -- 172.21.15.22:0/1849156645 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bdc009c80 con 0x55d4bf1c6d70 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8be3e00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc012700 tx=0x7f8bcc012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01f1930d2e5239b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:26.424 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.419+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc017070 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d4bee69720 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f8bcc00f920 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8bcc00fae0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8becbec740 10 client.27643 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d4bf12ba70 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.420+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d4bee69720 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8bcc010d10 con 0x55d4bf1c6d70 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 10 client.27643 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 10 client.27643 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:49:26.425 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 20 client.27643 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf1dfe80 0x55d4bf1e22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 4 client.27643 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4beeb8890 con 0x55d4bf1dfe80 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.421+0000 7f8be3e00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf1dfe80 0x55d4bf1e22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.422+0000 7f8bd0c00640 10 client.27643 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:49:26.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.422+0000 7f8be3e00640 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf1dfe80 0x55d4bf1e22a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d53fd04bcc0c8533 server_cookie=a372515abeaf8881 in_seq=0 out_seq=0 2024-03-24T06:49:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.422+0000 7f8bd0c00640 1 -- 172.21.15.22:0/1849156645 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d4beeb8890 con 0x55d4bf1dfe80 2024-03-24T06:49:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.422+0000 7f8bd0c00640 10 client.27643 handle_command_reply: tid=0 2024-03-24T06:49:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 client.27643 shutdown 2024-03-24T06:49:26.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 msgr2=0x55d4bf1ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc012700 tx=0x7f8bcc012bc0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 shutdown_connections 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf1dfe80 0x55d4bf1e22a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1cf950 0x55d4bf1cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1ccf50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 --2- 172.21.15.22:0/1849156645 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8bd0c00640 0 client.27643 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 >> 172.21.15.22:0/1849156645 conn(0x55d4bf1c0b80 msgr2=0x55d4bf1c28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 shutdown_connections 2024-03-24T06:49:26.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 1 -- 172.21.15.22:0/1849156645 wait complete. 2024-03-24T06:49:26.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.423+0000 7f8becbec740 20 client.27643 trim_cache size 0 max 16384 2024-03-24T06:49:26.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:26 smithi143 ceph-mon[36520]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:26.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:26 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/629877380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:26.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.438+0000 7f8becbec740 1 Processor -- start 2024-03-24T06:49:26.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.438+0000 7f8becbec740 1 -- start start 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf1db1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf1bc4b0 con 0x55d4bf1c6450 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf1c5bf0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8becbec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf1c7690 con 0x55d4bf1cf950 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34428/0 (socket says 172.21.15.22:34428) 2024-03-24T06:49:26.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2625627617 learned_addr learned my addr 172.21.15.22:0/2625627617 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8be3e00640 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf1db1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2625627617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 msgr2=0x55d4bf1db1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.439+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf1db1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2625627617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 msgr2=0x55d4bf1cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2625627617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4bf0c7b00 con 0x55d4bf1c6d70 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01c730 tx=0x7f8bcc00f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29a08ba33011c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bd3400640 1 -- 172.21.15.22:0/2625627617 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc017070 con 0x55d4bf1c6d70 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bd3400640 1 -- 172.21.15.22:0/2625627617 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bcc00fe00 con 0x55d4bf1c6d70 2024-03-24T06:49:26.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.440+0000 7f8bd3400640 1 -- 172.21.15.22:0/2625627617 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc0054b0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- 172.21.15.22:0/2625627617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 msgr2=0x55d4bf1c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01c730 tx=0x7f8bcc00f810 comp rx=0 tx=0).stop 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- 172.21.15.22:0/2625627617 shutdown_connections 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf1db1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1c7150 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 --2- 172.21.15.22:0/2625627617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6450 0x55d4bf1cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- 172.21.15.22:0/2625627617 >> 172.21.15.22:0/2625627617 conn(0x55d4bf1c0b80 msgr2=0x55d4bf1cc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- 172.21.15.22:0/2625627617 shutdown_connections 2024-03-24T06:49:26.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- 172.21.15.22:0/2625627617 wait complete. 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 -- start start 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.441+0000 7f8becbec740 1 Processor -- start 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf2401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf1d7f30 con 0x55d4bf1c6d70 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf2406f0 con 0x55d4bf1c6450 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d4bf2408b0 con 0x55d4bf1cf950 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8becbec740 10 client.0 _ll_register_callbacks cb 0x7f8bcc009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8bdbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50950/0 (socket says 172.21.15.22:50950) 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8be3e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34434/0 (socket says 172.21.15.22:34434) 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2091284804 learned_addr learned my addr 172.21.15.22:0/2091284804 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.442+0000 7f8be3e00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf2401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2091284804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 msgr2=0x55d4bf2401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf2401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 msgr2=0x55d4bf1ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 -- 172.21.15.22:0/2091284804 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4bf12ba70 con 0x55d4bf1c6d70 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bdbe00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01c540 tx=0x7f8bcc01cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa60744a3b678fff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.443+0000 7f8bd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc017040 con 0x55d4bf1c6d70 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f8bcc000e40 con 0x55d4bf1c6d70 2024-03-24T06:49:26.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d4bee2c7e0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8bcc01ccc0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8becbec740 10 client.17262 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d4bf0ed6a0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.444+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d4bee2c7e0 con 0x55d4bf1c6d70 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f8bcc003d00 con 0x55d4bf1c6d70 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 10 client.17262 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 10 client.17262 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 20 client.17262 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf24f850 0x55d4bf251c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 4 client.17262 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4bedd8700 con 0x55d4bf24f850 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8be3e00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf24f850 0x55d4bf251c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8be3e00640 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf24f850 0x55d4bf251c70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7168d2f96c46727 server_cookie=30a03e480e318dfd in_seq=0 out_seq=0 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.445+0000 7f8bd0c00640 10 client.17262 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8bd0c00640 1 -- 172.21.15.22:0/2091284804 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4bedd8700 con 0x55d4bf24f850 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8bd0c00640 10 client.17262 handle_command_reply: tid=0 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 client.17262 shutdown 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 msgr2=0x55d4bf1d7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01c540 tx=0x7f8bcc01cad0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 shutdown_connections 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d4bf1cf950 0x55d4bf2401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d4bf1c6d70 0x55d4bf1d7810 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55d4bf24f850 0x55d4bf251c70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 --2- 172.21.15.22:0/2091284804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d4bf1c6450 0x55d4bf1ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 >> 172.21.15.22:0/2091284804 conn(0x55d4bf1c0b80 msgr2=0x55d4bf1bfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8bd0c00640 0 client.17262 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 shutdown_connections 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 1 -- 172.21.15.22:0/2091284804 wait complete. 2024-03-24T06:49:26.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.446+0000 7f8becbec740 20 client.17262 trim_cache size 0 max 16384 2024-03-24T06:49:26.453 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:26.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 msgr2=0x7f8bc80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 0x7f8bc80a4770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55d4bf1c5f20 tx=0x7f8bd40f6530 comp rx=0 tx=0).stop 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 msgr2=0x7f8be419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4000c00 tx=0x7f8bd402c6c0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 shutdown_connections 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f8bc80a22b0 0x7f8bc80a4770 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8be4106550 0x7f8be4197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8be4100e50 0x7f8be419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.454 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 --2- 172.21.15.22:0/2897997459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be4100490 0x7f8be419d7c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 >> 172.21.15.22:0/2897997459 conn(0x7f8be40fbf60 msgr2=0x7f8be40fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 shutdown_connections 2024-03-24T06:49:26.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.447+0000 7f8be8a00640 1 -- 172.21.15.22:0/2897997459 wait complete. 2024-03-24T06:49:26.462 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:26 smithi022 ceph-mon[32003]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:26.463 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:26 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/629877380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:26.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:49:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:26 smithi155 ceph-mon[36262]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:49:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:26 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/629877380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.684+0000 7ff99cc00640 1 -- 172.21.15.22:0/3495116617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998106550 msgr2=0x7ff998106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.684+0000 7ff99cc00640 1 --2- 172.21.15.22:0/3495116617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998106550 0x7ff998106930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff988009cd0 tx=0x7ff98802c0e0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 -- 172.21.15.22:0/3495116617 shutdown_connections 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 --2- 172.21.15.22:0/3495116617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99810d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 --2- 172.21.15.22:0/3495116617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998100490 0x7ff998100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 --2- 172.21.15.22:0/3495116617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998106550 0x7ff998106930 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 -- 172.21.15.22:0/3495116617 >> 172.21.15.22:0/3495116617 conn(0x7ff9980fbf60 msgr2=0x7ff9980fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 -- 172.21.15.22:0/3495116617 shutdown_connections 2024-03-24T06:49:26.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.685+0000 7ff99cc00640 1 -- 172.21.15.22:0/3495116617 wait complete. 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 Processor -- start 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 -- start start 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998106550 0x7ff99819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7ff99819ec50 con 0x7ff998106550 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff99819edc0 con 0x7ff998100e50 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.686+0000 7ff99cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff99819ef80 con 0x7ff998100490 2024-03-24T06:49:26.687 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44618/0 (socket says 172.21.15.22:44618) 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 -- 172.21.15.22:0/399090528 learned_addr learned my addr 172.21.15.22:0/399090528 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff98ea00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997e00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998106550 0x7ff99819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 -- 172.21.15.22:0/399090528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 msgr2=0x7ff99819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 -- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998106550 msgr2=0x7ff99819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998106550 0x7ff99819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.688 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff997400640 1 -- 172.21.15.22:0/399090528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff988009900 con 0x7ff998100490 2024-03-24T06:49:26.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.687+0000 7ff98ea00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff997400640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff988004af0 tx=0x7ff988034cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff994c00640 1 -- 172.21.15.22:0/399090528 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880328c0 con 0x7ff998100490 2024-03-24T06:49:26.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff994c00640 1 -- 172.21.15.22:0/399090528 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff988003e50 con 0x7ff998100490 2024-03-24T06:49:26.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff994c00640 1 -- 172.21.15.22:0/399090528 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff988040450 con 0x7ff998100490 2024-03-24T06:49:26.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99819f200 con 0x7ff998100490 2024-03-24T06:49:26.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.688+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9981a6900 con 0x7ff998100490 2024-03-24T06:49:26.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.691+0000 7ff994c00640 1 -- 172.21.15.22:0/399090528 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7ff98800a040 con 0x7ff998100490 2024-03-24T06:49:26.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.691+0000 7ff9a1018740 1 Processor -- start 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 -- start start 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385450 0x55d118385830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d11838e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d1182f36e0 con 0x55d118385d70 2024-03-24T06:49:26.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d1182f0c10 con 0x55d118385450 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff9a1018740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d1182ec3f0 con 0x55d11838e950 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.692+0000 7ff997400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d11838e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44620/0 (socket says 172.21.15.22:44620) 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 -- 172.21.15.22:0/3834428067 learned_addr learned my addr 172.21.15.22:0/3834428067 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff997e00640 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385450 0x55d118385830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 -- 172.21.15.22:0/3834428067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385450 msgr2=0x55d118385830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385450 0x55d118385830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff994c00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 0x7ff97c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 -- 172.21.15.22:0/3834428067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 msgr2=0x55d11838e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d11838e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 -- 172.21.15.22:0/3834428067 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1182ade40 con 0x55d11838e950 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff994c00640 1 -- 172.21.15.22:0/399090528 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff9880f0c60 con 0x7ff998100490 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff98ea00640 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff980010480 tx=0x7ff980010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6b0d2b3caf20e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.693+0000 7ff997e00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 0x7ff97c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff987e00640 1 -- 172.21.15.22:0/3834428067 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980011760 con 0x55d11838e950 2024-03-24T06:49:26.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff987e00640 1 -- 172.21.15.22:0/3834428067 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff980015070 con 0x55d11838e950 2024-03-24T06:49:26.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff987e00640 1 -- 172.21.15.22:0/3834428067 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98001cde0 con 0x55d11838e950 2024-03-24T06:49:26.699 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 -- 172.21.15.22:0/3834428067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 msgr2=0x55d118390d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff980010480 tx=0x7ff980010940 comp rx=0 tx=0).stop 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff997e00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 0x7ff97c0a4770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff990009d80 tx=0x7ff990005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 -- 172.21.15.22:0/3834428067 shutdown_connections 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d118390d40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d11838e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 --2- 172.21.15.22:0/3834428067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385450 0x55d118385830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.694+0000 7ff9a1018740 1 -- 172.21.15.22:0/3834428067 >> 172.21.15.22:0/3834428067 conn(0x55d11837fb80 msgr2=0x55d11837ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.695+0000 7ff9a1018740 1 -- 172.21.15.22:0/3834428067 shutdown_connections 2024-03-24T06:49:26.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.695+0000 7ff9a1018740 1 -- 172.21.15.22:0/3834428067 wait complete. 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.695+0000 7ff9a1018740 1 -- start start 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.695+0000 7ff9a1018740 1 Processor -- start 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d11838c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d118291060 con 0x55d118385450 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d118290d00 con 0x55d118385d70 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d11805a1c0 con 0x55d11838e950 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff9a1018740 10 client.0 _ll_register_callbacks cb 0x55d117ffcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff997400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff997e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d11838c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34476/0 (socket says 172.21.15.22:34476) 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 -- 172.21.15.22:0/4100680811 learned_addr learned my addr 172.21.15.22:0/4100680811 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 -- 172.21.15.22:0/4100680811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 msgr2=0x55d11838c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d11838c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 -- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 msgr2=0x55d11838b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.696+0000 7ff98ea00640 1 -- 172.21.15.22:0/4100680811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98000bcd0 con 0x55d118385d70 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff98ea00640 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff98001cac0 tx=0x7ff98001dda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff0e9c34342bdc70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff984c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980005440 con 0x55d118385d70 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d117feb7e0 con 0x55d118385d70 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff980017070 con 0x55d118385d70 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff98001c780 con 0x55d118385d70 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff9a1018740 10 client.27635 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.697+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d1182eaa70 con 0x55d118385d70 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d117feb7e0 con 0x55d118385d70 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff980011560 con 0x55d118385d70 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 10 client.27635 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 10 client.27635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 20 client.27635 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d11839ec40 0x55d1183a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 4 client.27635 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d118028720 con 0x55d11839ec40 2024-03-24T06:49:26.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.698+0000 7ff997400640 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d11839ec40 0x55d1183a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.699+0000 7ff997400640 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d11839ec40 0x55d1183a10a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cef1c29f087d03b9 server_cookie=217c06c82552f375 in_seq=0 out_seq=0 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.699+0000 7ff984c00640 10 client.27635 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff984c00640 1 -- 172.21.15.22:0/4100680811 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d118028720 con 0x55d11839ec40 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff984c00640 10 client.27635 handle_command_reply: tid=0 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 client.27635 shutdown 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 msgr2=0x55d11838be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff98001cac0 tx=0x7ff98001dda0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 shutdown_connections 2024-03-24T06:49:26.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d11838e950 0x55d11838c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d11839ec40 0x55d1183a10a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d118385d70 0x55d11838be50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 --2- 172.21.15.22:0/4100680811 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff984c00640 0 client.27635 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 >> 172.21.15.22:0/4100680811 conn(0x55d11837fb80 msgr2=0x55d1183818d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 shutdown_connections 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 1 -- 172.21.15.22:0/4100680811 wait complete. 2024-03-24T06:49:26.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.700+0000 7ff9a1018740 20 client.27635 trim_cache size 0 max 16384 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.715+0000 7ff9a1018740 1 Processor -- start 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.715+0000 7ff9a1018740 1 -- start start 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d118384860 con 0x55d118385450 2024-03-24T06:49:26.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d11837f3d0 con 0x55d11838e950 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff9a1018740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d118391310 con 0x55d118385d70 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff997400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff997e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44644/0 (socket says 172.21.15.22:44644) 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff98ea00640 1 -- 172.21.15.22:0/416730073 learned_addr learned my addr 172.21.15.22:0/416730073 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.716+0000 7ff997e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34482/0 (socket says 172.21.15.22:34482) 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 -- 172.21.15.22:0/416730073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 msgr2=0x55d1183a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 -- 172.21.15.22:0/416730073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 msgr2=0x55d11838e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 -- 172.21.15.22:0/416730073 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d118286b00 con 0x55d118385d70 2024-03-24T06:49:26.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff997e00640 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff98ea00640 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff980011b30 tx=0x7ff980010e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6da84c1c519a4e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff987400640 1 -- 172.21.15.22:0/416730073 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98001de10 con 0x55d118385d70 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff987400640 1 -- 172.21.15.22:0/416730073 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff980021040 con 0x55d118385d70 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.717+0000 7ff987400640 1 -- 172.21.15.22:0/416730073 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980011e10 con 0x55d118385d70 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 -- 172.21.15.22:0/416730073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 msgr2=0x55d11839a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff980011b30 tx=0x7ff980010e60 comp rx=0 tx=0).stop 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 -- 172.21.15.22:0/416730073 shutdown_connections 2024-03-24T06:49:26.723 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d1183a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385d70 0x55d11839a1e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 --2- 172.21.15.22:0/416730073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385450 0x55d11838e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 -- 172.21.15.22:0/416730073 >> 172.21.15.22:0/416730073 conn(0x55d11837fb80 msgr2=0x55d11837ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 -- 172.21.15.22:0/416730073 shutdown_connections 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.718+0000 7ff9a1018740 1 -- 172.21.15.22:0/416730073 wait complete. 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 -- start start 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 Processor -- start 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.724 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d118392910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d118392e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d118404180 con 0x55d118385d70 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d1184042f0 con 0x55d11838e950 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d1184044b0 con 0x55d118385450 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff9a1018740 10 client.0 _ll_register_callbacks cb 0x7ff9880f45a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff997400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.719+0000 7ff98ea00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d118392910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff997400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44660/0 (socket says 172.21.15.22:44660) 2024-03-24T06:49:26.725 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff997400640 1 -- 172.21.15.22:0/1168000861 learned_addr learned my addr 172.21.15.22:0/1168000861 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff997e00640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d118392e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 -- 172.21.15.22:0/1168000861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 msgr2=0x55d1183923d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 -- 172.21.15.22:0/1168000861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 msgr2=0x55d118392e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d118392e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 -- 172.21.15.22:0/1168000861 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff990009e50 con 0x55d118385d70 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff997400640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff98ea00640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d118392910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55d11838b4e0 tx=0x7ff980005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d4e88a8e8c6a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:26.726 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.720+0000 7ff984c00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98001c480 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff980015070 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7ff980017050 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff980011bb0 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 10 client.17280 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d1182eaa70 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff980015070 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7ff980004020 con 0x55d118385d70 2024-03-24T06:49:26.727 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 10 client.17280 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 20 client.17280 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.721+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d118405810 0x55d118407c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.722+0000 7ff9a1018740 4 client.17280 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.722+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d118077890 con 0x55d118405810 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.722+0000 7ff997400640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d118405810 0x55d118407c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.722+0000 7ff997400640 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d118405810 0x55d118407c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7241a7109f7979b6 server_cookie=89acddfcc192b940 in_seq=0 out_seq=0 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.722+0000 7ff984c00640 10 client.17280 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:26.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff984c00640 1 -- 172.21.15.22:0/1168000861 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d118077890 con 0x55d118405810 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff984c00640 10 client.17280 handle_command_reply: tid=0 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 client.17280 shutdown 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 msgr2=0x55d118392910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d118392910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55d11838b4e0 tx=0x7ff980005e50 comp rx=0 tx=0).stop 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 shutdown_connections 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d118405810 0x55d118407c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d11838e950 0x55d118392e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d118385d70 0x55d118392910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 --2- 172.21.15.22:0/1168000861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d118385450 0x55d1183923d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff984c00640 0 client.17280 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:26.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 >> 172.21.15.22:0/1168000861 conn(0x55d11837fb80 msgr2=0x55d11838b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 shutdown_connections 2024-03-24T06:49:26.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 1 -- 172.21.15.22:0/1168000861 wait complete. 2024-03-24T06:49:26.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.723+0000 7ff9a1018740 20 client.17280 trim_cache size 0 max 16384 2024-03-24T06:49:26.730 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 msgr2=0x7ff97c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 0x7ff97c0a4770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff990009d80 tx=0x7ff990005f90 comp rx=0 tx=0).stop 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 msgr2=0x7ff998199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff988004af0 tx=0x7ff988034cc0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 shutdown_connections 2024-03-24T06:49:26.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7ff97c0a22b0 0x7ff97c0a4770 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff998106550 0x7ff99819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff998100e50 0x7ff99819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 --2- 172.21.15.22:0/399090528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff998100490 0x7ff998199d90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 >> 172.21.15.22:0/399090528 conn(0x7ff9980fbf60 msgr2=0x7ff998068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 shutdown_connections 2024-03-24T06:49:26.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:26.724+0000 7ff99cc00640 1 -- 172.21.15.22:0/399090528 wait complete. 2024-03-24T06:49:27.742 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:49:27.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:49:27.742 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:49:27.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 -- 172.21.15.22:0/4189050275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 msgr2=0x7fb54010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 --2- 172.21.15.22:0/4189050275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54010e570 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb538009960 tx=0x7fb53802bc20 comp rx=0 tx=0).stop 2024-03-24T06:49:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 -- 172.21.15.22:0/4189050275 shutdown_connections 2024-03-24T06:49:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 --2- 172.21.15.22:0/4189050275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54010e570 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 --2- 172.21.15.22:0/4189050275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 --2- 172.21.15.22:0/4189050275 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 0x7fb540107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 -- 172.21.15.22:0/4189050275 >> 172.21.15.22:0/4189050275 conn(0x7fb540077710 msgr2=0x7fb540077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:27.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 -- 172.21.15.22:0/4189050275 shutdown_connections 2024-03-24T06:49:27.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.983+0000 7fb547600640 1 -- 172.21.15.22:0/4189050275 wait complete. 2024-03-24T06:49:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 Processor -- start 2024-03-24T06:49:27.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 -- start start 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 0x7fb54019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb54019ed80 con 0x7fb540107760 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb54019eef0 con 0x7fb5400692d0 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.984+0000 7fb547600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb54019f0b0 con 0x7fb540068910 2024-03-24T06:49:27.987 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb544c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34516/0 (socket says 172.21.15.22:34516) 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb545600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 0x7fb54019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 -- 172.21.15.22:0/903895196 learned_addr learned my addr 172.21.15.22:0/903895196 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb544c00640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44676/0 (socket says 172.21.15.22:44676) 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 -- 172.21.15.22:0/903895196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 msgr2=0x7fb540199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:27.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 -- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 msgr2=0x7fb54019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 0x7fb54019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.985+0000 7fb537e00640 1 -- 172.21.15.22:0/903895196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb538009560 con 0x7fb5400692d0 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb537e00640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9f0 tx=0x7fb52800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb53e000640 1 -- 172.21.15.22:0/903895196 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528009700 con 0x7fb5400692d0 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb53e000640 1 -- 172.21.15.22:0/903895196 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb52800cd80 con 0x7fb5400692d0 2024-03-24T06:49:27.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54019f390 con 0x7fb5400692d0 2024-03-24T06:49:27.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb53e000640 1 -- 172.21.15.22:0/903895196 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528018920 con 0x7fb5400692d0 2024-03-24T06:49:27.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.986+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5401a6c10 con 0x7fb5400692d0 2024-03-24T06:49:27.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 Processor -- start 2024-03-24T06:49:27.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 -- start start 2024-03-24T06:49:27.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4ca22d0 con 0x55dfb4d3d330 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4c9f800 con 0x55dfb4d34750 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb54a40b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4c9afe0 con 0x55dfb4d33e30 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb545600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb544c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:27.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb544c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34524/0 (socket says 172.21.15.22:34524) 2024-03-24T06:49:27.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.990+0000 7fb544c00640 1 -- 172.21.15.22:0/2803150068 learned_addr learned my addr 172.21.15.22:0/2803150068 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:27.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 -- 172.21.15.22:0/2803150068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 msgr2=0x55dfb4d34210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:27.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d34210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 -- 172.21.15.22:0/2803150068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 msgr2=0x55dfb4d3f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:49:27.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 -- 172.21.15.22:0/2803150068 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb4c5ce90 con 0x55dfb4d34750 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb53e000640 1 -- 172.21.15.22:0/903895196 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb528018a80 con 0x7fb5400692d0 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb544c00640 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb5300114f0 tx=0x7fb5300119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbdf1d6d9a90ebc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb535c00640 1 -- 172.21.15.22:0/2803150068 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000f620 con 0x55dfb4d34750 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb535c00640 1 -- 172.21.15.22:0/2803150068 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb53000fc60 con 0x55dfb4d34750 2024-03-24T06:49:27.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.991+0000 7fb535c00640 1 -- 172.21.15.22:0/2803150068 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530019d00 con 0x55dfb4d34750 2024-03-24T06:49:28.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 -- 172.21.15.22:0/2803150068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 msgr2=0x55dfb4d3cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb5300114f0 tx=0x7fb5300119b0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 -- 172.21.15.22:0/2803150068 shutdown_connections 2024-03-24T06:49:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3cdf0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2803150068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d34210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 -- 172.21.15.22:0/2803150068 >> 172.21.15.22:0/2803150068 conn(0x55dfb4d2e560 msgr2=0x55dfb4d2e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 -- 172.21.15.22:0/2803150068 shutdown_connections 2024-03-24T06:49:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.992+0000 7fb54a40b740 1 -- 172.21.15.22:0/2803150068 wait complete. 2024-03-24T06:49:28.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.993+0000 7fb54a40b740 1 -- start start 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.993+0000 7fb53e000640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 0x7fb5240a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.993+0000 7fb54a40b740 1 Processor -- start 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.993+0000 7fb53e000640 1 -- 172.21.15.22:0/903895196 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb528014070 con 0x7fb5400692d0 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.993+0000 7fb545600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 0x7fb5240a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4c3ffc0 con 0x55dfb4d3d330 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4c3fc60 con 0x55dfb4d34750 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfb49e30c0 con 0x55dfb4d33e30 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb54a40b740 10 client.0 _ll_register_callbacks cb 0x55dfb4c57e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb544c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb544c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44692/0 (socket says 172.21.15.22:44692) 2024-03-24T06:49:28.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb544c00640 1 -- 172.21.15.22:0/2153780168 learned_addr learned my addr 172.21.15.22:0/2153780168 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb545600640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb545600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 0x7fb5240a4680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb538005ab0 tx=0x7fb538005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 -- 172.21.15.22:0/2153780168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 msgr2=0x55dfb4d3a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 -- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 msgr2=0x55dfb4d3aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb537e00640 1 -- 172.21.15.22:0/2153780168 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb530009c80 con 0x55dfb4d34750 2024-03-24T06:49:28.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.994+0000 7fb545600640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb537e00640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3a960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb5280cbad0 tx=0x7fb52809ae20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32115b1d5aa525af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb52e000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5280ce340 con 0x55dfb4d34750 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb5280ce4a0 con 0x55dfb4d34750 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfb4cbfe90 con 0x55dfb4d34750 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb52809a270 con 0x55dfb4d34750 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb54a40b740 10 client.27653 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:28.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dfb4c99660 con 0x55dfb4d34750 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.995+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5280b9030 con 0x55dfb4d34750 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb528099d50 con 0x55dfb4d34750 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 10 client.27653 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 10 client.27653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 20 client.27653 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4d4d8d0 0x55dfb4d4fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.007 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 4 client.27653 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfb4c91210 con 0x55dfb4d4d8d0 2024-03-24T06:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.996+0000 7fb544c00640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4d4d8d0 0x55dfb4d4fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.997+0000 7fb544c00640 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4d4d8d0 0x55dfb4d4fcf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=783b72f62d052254 server_cookie=f2b7638880f56a0f in_seq=0 out_seq=0 2024-03-24T06:49:28.008 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.997+0000 7fb52e000640 10 client.27653 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.997+0000 7fb52e000640 1 -- 172.21.15.22:0/2153780168 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfb4c91210 con 0x55dfb4d4d8d0 2024-03-24T06:49:28.009 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.997+0000 7fb52e000640 10 client.27653 handle_command_reply: tid=0 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 client.27653 shutdown 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 msgr2=0x55dfb4d3a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3a960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb5280cbad0 tx=0x7fb52809ae20 comp rx=0 tx=0).stop 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 shutdown_connections 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d3d330 0x55dfb4d3aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4d4d8d0 0x55dfb4d4fcf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.010 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d3a960 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 --2- 172.21.15.22:0/2153780168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb52e000640 0 client.27653 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 >> 172.21.15.22:0/2153780168 conn(0x55dfb4d2e560 msgr2=0x55dfb4d302b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 shutdown_connections 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 1 -- 172.21.15.22:0/2153780168 wait complete. 2024-03-24T06:49:28.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:27.998+0000 7fb54a40b740 20 client.27653 trim_cache size 0 max 16384 2024-03-24T06:49:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 Processor -- start 2024-03-24T06:49:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 -- start start 2024-03-24T06:49:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d48bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d34570 con 0x55dfb4d33e30 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d3b380 con 0x55dfb4d34750 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.018+0000 7fb54a40b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d49100 con 0x55dfb4d3d330 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb544c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44698/0 (socket says 172.21.15.22:44698) 2024-03-24T06:49:28.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 -- 172.21.15.22:0/3221603432 learned_addr learned my addr 172.21.15.22:0/3221603432 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb537e00640 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d48bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 -- 172.21.15.22:0/3221603432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 msgr2=0x55dfb4d48bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d48bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 -- 172.21.15.22:0/3221603432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d33e30 msgr2=0x55dfb4d3cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.019+0000 7fb545600640 1 -- 172.21.15.22:0/3221603432 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb4c35ae0 con 0x55dfb4d3d330 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.020+0000 7fb545600640 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb53802c230 tx=0x7fb538042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ac425b93c256a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.025 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.020+0000 7fb535200640 1 -- 172.21.15.22:0/3221603432 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53802dea0 con 0x55dfb4d3d330 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.020+0000 7fb535200640 1 -- 172.21.15.22:0/3221603432 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb538050070 con 0x55dfb4d3d330 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.020+0000 7fb535200640 1 -- 172.21.15.22:0/3221603432 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538003830 con 0x55dfb4d3d330 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- 172.21.15.22:0/3221603432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 msgr2=0x55dfb4d555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb53802c230 tx=0x7fb538042000 comp rx=0 tx=0).stop 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- 172.21.15.22:0/3221603432 shutdown_connections 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d3d330 0x55dfb4d555e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d34750 0x55dfb4d48bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 --2- 172.21.15.22:0/3221603432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d33e30 0x55dfb4d3cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- 172.21.15.22:0/3221603432 >> 172.21.15.22:0/3221603432 conn(0x55dfb4d2e560 msgr2=0x55dfb4d3ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- 172.21.15.22:0/3221603432 shutdown_connections 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- 172.21.15.22:0/3221603432 wait complete. 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 -- start start 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.021+0000 7fb54a40b740 1 Processor -- start 2024-03-24T06:49:28.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 0x55dfb4d415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d3d330 0x55dfb4d41af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d42030 con 0x55dfb4d34750 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d421a0 con 0x55dfb4d3d330 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfb4d42360 con 0x55dfb4d33e30 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb54a40b740 10 client.0 _ll_register_callbacks cb 0x7fb52800cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb544c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb537e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 0x55dfb4d415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb544c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44700/0 (socket says 172.21.15.22:44700) 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb544c00640 1 -- 172.21.15.22:0/1382471698 learned_addr learned my addr 172.21.15.22:0/1382471698 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.022+0000 7fb545600640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d3d330 0x55dfb4d41af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 -- 172.21.15.22:0/1382471698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d3d330 msgr2=0x55dfb4d41af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d3d330 0x55dfb4d41af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 -- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 msgr2=0x55dfb4d415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 0x55dfb4d415b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 -- 172.21.15.22:0/1382471698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54010ee70 con 0x55dfb4d33e30 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb537e00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 0x55dfb4d415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb544c00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb530019930 tx=0x7fb530019860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb22ca46e5ce95d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb52e000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530005df0 con 0x55dfb4d33e30 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfb4d35df0 con 0x55dfb4d33e30 2024-03-24T06:49:28.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb530011020 con 0x55dfb4d33e30 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.023+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb530011220 con 0x55dfb4d33e30 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 10 client.27685 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dfb4c5b6f0 con 0x55dfb4d33e30 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dfb4d35df0 con 0x55dfb4d33e30 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb53000f560 con 0x55dfb4d33e30 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 10 client.27685 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 10 client.27685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:28.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 20 client.27685 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4db50a0 0x55dfb4db7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 4 client.27685 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fb53000fc60 con 0x55dfb4db50a0 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.024+0000 7fb537e00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4db50a0 0x55dfb4db7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.025+0000 7fb537e00640 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4db50a0 0x55dfb4db7490 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46bab1a6fae11e6c server_cookie=163e9ffb79f849ab in_seq=0 out_seq=0 2024-03-24T06:49:28.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.025+0000 7fb52e000640 10 client.27685 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb52e000640 1 -- 172.21.15.22:0/1382471698 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fb53000fc60 con 0x55dfb4db50a0 2024-03-24T06:49:28.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb52e000640 10 client.27685 handle_command_reply: tid=0 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 client.27685 shutdown 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 msgr2=0x55dfb4d451e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb530019930 tx=0x7fb530019860 comp rx=0 tx=0).stop 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 shutdown_connections 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55dfb4db50a0 0x55dfb4db7490 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dfb4d3d330 0x55dfb4d41af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55dfb4d34750 0x55dfb4d415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 --2- 172.21.15.22:0/1382471698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfb4d33e30 0x55dfb4d451e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 >> 172.21.15.22:0/1382471698 conn(0x55dfb4d2e560 msgr2=0x55dfb4d2cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.026+0000 7fb52e000640 0 client.27685 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.027+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 shutdown_connections 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.027+0000 7fb54a40b740 1 -- 172.21.15.22:0/1382471698 wait complete. 2024-03-24T06:49:28.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.027+0000 7fb54a40b740 20 client.27685 trim_cache size 0 max 16384 2024-03-24T06:49:28.036 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:49:28.036 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:49:28.036 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "e2b69965-769c-4865-9124-599a3a467e21", 2024-03-24T06:49:28.036 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:49:28.036 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:49:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 msgr2=0x7fb5240a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 0x7fb5240a4680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb538005ab0 tx=0x7fb538005a20 comp rx=0 tx=0).stop 2024-03-24T06:49:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 msgr2=0x7fb54019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e9f0 tx=0x7fb52800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 shutdown_connections 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb5240a21c0 0x7fb5240a4680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb540107760 0x7fb54019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5400692d0 0x7fb54019a400 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 --2- 172.21.15.22:0/903895196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb540068910 0x7fb540199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 >> 172.21.15.22:0/903895196 conn(0x7fb540077710 msgr2=0x7fb54010e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.028+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 shutdown_connections 2024-03-24T06:49:28.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.029+0000 7fb547600640 1 -- 172.21.15.22:0/903895196 wait complete. 2024-03-24T06:49:28.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:49:28.053 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:49:28.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.255+0000 7fc034a00640 1 -- 172.21.15.22:0/4257925653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 msgr2=0x7fc030107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.255+0000 7fc034a00640 1 --2- 172.21.15.22:0/4257925653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc030107a40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc020009930 tx=0x7fc02002bc30 comp rx=0 tx=0).stop 2024-03-24T06:49:28.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.255+0000 7fc034a00640 1 -- 172.21.15.22:0/4257925653 shutdown_connections 2024-03-24T06:49:28.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 --2- 172.21.15.22:0/4257925653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 0x7fc03010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 --2- 172.21.15.22:0/4257925653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 --2- 172.21.15.22:0/4257925653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc030107a40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 -- 172.21.15.22:0/4257925653 >> 172.21.15.22:0/4257925653 conn(0x7fc0300fd0b0 msgr2=0x7fc0300ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 -- 172.21.15.22:0/4257925653 shutdown_connections 2024-03-24T06:49:28.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.256+0000 7fc034a00640 1 -- 172.21.15.22:0/4257925653 wait complete. 2024-03-24T06:49:28.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 Processor -- start 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 -- start start 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 0x7fc03019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc03019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fc03019ec50 con 0x7fc030101f60 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc03019edc0 con 0x7fc0301015a0 2024-03-24T06:49:28.259 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.257+0000 7fc034a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc03019ef80 con 0x7fc030107660 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34554/0 (socket says 172.21.15.22:34554) 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 -- 172.21.15.22:0/1349317648 learned_addr learned my addr 172.21.15.22:0/1349317648 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02ea00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 0x7fc03019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02fe00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc03019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 -- 172.21.15.22:0/1349317648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 msgr2=0x7fc03019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc03019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 -- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 msgr2=0x7fc03019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 0x7fc03019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 -- 172.21.15.22:0/1349317648 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc020009560 con 0x7fc0301015a0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.258+0000 7fc02f400640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc020000c00 tx=0x7fc02002cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.259+0000 7fc027e00640 1 -- 172.21.15.22:0/1349317648 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020004350 con 0x7fc0301015a0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.259+0000 7fc027e00640 1 -- 172.21.15.22:0/1349317648 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0200044b0 con 0x7fc0301015a0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.259+0000 7fc027e00640 1 -- 172.21.15.22:0/1349317648 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020003d30 con 0x7fc0301015a0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.259+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03019f200 con 0x7fc0301015a0 2024-03-24T06:49:28.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.259+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0301a6ae0 con 0x7fc0301015a0 2024-03-24T06:49:28.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 Processor -- start 2024-03-24T06:49:28.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc027e00640 1 -- 172.21.15.22:0/1349317648 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fc020004df0 con 0x7fc0301015a0 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 -- start start 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 0x558ff70b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70c02d0 0x558ff70c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ff7025790 con 0x558ff70b6f60 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ff7022cc0 con 0x558ff70b7c40 2024-03-24T06:49:28.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc038dd1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ff701e4a0 con 0x558ff70c02d0 2024-03-24T06:49:28.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc02fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc02fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34568/0 (socket says 172.21.15.22:34568) 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.264+0000 7fc02fe00640 1 -- 172.21.15.22:0/2687333841 learned_addr learned my addr 172.21.15.22:0/2687333841 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02ea00640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70c02d0 0x558ff70c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02f400640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 0x558ff70b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 -- 172.21.15.22:0/2687333841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70c02d0 msgr2=0x558ff70c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70c02d0 0x558ff70c26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 -- 172.21.15.22:0/2687333841 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 msgr2=0x558ff70b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 0x558ff70b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 -- 172.21.15.22:0/2687333841 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ff6fdfc20 con 0x558ff70b7c40 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02f400640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 0x558ff70b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.265+0000 7fc02fe00640 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc028010370 tx=0x7fc028010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea77affa85a9d08d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc01be00640 1 -- 172.21.15.22:0/2687333841 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028004d20 con 0x558ff70b7c40 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc01be00640 1 -- 172.21.15.22:0/2687333841 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc028004e80 con 0x558ff70b7c40 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc01be00640 1 -- 172.21.15.22:0/2687333841 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028005560 con 0x558ff70b7c40 2024-03-24T06:49:28.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc027e00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 0x7fc0140a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc027e00640 1 -- 172.21.15.22:0/1349317648 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc0200f09f0 con 0x7fc0301015a0 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc02f400640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 0x7fc0140a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc038dd1740 1 -- 172.21.15.22:0/2687333841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 msgr2=0x558ff70b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.266+0000 7fc038dd1740 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc028010370 tx=0x7fc028010830 comp rx=0 tx=0).stop 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 -- 172.21.15.22:0/2687333841 shutdown_connections 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70c02d0 0x558ff70c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b6f60 0x558ff70b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 --2- 172.21.15.22:0/2687333841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70b6990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 -- 172.21.15.22:0/2687333841 >> 172.21.15.22:0/2687333841 conn(0x558ff707db30 msgr2=0x558ff70b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc02f400640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 0x7fc0140a4770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc0200edf50 tx=0x7fc02002bc30 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 -- 172.21.15.22:0/2687333841 shutdown_connections 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 -- 172.21.15.22:0/2687333841 wait complete. 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.267+0000 7fc038dd1740 1 -- start start 2024-03-24T06:49:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 Processor -- start 2024-03-24T06:49:28.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b7c40 0x558ff70bd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70c02d0 0x558ff70bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.274 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ff6fc2f20 con 0x558ff70b7c40 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ff6fc2bc0 con 0x558ff70c02d0 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ff6d8c1c0 con 0x558ff70b6f60 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc038dd1740 10 client.0 _ll_register_callbacks cb 0x558ff6d2ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc02fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc02fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44740/0 (socket says 172.21.15.22:44740) 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.268+0000 7fc02fe00640 1 -- 172.21.15.22:0/993775228 learned_addr learned my addr 172.21.15.22:0/993775228 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70c02d0 0x558ff70bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.275 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02ea00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b7c40 0x558ff70bd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 -- 172.21.15.22:0/993775228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70c02d0 msgr2=0x558ff70bdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70c02d0 0x558ff70bdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 -- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b7c40 msgr2=0x558ff70bd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b7c40 0x558ff70bd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 -- 172.21.15.22:0/993775228 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc028009c50 con 0x558ff70b6f60 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc02fe00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc0280059b0 tx=0x7fc028009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e8f005e82b8d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.276 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc018c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.269+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02801e020 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ff6d1d7e0 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc028005a10 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc028003970 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 10 client.27697 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558ff701cb20 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ff6d1d7e0 con 0x558ff70b6f60 2024-03-24T06:49:28.277 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc028012510 con 0x558ff70b6f60 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 10 client.27697 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 10 client.27697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 20 client.27697 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff70d06b0 0x558ff70d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 4 client.27697 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.270+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ff6d5a720 con 0x558ff70d06b0 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.271+0000 7fc02ea00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff70d06b0 0x558ff70d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.271+0000 7fc02ea00640 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff70d06b0 0x558ff70d2ad0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40ceba9d8541d31a server_cookie=9c85d026b8348c5f in_seq=0 out_seq=0 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.271+0000 7fc018c00640 10 client.27697 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc018c00640 1 -- 172.21.15.22:0/993775228 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ff6d5a720 con 0x558ff70d06b0 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc018c00640 10 client.27697 handle_command_reply: tid=0 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 client.27697 shutdown 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 msgr2=0x558ff70bd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc0280059b0 tx=0x7fc028009bb0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 shutdown_connections 2024-03-24T06:49:28.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff70d06b0 0x558ff70d2ad0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70c02d0 0x558ff70bdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70b7c40 0x558ff70bd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc018c00640 0 client.27697 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 --2- 172.21.15.22:0/993775228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bd240 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.272+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 >> 172.21.15.22:0/993775228 conn(0x558ff707db30 msgr2=0x558ff70b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.273+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 shutdown_connections 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.273+0000 7fc038dd1740 1 -- 172.21.15.22:0/993775228 wait complete. 2024-03-24T06:49:28.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.273+0000 7fc038dd1740 20 client.27697 trim_cache size 0 max 16384 2024-03-24T06:49:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.292+0000 7fc038dd1740 1 Processor -- start 2024-03-24T06:49:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.292+0000 7fc038dd1740 1 -- start start 2024-03-24T06:49:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.292+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.292+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff70cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc038dd1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ff70d3160 con 0x558ff70c02d0 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc038dd1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ff70b61e0 con 0x558ff70b7c40 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc038dd1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ff70bd060 con 0x558ff70b6f60 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:34590/0 (socket says 172.21.15.22:34590) 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 -- 172.21.15.22:0/3884262498 learned_addr learned my addr 172.21.15.22:0/3884262498 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff70cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 -- 172.21.15.22:0/3884262498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 msgr2=0x558ff70bf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 -- 172.21.15.22:0/3884262498 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 msgr2=0x558ff70cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff70cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02ea00640 1 -- 172.21.15.22:0/3884262498 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ff6fb89a0 con 0x558ff70b7c40 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.293+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:28.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.294+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x558ff70d3340 tx=0x7fc01c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc5b50deccca9dec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.294+0000 7fc01b400640 1 -- 172.21.15.22:0/3884262498 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c017030 con 0x558ff70b7c40 2024-03-24T06:49:28.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.294+0000 7fc01b400640 1 -- 172.21.15.22:0/3884262498 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc01c018040 con 0x558ff70b7c40 2024-03-24T06:49:28.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.294+0000 7fc01b400640 1 -- 172.21.15.22:0/3884262498 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c010960 con 0x558ff70b7c40 2024-03-24T06:49:28.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- 172.21.15.22:0/3884262498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 msgr2=0x558ff70bc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x558ff70d3340 tx=0x7fc01c005f90 comp rx=0 tx=0).stop 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- 172.21.15.22:0/3884262498 shutdown_connections 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff70cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b7c40 0x558ff70bc3e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3884262498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b6f60 0x558ff70bf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- 172.21.15.22:0/3884262498 >> 172.21.15.22:0/3884262498 conn(0x558ff707db30 msgr2=0x558ff70b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- 172.21.15.22:0/3884262498 shutdown_connections 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- 172.21.15.22:0/3884262498 wait complete. 2024-03-24T06:49:28.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.295+0000 7fc038dd1740 1 -- start start 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 Processor -- start 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b6f60 0x558ff70ba450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff71359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558ff70bb040 con 0x558ff70c02d0 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558ff7135f10 con 0x558ff70b6f60 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558ff7136050 con 0x558ff70b7c40 2024-03-24T06:49:28.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc038dd1740 10 client.0 _ll_register_callbacks cb 0x7fc01c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc02ea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44768/0 (socket says 172.21.15.22:44768) 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.296+0000 7fc02ea00640 1 -- 172.21.15.22:0/3680316277 learned_addr learned my addr 172.21.15.22:0/3680316277 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b6f60 0x558ff70ba450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff71359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 -- 172.21.15.22:0/3680316277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b6f60 msgr2=0x558ff70ba450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b6f60 0x558ff70ba450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 -- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 msgr2=0x558ff71359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff71359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 -- 172.21.15.22:0/3680316277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c009b00 con 0x558ff70b7c40 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc02ea00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x558ff70cd2d0 tx=0x7fc01c003d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e4cd7fb94bed0c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.297+0000 7fc018c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c017040 con 0x558ff70b7c40 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ff7016160 con 0x558ff70b7c40 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fc01c010af0 con 0x558ff70b7c40 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc01c010cb0 con 0x558ff70b7c40 2024-03-24T06:49:28.304 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 10 client.27703 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558ff6fde480 con 0x558ff70b7c40 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ff7016160 con 0x558ff70b7c40 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fc01c026070 con 0x558ff70b7c40 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 10 client.27703 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 10 client.27703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 20 client.27703 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.298+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff71378c0 0x558ff7139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.299+0000 7fc038dd1740 4 client.27703 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:49:28.305 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.299+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558ff6da9890 con 0x558ff71378c0 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.299+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff71378c0 0x558ff7139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.299+0000 7fc02fe00640 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff71378c0 0x558ff7139ce0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7770c2ceba49ff99 server_cookie=c9f078cd6d577388 in_seq=0 out_seq=0 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.299+0000 7fc018c00640 10 client.27703 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc018c00640 1 -- 172.21.15.22:0/3680316277 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558ff6da9890 con 0x558ff71378c0 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc018c00640 10 client.27703 handle_command_reply: tid=0 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 client.27703 shutdown 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 msgr2=0x558ff70ba990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x558ff70cd2d0 tx=0x7fc01c003d40 comp rx=0 tx=0).stop 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 shutdown_connections 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558ff70c02d0 0x558ff71359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558ff70b7c40 0x558ff70ba990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558ff71378c0 0x558ff7139ce0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 --2- 172.21.15.22:0/3680316277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558ff70b6f60 0x558ff70ba450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 >> 172.21.15.22:0/3680316277 conn(0x558ff707db30 msgr2=0x558ff70bda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc018c00640 0 client.27703 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 shutdown_connections 2024-03-24T06:49:28.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 1 -- 172.21.15.22:0/3680316277 wait complete. 2024-03-24T06:49:28.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.300+0000 7fc038dd1740 20 client.27703 trim_cache size 0 max 16384 2024-03-24T06:49:28.308 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:49:28.308 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:49:28.308 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:49:28.308 INFO:teuthology.orchestra.run.smithi022.stdout: "e2b69965-769c-4865-9124-599a3a467e21": { 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "e2b69965-769c-4865-9124-599a3a467e21", 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:49:28.309 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:49:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 msgr2=0x7fc0140a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 0x7fc0140a4770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc0200edf50 tx=0x7fc02002bc30 comp rx=0 tx=0).stop 2024-03-24T06:49:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 msgr2=0x7fc030199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc020000c00 tx=0x7fc02002cd80 comp rx=0 tx=0).stop 2024-03-24T06:49:28.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 shutdown_connections 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fc0140a22b0 0x7fc0140a4770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc030107660 0x7fc03019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc030101f60 0x7fc03019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 --2- 172.21.15.22:0/1349317648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0301015a0 0x7fc030199d90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 >> 172.21.15.22:0/1349317648 conn(0x7fc0300fd0b0 msgr2=0x7fc03010cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 shutdown_connections 2024-03-24T06:49:28.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:28.301+0000 7fc034a00640 1 -- 172.21.15.22:0/1349317648 wait complete. 2024-03-24T06:49:28.319 INFO:tasks.cephfs.filesystem:scrub status for tag:e2b69965-769c-4865-9124-599a3a467e21 - {'path': '/', 'tag': 'e2b69965-769c-4865-9124-599a3a467e21', 'options': 'recursive,force'} 2024-03-24T06:49:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:28 smithi022 ceph-mon[32003]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:28 smithi155 ceph-mon[36262]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:28.694 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:28 smithi143 ceph-mon[36520]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:49:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:49:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:49:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:29 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:49:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:49:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:49:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:29 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:49:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:49:29.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:49:29.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:49:29.680 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:29 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:49:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:30 smithi155 ceph-mon[36262]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:30 smithi022 ceph-mon[32003]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:30.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:30 smithi143 ceph-mon[36520]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:32.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:32 smithi155 ceph-mon[36262]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:49:32.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:32 smithi022 ceph-mon[32003]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:49:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:32 smithi143 ceph-mon[36520]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T06:49:34.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:34 smithi155 ceph-mon[36262]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:34 smithi022 ceph-mon[32003]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:34 smithi143 ceph-mon[36520]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:36.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:36 smithi155 ceph-mon[36262]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:36.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:36 smithi022 ceph-mon[32003]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:36.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:36 smithi143 ceph-mon[36520]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:38.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:38 smithi155 ceph-mon[36262]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:38.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:38 smithi022 ceph-mon[32003]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:38.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:38 smithi143 ceph-mon[36520]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-24T06:49:40.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:40 smithi155 ceph-mon[36262]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:40.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:40 smithi022 ceph-mon[32003]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:40.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:40 smithi143 ceph-mon[36520]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T06:49:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:42 smithi155 ceph-mon[36262]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:42.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:42 smithi022 ceph-mon[32003]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:42.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:42 smithi143 ceph-mon[36520]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T06:49:43.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:43.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:43.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:43.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:43 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:43.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:43 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:43.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:43 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:44 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:49:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:44 smithi155 ceph-mon[36262]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:49:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:44 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:49:44.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:44 smithi022 ceph-mon[32003]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:49:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:44 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:49:44.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:44 smithi143 ceph-mon[36520]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-03-24T06:49:46.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:46 smithi155 ceph-mon[36262]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:46.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:46 smithi022 ceph-mon[32003]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:46.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:46 smithi143 ceph-mon[36520]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:48 smithi155 ceph-mon[36262]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:48.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:48 smithi022 ceph-mon[32003]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:48.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:48 smithi143 ceph-mon[36520]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:50.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:50 smithi155 ceph-mon[36262]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:49:50.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:50 smithi022 ceph-mon[32003]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:49:50.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:50 smithi143 ceph-mon[36520]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:49:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:52 smithi155 ceph-mon[36262]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:52.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:52 smithi022 ceph-mon[32003]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:52.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:52 smithi143 ceph-mon[36520]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:54.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:54 smithi155 ceph-mon[36262]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:54.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:54 smithi022 ceph-mon[32003]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:54.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:54 smithi143 ceph-mon[36520]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:56 smithi155 ceph-mon[36262]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:56.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:56 smithi022 ceph-mon[32003]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:56.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:56 smithi143 ceph-mon[36520]: pgmap v308: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:49:58.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:49:58.320 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:49:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:58 smithi155 ceph-mon[36262]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:58.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 -- 172.21.15.22:0/76949555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d14108870 msgr2=0x7f5d1410c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 --2- 172.21.15.22:0/76949555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d14108870 0x7f5d1410c830 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009900 tx=0x7f5d0002bc40 comp rx=0 tx=0).stop 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 -- 172.21.15.22:0/76949555 shutdown_connections 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 --2- 172.21.15.22:0/76949555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1410cf10 0x7f5d14114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 --2- 172.21.15.22:0/76949555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d14108870 0x7f5d1410c830 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 --2- 172.21.15.22:0/76949555 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d14107ec0 0x7f5d141082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 -- 172.21.15.22:0/76949555 >> 172.21.15.22:0/76949555 conn(0x7f5d14077530 msgr2=0x7f5d14106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 -- 172.21.15.22:0/76949555 shutdown_connections 2024-03-24T06:49:58.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.564+0000 7f5d1a000640 1 -- 172.21.15.22:0/76949555 wait complete. 2024-03-24T06:49:58.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.566+0000 7f5d1a000640 1 Processor -- start 2024-03-24T06:49:58.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.566+0000 7f5d1a000640 1 -- start start 2024-03-24T06:49:58.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d141a2b00 0x7f5d141a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f5d141a34f0 con 0x7f5d141a2b00 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5d141a3660 con 0x7f5d14107ec0 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d1a000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d141a37d0 con 0x7f5d1410cf10 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d18c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50696/0 (socket says 172.21.15.22:50696) 2024-03-24T06:49:58.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d18c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49330/0 (socket says 172.21.15.22:49330) 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d13e00640 1 -- 172.21.15.22:0/1069643427 learned_addr learned my addr 172.21.15.22:0/1069643427 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.567+0000 7f5d19600640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d141a2b00 0x7f5d141a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 -- 172.21.15.22:0/1069643427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 msgr2=0x7f5d1419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 -- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d141a2b00 msgr2=0x7f5d141a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d141a2b00 0x7f5d141a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 -- 172.21.15.22:0/1069643427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d00009560 con 0x7f5d14107ec0 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d13e00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:58.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d18c00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e9f0 tx=0x7f5d0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d11600640 1 -- 172.21.15.22:0/1069643427 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800ccc0 con 0x7f5d14107ec0 2024-03-24T06:49:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d141aadb0 con 0x7f5d14107ec0 2024-03-24T06:49:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d11600640 1 -- 172.21.15.22:0/1069643427 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d0800ce20 con 0x7f5d14107ec0 2024-03-24T06:49:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d11600640 1 -- 172.21.15.22:0/1069643427 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08018920 con 0x7f5d14107ec0 2024-03-24T06:49:58.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.568+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d141ab250 con 0x7f5d14107ec0 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 Processor -- start 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 -- start start 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d11600640 1 -- 172.21.15.22:0/1069643427 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f5d08018a80 con 0x7f5d14107ec0 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f6413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f64c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a26f5af790 con 0x55a26f64a2d0 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a26f5accc0 con 0x55a26f640f60 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a26f5a84a0 con 0x55a26f641c40 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d19600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d19600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50704/0 (socket says 172.21.15.22:50704) 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d19600640 1 -- 172.21.15.22:0/3289444502 learned_addr learned my addr 172.21.15.22:0/3289444502 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.572+0000 7f5d18c00640 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f6413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f64c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 -- 172.21.15.22:0/3289444502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 msgr2=0x55a26f6413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f6413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 -- 172.21.15.22:0/3289444502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 msgr2=0x55a26f64c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f64c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 -- 172.21.15.22:0/3289444502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a26f569c20 con 0x55a26f641c40 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d19600640 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009f40 tx=0x7f5d0c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3731bc4667b97d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.573+0000 7f5d05c00640 1 -- 172.21.15.22:0/3289444502 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c004d20 con 0x55a26f641c40 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d05c00640 1 -- 172.21.15.22:0/3289444502 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d0c004e80 con 0x55a26f641c40 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d05c00640 1 -- 172.21.15.22:0/3289444502 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c005220 con 0x55a26f641c40 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d11600640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 0x7f5cfc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d11600640 1 -- 172.21.15.22:0/1069643427 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f5d08014070 con 0x7f5d14107ec0 2024-03-24T06:49:58.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d18c00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 0x7f5cfc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3289444502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 msgr2=0x55a26f640990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.574+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009f40 tx=0x7f5d0c0118d0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d18c00640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 0x7f5cfc0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55a26f642190 tx=0x7f5d0800cf90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3289444502 shutdown_connections 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f64c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f6413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3289444502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f640990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3289444502 >> 172.21.15.22:0/3289444502 conn(0x55a26f607b30 msgr2=0x55a26f63b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3289444502 shutdown_connections 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3289444502 wait complete. 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 -- start start 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.575+0000 7f5d1e41c740 1 Processor -- start 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f647250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f647cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a26f54cf20 con 0x55a26f64a2d0 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a26f54cbc0 con 0x55a26f640f60 2024-03-24T06:49:58.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a26f3161c0 con 0x55a26f641c40 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d1e41c740 10 client.0 _ll_register_callbacks cb 0x55a26f2b8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d19600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f647250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50718/0 (socket says 172.21.15.22:50718) 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d13e00640 1 -- 172.21.15.22:0/1338991224 learned_addr learned my addr 172.21.15.22:0/1338991224 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.576+0000 7f5d19600640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f647cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 -- 172.21.15.22:0/1338991224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 msgr2=0x55a26f647250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f647250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 -- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 msgr2=0x55a26f647cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f647cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 -- 172.21.15.22:0/1338991224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c009bf0 con 0x55a26f641c40 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d19600640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f647cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d13e00640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5d0002c180 tx=0x7f5d00004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf7e68a00a173183 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5cfa000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0002c600 con 0x55a26f641c40 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a26f2a77e0 con 0x55a26f641c40 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.577+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f5d00004df0 con 0x55a26f641c40 2024-03-24T06:49:58.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5d00004fb0 con 0x55a26f641c40 2024-03-24T06:49:58.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 10 client.27715 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:58.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a26f5a6b20 con 0x55a26f641c40 2024-03-24T06:49:58.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a26f2a77e0 con 0x55a26f641c40 2024-03-24T06:49:58.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5d0003c040 con 0x55a26f641c40 2024-03-24T06:49:58.584 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 10 client.27715 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 10 client.27715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 20 client.27715 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f65a8b0 0x55a26f65ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 4 client.27715 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.578+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a26f2e4720 con 0x55a26f65a8b0 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.579+0000 7f5d19600640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f65a8b0 0x55a26f65ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.579+0000 7f5d19600640 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f65a8b0 0x55a26f65ccd0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cef63dedd4f6163 server_cookie=65137988aed3aae1 in_seq=0 out_seq=0 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.579+0000 7f5cfa000640 10 client.27715 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:58.585 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5cfa000640 1 -- 172.21.15.22:0/1338991224 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a26f2e4720 con 0x55a26f65a8b0 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5cfa000640 10 client.27715 handle_command_reply: tid=0 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 client.27715 shutdown 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 msgr2=0x55a26f647790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5d0002c180 tx=0x7f5d00004270 comp rx=0 tx=0).stop 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 shutdown_connections 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f64a2d0 0x55a26f647cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f641c40 0x55a26f647790 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f65a8b0 0x55a26f65ccd0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/1338991224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f647250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 >> 172.21.15.22:0/1338991224 conn(0x55a26f607b30 msgr2=0x55a26f63d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5cfa000640 0 client.27715 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:58.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 shutdown_connections 2024-03-24T06:49:58.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.580+0000 7f5d1e41c740 1 -- 172.21.15.22:0/1338991224 wait complete. 2024-03-24T06:49:58.587 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.581+0000 7f5d1e41c740 20 client.27715 trim_cache size 0 max 16384 2024-03-24T06:49:58.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 Processor -- start 2024-03-24T06:49:58.600 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 -- start start 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f655c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f63aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a26f6407d0 con 0x55a26f641c40 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a26f6401e0 con 0x55a26f640f60 2024-03-24T06:49:58.601 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.600+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a26f63c9b0 con 0x55a26f64a2d0 2024-03-24T06:49:58.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d19600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.603 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f655c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d19600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49366/0 (socket says 172.21.15.22:49366) 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d19600640 1 -- 172.21.15.22:0/3684290097 learned_addr learned my addr 172.21.15.22:0/3684290097 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d19600640 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f63aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 -- 172.21.15.22:0/3684290097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 msgr2=0x55a26f63aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f63aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 -- 172.21.15.22:0/3684290097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 msgr2=0x55a26f649980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 -- 172.21.15.22:0/3684290097 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a26f5429a0 con 0x55a26f641c40 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d19600640 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.601+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f655c20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006010 tx=0x7f5d000042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913fb11c140ce6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d05200640 1 -- 172.21.15.22:0/3684290097 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0002c600 con 0x55a26f641c40 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d05200640 1 -- 172.21.15.22:0/3684290097 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d00003d70 con 0x55a26f641c40 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d05200640 1 -- 172.21.15.22:0/3684290097 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d000349b0 con 0x55a26f641c40 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3684290097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 msgr2=0x55a26f655c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f655c20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5d00006010 tx=0x7f5d000042e0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3684290097 shutdown_connections 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f63aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f655c20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3684290097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f649980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.602+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3684290097 >> 172.21.15.22:0/3684290097 conn(0x55a26f607b30 msgr2=0x55a26f63a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.603+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3684290097 shutdown_connections 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.603+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3684290097 wait complete. 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.603+0000 7f5d1e41c740 1 -- start start 2024-03-24T06:49:58.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.603+0000 7f5d1e41c740 1 Processor -- start 2024-03-24T06:49:58.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f644580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f6bfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55a26f645260 con 0x55a26f641c40 2024-03-24T06:49:58.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a26f6c0160 con 0x55a26f640f60 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a26f6c0320 con 0x55a26f64a2d0 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d1e41c740 10 client.0 _ll_register_callbacks cb 0x7f5d0c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.604+0000 7f5d19600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f644580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34262/0 (socket says 172.21.15.22:34262) 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 -- 172.21.15.22:0/3155889130 learned_addr learned my addr 172.21.15.22:0/3155889130 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d19600640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f6bfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 -- 172.21.15.22:0/3155889130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 msgr2=0x55a26f6bfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f6bfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 -- 172.21.15.22:0/3155889130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 msgr2=0x55a26f644580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f644580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 -- 172.21.15.22:0/3155889130 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c030040 con 0x55a26f641c40 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d19600640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f644580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5d13e00640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d00003b80 tx=0x7f5d00003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af21e9957a57c68c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.605+0000 7f5cfa000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00003ee0 con 0x55a26f641c40 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f5d00000e40 con 0x55a26f641c40 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a26f6418e0 con 0x55a26f641c40 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5d0002c600 con 0x55a26f641c40 2024-03-24T06:49:58.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5d1e41c740 10 client.17331 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a26f5a6b20 con 0x55a26f641c40 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a26f6418e0 con 0x55a26f641c40 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.606+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f5d0003c030 con 0x55a26f641c40 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 10 client.17331 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 10 client.17331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 20 client.17331 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f6c1830 0x55a26f6c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 4 client.17331 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:49:58.613 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a26f5a0160 con 0x55a26f6c1830 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d19600640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f6c1830 0x55a26f6c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5d19600640 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f6c1830 0x55a26f6c3c50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e06a5e214cf3b6d server_cookie=d1e8104a6d6e3cd6 in_seq=0 out_seq=0 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.607+0000 7f5cfa000640 10 client.17331 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5cfa000640 1 -- 172.21.15.22:0/3155889130 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a26f5a0160 con 0x55a26f6c1830 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5cfa000640 10 client.17331 handle_command_reply: tid=0 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 client.17331 shutdown 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 msgr2=0x55a26f644ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5d00003b80 tx=0x7f5d00003820 comp rx=0 tx=0).stop 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 shutdown_connections 2024-03-24T06:49:58.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a26f64a2d0 0x55a26f6bfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55a26f641c40 0x55a26f644ac0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55a26f6c1830 0x55a26f6c3c50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 --2- 172.21.15.22:0/3155889130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a26f640f60 0x55a26f644580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 >> 172.21.15.22:0/3155889130 conn(0x55a26f607b30 msgr2=0x55a26f63abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5cfa000640 0 client.17331 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 shutdown_connections 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 1 -- 172.21.15.22:0/3155889130 wait complete. 2024-03-24T06:49:58.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.608+0000 7f5d1e41c740 20 client.17331 trim_cache size 0 max 16384 2024-03-24T06:49:58.616 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:49:58.616 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:49:58.616 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:49:58.616 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 msgr2=0x7f5cfc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 0x7f5cfc0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55a26f642190 tx=0x7f5d0800cf90 comp rx=0 tx=0).stop 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 msgr2=0x7f5d1419e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800e9f0 tx=0x7f5d0800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 shutdown_connections 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f5cfc0a22b0 0x7f5cfc0a4770 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d141a2b00 0x7f5d141a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d1410cf10 0x7f5d1419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 --2- 172.21.15.22:0/1069643427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d14107ec0 0x7f5d1419e290 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 >> 172.21.15.22:0/1069643427 conn(0x7f5d14077530 msgr2=0x7f5d14079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 shutdown_connections 2024-03-24T06:49:58.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.609+0000 7f5d1a000640 1 -- 172.21.15.22:0/1069643427 wait complete. 2024-03-24T06:49:58.624 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:58 smithi022 ceph-mon[32003]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:58.628 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:49:58.628 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:49:58.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:58 smithi143 ceph-mon[36520]: pgmap v309: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:49:58.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 -- 172.21.15.22:0/1394100875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 msgr2=0x7f41b8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 --2- 172.21.15.22:0/1394100875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b8107a90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009890 tx=0x7f41a802bb70 comp rx=0 tx=0).stop 2024-03-24T06:49:58.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 -- 172.21.15.22:0/1394100875 shutdown_connections 2024-03-24T06:49:58.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 --2- 172.21.15.22:0/1394100875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 0x7f41b810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 --2- 172.21.15.22:0/1394100875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 --2- 172.21.15.22:0/1394100875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b8107a90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.840+0000 7f41bd600640 1 -- 172.21.15.22:0/1394100875 >> 172.21.15.22:0/1394100875 conn(0x7f41b80fd1b0 msgr2=0x7f41b80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:58.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.841+0000 7f41bd600640 1 -- 172.21.15.22:0/1394100875 shutdown_connections 2024-03-24T06:49:58.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.841+0000 7f41bd600640 1 -- 172.21.15.22:0/1394100875 wait complete. 2024-03-24T06:49:58.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 Processor -- start 2024-03-24T06:49:58.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 -- start start 2024-03-24T06:49:58.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.844 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 0x7f41b81982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f41b819cbf0 con 0x7f41b81076b0 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f41b819cd60 con 0x7f41b81016a0 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41bd600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f41b819cf20 con 0x7f41b8102060 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41b7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49402/0 (socket says 172.21.15.22:49402) 2024-03-24T06:49:58.845 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41b6a00640 1 -- 172.21.15.22:0/1398085784 learned_addr learned my addr 172.21.15.22:0/1398085784 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.842+0000 7f41b6000640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 0x7f41b81982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 -- 172.21.15.22:0/1398085784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 msgr2=0x7f41b81982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 0x7f41b81982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 -- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 msgr2=0x7f41b819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 -- 172.21.15.22:0/1398085784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a4009560 con 0x7f41b81016a0 2024-03-24T06:49:58.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41b6a00640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009860 tx=0x7f41a8004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8034950 con 0x7f41b81016a0 2024-03-24T06:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.843+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a802c9f0 con 0x7f41b81016a0 2024-03-24T06:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.844+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a8009490 con 0x7f41b81016a0 2024-03-24T06:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.844+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8032600 con 0x7f41b81016a0 2024-03-24T06:49:58.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.844+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b819d3d0 con 0x7f41b81016a0 2024-03-24T06:49:58.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.845+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b8068950 con 0x7f41b81016a0 2024-03-24T06:49:58.848 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.847+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f41a803b040 con 0x7f41b81016a0 2024-03-24T06:49:58.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.849+0000 7f41af400640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 0x7f419c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:58.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.849+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f41a80efda0 con 0x7f41b81016a0 2024-03-24T06:49:58.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.849+0000 7f41b6000640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 0x7f419c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:58.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.850+0000 7f41b6000640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 0x7f419c0a4770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41b80ff880 tx=0x7f41a4009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:58.852 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:58.852+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f41a80b87e0 con 0x7f41b81016a0 2024-03-24T06:49:59.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.098+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41b8109c40 con 0x7f41b81016a0 2024-03-24T06:49:59.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.099+0000 7f41af400640 1 -- 172.21.15.22:0/1398085784 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f41a80bbea0 con 0x7f41b81016a0 2024-03-24T06:49:59.100 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:49:59.101 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:49:59.101 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 msgr2=0x7f419c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 0x7f419c0a4770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41b80ff880 tx=0x7f41a4009260 comp rx=0 tx=0).stop 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 msgr2=0x7f41b8197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009860 tx=0x7f41a8004530 comp rx=0 tx=0).stop 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 shutdown_connections 2024-03-24T06:49:59.103 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f419c0a22b0 0x7f419c0a4770 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41b81076b0 0x7f41b819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b8102060 0x7f41b81982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 --2- 172.21.15.22:0/1398085784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41b81016a0 0x7f41b8197d80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 >> 172.21.15.22:0/1398085784 conn(0x7f41b80fd1b0 msgr2=0x7f41b80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 shutdown_connections 2024-03-24T06:49:59.104 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.102+0000 7f41bd600640 1 -- 172.21.15.22:0/1398085784 wait complete. 2024-03-24T06:49:59.122 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:49:59.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 -- 172.21.15.22:0/2433252623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c073b40 msgr2=0x7fde5c073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 --2- 172.21.15.22:0/2433252623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c073b40 0x7fde5c073f20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fde4c009ca0 tx=0x7fde4c02c190 comp rx=0 tx=0).stop 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 -- 172.21.15.22:0/2433252623 shutdown_connections 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 --2- 172.21.15.22:0/2433252623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 0x7fde5c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 --2- 172.21.15.22:0/2433252623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c074460 0x7fde5c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 --2- 172.21.15.22:0/2433252623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c073b40 0x7fde5c073f20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.337+0000 7fde62e00640 1 -- 172.21.15.22:0/2433252623 >> 172.21.15.22:0/2433252623 conn(0x7fde5c0fd150 msgr2=0x7fde5c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.338+0000 7fde62e00640 1 -- 172.21.15.22:0/2433252623 shutdown_connections 2024-03-24T06:49:59.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.338+0000 7fde62e00640 1 -- 172.21.15.22:0/2433252623 wait complete. 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 Processor -- start 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 -- start start 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c073b40 0x7fde5c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 0x7fde5c19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c19eeb0 con 0x7fde5c073b40 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c19f020 con 0x7fde5c10dee0 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde62e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde5c19f1e0 con 0x7fde5c074460 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50780/0 (socket says 172.21.15.22:50780) 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde5be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c073b40 0x7fde5c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.339+0000 7fde53400640 1 -- 172.21.15.22:0/1396278377 learned_addr learned my addr 172.21.15.22:0/1396278377 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde60e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 0x7fde5c19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 -- 172.21.15.22:0/1396278377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 msgr2=0x7fde5c19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 0x7fde5c19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 -- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c073b40 msgr2=0x7fde5c19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c073b40 0x7fde5c19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 -- 172.21.15.22:0/1396278377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4c009900 con 0x7fde5c074460 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde53400640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fde4400e9f0 tx=0x7fde4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.340+0000 7fde59600640 1 -- 172.21.15.22:0/1396278377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44009700 con 0x7fde5c074460 2024-03-24T06:49:59.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.341+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5c19f4c0 con 0x7fde5c074460 2024-03-24T06:49:59.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.341+0000 7fde59600640 1 -- 172.21.15.22:0/1396278377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde4400cd80 con 0x7fde5c074460 2024-03-24T06:49:59.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.341+0000 7fde59600640 1 -- 172.21.15.22:0/1396278377 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44018920 con 0x7fde5c074460 2024-03-24T06:49:59.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.341+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5c1a6d40 con 0x7fde5c074460 2024-03-24T06:49:59.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 Processor -- start 2024-03-24T06:49:59.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde59600640 1 -- 172.21.15.22:0/1396278377 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fde44018a80 con 0x7fde5c074460 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 -- start start 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f5a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f5cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644b5ebf6e0 con 0x5644b5f51ba0 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644b5ebcc10 con 0x5644b5f5a780 2024-03-24T06:49:59.351 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.344+0000 7fde65b87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5644b5eb83f0 con 0x5644b5f51280 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde5be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f5a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50796/0 (socket says 172.21.15.22:50796) 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 -- 172.21.15.22:0/3707434195 learned_addr learned my addr 172.21.15.22:0/3707434195 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde53400640 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f5cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 -- 172.21.15.22:0/3707434195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 msgr2=0x5644b5f5cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f5cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 -- 172.21.15.22:0/3707434195 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 msgr2=0x5644b5f5a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f5a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde60e00640 1 -- 172.21.15.22:0/3707434195 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b5e79e40 con 0x5644b5f51280 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.345+0000 7fde5be00640 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f5a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde60e00640 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fde54009c20 tx=0x7fde54011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3270cff44f34550 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde51600640 1 -- 172.21.15.22:0/3707434195 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54019630 con 0x5644b5f51280 2024-03-24T06:49:59.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde51600640 1 -- 172.21.15.22:0/3707434195 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde54009e90 con 0x5644b5f51280 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde59600640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 0x7fde400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde51600640 1 -- 172.21.15.22:0/3707434195 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400eb00 con 0x5644b5f51280 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde59600640 1 -- 172.21.15.22:0/1396278377 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fde44014070 con 0x7fde5c074460 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.346+0000 7fde5be00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 0x7fde400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 -- 172.21.15.22:0/3707434195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 msgr2=0x5644b5f51660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fde54009c20 tx=0x7fde54011810 comp rx=0 tx=0).stop 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 -- 172.21.15.22:0/3707434195 shutdown_connections 2024-03-24T06:49:59.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f5cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f5a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 --2- 172.21.15.22:0/3707434195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f51660 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 -- 172.21.15.22:0/3707434195 >> 172.21.15.22:0/3707434195 conn(0x5644b5f4b9b0 msgr2=0x5644b5f4bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 -- 172.21.15.22:0/3707434195 shutdown_connections 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde65b87740 1 -- 172.21.15.22:0/3707434195 wait complete. 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.347+0000 7fde5be00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 0x7fde400a46f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fde4c004af0 tx=0x7fde4c005c50 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 -- start start 2024-03-24T06:49:59.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 Processor -- start 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 0x5644b5f57880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f57dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644b5e5d060 con 0x5644b5f51ba0 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644b5e5cd00 con 0x5644b5f51280 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5644b5c261c0 con 0x5644b5f5a780 2024-03-24T06:49:59.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.348+0000 7fde65b87740 10 client.0 _ll_register_callbacks cb 0x5644b5bc8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde53400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f57dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde5be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 0x5644b5f57880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50812/0 (socket says 172.21.15.22:50812) 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 -- 172.21.15.22:0/2974299373 learned_addr learned my addr 172.21.15.22:0/2974299373 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 -- 172.21.15.22:0/2974299373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 msgr2=0x5644b5f57880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 0x5644b5f57880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 -- 172.21.15.22:0/2974299373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 msgr2=0x5644b5f57dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f57dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde60e00640 1 -- 172.21.15.22:0/2974299373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde540098d0 con 0x5644b5f5a780 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde53400640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f57dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.349+0000 7fde5be00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 0x5644b5f57880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde60e00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fde54011d10 tx=0x7fde54005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cee64899a5baed96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde4a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:59.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400ec50 con 0x5644b5f5a780 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5644b5bb77e0 con 0x5644b5f5a780 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fde5400edb0 con 0x5644b5f5a780 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde54004020 con 0x5644b5f5a780 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde65b87740 10 client.27739 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5644b5eb6a70 con 0x5644b5f5a780 2024-03-24T06:49:59.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.350+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5644b5bb77e0 con 0x5644b5f5a780 2024-03-24T06:49:59.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fde540035f0 con 0x5644b5f5a780 2024-03-24T06:49:59.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 10 client.27739 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 10 client.27739 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 20 client.27739 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5f6abf0 0x5644b5f6d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 4 client.27739 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644b5bf4720 con 0x5644b5f6abf0 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.351+0000 7fde5be00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5f6abf0 0x5644b5f6d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.352+0000 7fde5be00640 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5f6abf0 0x5644b5f6d010 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a6777bbf42f2343 server_cookie=970f71289160d5b3 in_seq=0 out_seq=0 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.352+0000 7fde4a000640 10 client.27739 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.352+0000 7fde4a000640 1 -- 172.21.15.22:0/2974299373 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5644b5bf4720 con 0x5644b5f6abf0 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.352+0000 7fde4a000640 10 client.27739 handle_command_reply: tid=0 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.352+0000 7fde65b87740 1 client.27739 shutdown 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 msgr2=0x5644b5f58300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fde54011d10 tx=0x7fde54005750 comp rx=0 tx=0).stop 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 shutdown_connections 2024-03-24T06:49:59.363 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5f6abf0 0x5644b5f6d010 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f5a780 0x5644b5f58300 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f57dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 --2- 172.21.15.22:0/2974299373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f51280 0x5644b5f57880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde4a000640 0 client.27739 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 >> 172.21.15.22:0/2974299373 conn(0x5644b5f4b9b0 msgr2=0x5644b5f4d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 shutdown_connections 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 1 -- 172.21.15.22:0/2974299373 wait complete. 2024-03-24T06:49:59.364 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.353+0000 7fde65b87740 20 client.27739 trim_cache size 0 max 16384 2024-03-24T06:49:59.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 Processor -- start 2024-03-24T06:49:59.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 -- start start 2024-03-24T06:49:59.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f66010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f58120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f47330 con 0x5644b5f51ba0 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f4c9e0 con 0x5644b5f5a780 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde65b87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f66550 con 0x5644b5f51280 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde60e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde60e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49452/0 (socket says 172.21.15.22:49452) 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde60e00640 1 -- 172.21.15.22:0/4188592039 learned_addr learned my addr 172.21.15.22:0/4188592039 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.370 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde53400640 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f58120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.368+0000 7fde5be00640 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f66010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 -- 172.21.15.22:0/4188592039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 msgr2=0x5644b5f66010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f66010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 -- 172.21.15.22:0/4188592039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 msgr2=0x5644b5f58120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f58120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 -- 172.21.15.22:0/4188592039 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b5e52b00 con 0x5644b5f5a780 2024-03-24T06:49:59.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde60e00640 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde54009800 tx=0x7fde54009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4c42920a1c24b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde50c00640 1 -- 172.21.15.22:0/4188592039 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400e3b0 con 0x5644b5f5a780 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde50c00640 1 -- 172.21.15.22:0/4188592039 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde54011d40 con 0x5644b5f5a780 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.369+0000 7fde50c00640 1 -- 172.21.15.22:0/4188592039 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde540041b0 con 0x5644b5f5a780 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 -- 172.21.15.22:0/4188592039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 msgr2=0x5644b5f4a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fde54009800 tx=0x7fde54009620 comp rx=0 tx=0).stop 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 -- 172.21.15.22:0/4188592039 shutdown_connections 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f4a8f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.377 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51ba0 0x5644b5f58120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 --2- 172.21.15.22:0/4188592039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51280 0x5644b5f66010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 -- 172.21.15.22:0/4188592039 >> 172.21.15.22:0/4188592039 conn(0x5644b5f4b9b0 msgr2=0x5644b5f4a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 -- 172.21.15.22:0/4188592039 shutdown_connections 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.370+0000 7fde65b87740 1 -- 172.21.15.22:0/4188592039 wait complete. 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 -- start start 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 Processor -- start 2024-03-24T06:49:59.378 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 0x5644b5f53f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f54440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f62d80 con 0x5644b5f51280 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f54980 con 0x5644b5f5a780 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5644b5f54b40 con 0x5644b5f51ba0 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde65b87740 10 client.0 _ll_register_callbacks cb 0x7fde5401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.379 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde5be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.371+0000 7fde53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 0x5644b5f53f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde60e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f54440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 0x5644b5f53f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50826/0 (socket says 172.21.15.22:50826) 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34338/0 (socket says 172.21.15.22:34338) 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde53400640 1 -- 172.21.15.22:0/225233068 learned_addr learned my addr 172.21.15.22:0/225233068 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 -- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 msgr2=0x5644b5f53f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 0x5644b5f53f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 -- 172.21.15.22:0/225233068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 msgr2=0x5644b5f54440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f54440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 -- 172.21.15.22:0/225233068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b5eb6a70 con 0x5644b5f51280 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde60e00640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f54440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde5be00640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fde4c006010 tx=0x7fde4c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60daa103df99011a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.372+0000 7fde4a000640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c04f040 con 0x5644b5f51280 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fde4c05f420 con 0x5644b5f51280 2024-03-24T06:49:59.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde54009e90 con 0x5644b5f51280 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde4c05f5e0 con 0x5644b5f51280 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde65b87740 10 client.17355 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5644b5e786a0 con 0x5644b5f51280 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde54009e90 con 0x5644b5f51280 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fde4c049840 con 0x5644b5f51280 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde65b87740 10 client.17355 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.373+0000 7fde65b87740 10 client.17355 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:49:59.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde65b87740 20 client.17355 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5fd12e0 0x5644b5fd3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde65b87740 4 client.17355 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644b5c43890 con 0x5644b5fd12e0 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde53400640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5fd12e0 0x5644b5fd3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde4a000640 10 client.17355 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.374+0000 7fde53400640 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5fd12e0 0x5644b5fd3700 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=173d1a44fc1b406f server_cookie=84b8d739f3933873 in_seq=0 out_seq=0 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde4a000640 1 -- 172.21.15.22:0/225233068 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644b5c43890 con 0x5644b5fd12e0 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde4a000640 10 client.17355 handle_command_reply: tid=0 2024-03-24T06:49:59.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 client.17355 shutdown 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 msgr2=0x5644b5f62640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fde4c006010 tx=0x7fde4c0095c0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 shutdown_connections 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x5644b5fd12e0 0x5644b5fd3700 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5644b5f5a780 0x5644b5f54440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5644b5f51ba0 0x5644b5f53f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 --2- 172.21.15.22:0/225233068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5644b5f51280 0x5644b5f62640 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde4a000640 0 client.17355 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:49:59.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 >> 172.21.15.22:0/225233068 conn(0x5644b5f4b9b0 msgr2=0x5644b5f4aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 shutdown_connections 2024-03-24T06:49:59.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 1 -- 172.21.15.22:0/225233068 wait complete. 2024-03-24T06:49:59.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.375+0000 7fde65b87740 20 client.17355 trim_cache size 0 max 16384 2024-03-24T06:49:59.385 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 msgr2=0x7fde400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 0x7fde400a46f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fde4c004af0 tx=0x7fde4c005c50 comp rx=0 tx=0).stop 2024-03-24T06:49:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 msgr2=0x7fde5c19a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.386 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fde4400e9f0 tx=0x7fde4400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 shutdown_connections 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fde400a2230 0x7fde400a46f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5c10dee0 0x7fde5c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde5c074460 0x7fde5c19a580 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 --2- 172.21.15.22:0/1396278377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde5c073b40 0x7fde5c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 >> 172.21.15.22:0/1396278377 conn(0x7fde5c0fd150 msgr2=0x7fde5c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 shutdown_connections 2024-03-24T06:49:59.387 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.376+0000 7fde62e00640 1 -- 172.21.15.22:0/1396278377 wait complete. 2024-03-24T06:49:59.394 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:49:59 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1398085784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:59.398 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:49:59.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:49:59 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1398085784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:59.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 -- 172.21.15.22:0/1144545860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278100530 msgr2=0x7f7278100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 --2- 172.21.15.22:0/1144545860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278100530 0x7f7278100910 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7268009ca0 tx=0x7f726802c1d0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 -- 172.21.15.22:0/1144545860 shutdown_connections 2024-03-24T06:49:59.604 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 --2- 172.21.15.22:0/1144545860 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278105930 0x7f7278105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 --2- 172.21.15.22:0/1144545860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100e50 0x7f7278105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 --2- 172.21.15.22:0/1144545860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278100530 0x7f7278100910 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 -- 172.21.15.22:0/1144545860 >> 172.21.15.22:0/1144545860 conn(0x7f72780fbf60 msgr2=0x7f72780fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 -- 172.21.15.22:0/1144545860 shutdown_connections 2024-03-24T06:49:59.605 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.604+0000 7f7277e00640 1 -- 172.21.15.22:0/1144545860 wait complete. 2024-03-24T06:49:59.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.605+0000 7f7277e00640 1 Processor -- start 2024-03-24T06:49:59.606 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 -- start start 2024-03-24T06:49:59.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278100e50 0x7f727819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 0x7f72781a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f72781a30d0 con 0x7f7278100e50 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f72781a3240 con 0x7f7278105930 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.606+0000 7f7277e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f72781a3400 con 0x7f7278100530 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50830/0 (socket says 172.21.15.22:50830) 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 -- 172.21.15.22:0/3198509835 learned_addr learned my addr 172.21.15.22:0/3198509835 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f726e000640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278100e50 0x7f727819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7277400640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 0x7f72781a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 -- 172.21.15.22:0/3198509835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 msgr2=0x7f72781a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 0x7f72781a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 -- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278100e50 msgr2=0x7f727819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278100e50 0x7f727819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 -- 172.21.15.22:0/3198509835 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7268009900 con 0x7f7278100530 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7277400640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 0x7f72781a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.609 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.607+0000 7f7276a00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7268005e30 tx=0x7f7268034830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.608+0000 7f726fe00640 1 -- 172.21.15.22:0/3198509835 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268003d80 con 0x7f7278100530 2024-03-24T06:49:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.608+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72781a3680 con 0x7f7278100530 2024-03-24T06:49:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.608+0000 7f726fe00640 1 -- 172.21.15.22:0/3198509835 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7268032c60 con 0x7f7278100530 2024-03-24T06:49:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.608+0000 7f726fe00640 1 -- 172.21.15.22:0/3198509835 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268004860 con 0x7f7278100530 2024-03-24T06:49:59.610 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.608+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72781a3950 con 0x7f7278100530 2024-03-24T06:49:59.611 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 Processor -- start 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 -- start start 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f726fe00640 1 -- 172.21.15.22:0/3198509835 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f726800a040 con 0x7f7278100530 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884be3450 0x564884be3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884bec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 0x564884beed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564884b516e0 con 0x564884be3450 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564884b4ec10 con 0x564884bec950 2024-03-24T06:49:59.612 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.611+0000 7f72809e5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564884b4a3f0 con 0x564884be3d70 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f7277400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884be3450 0x564884be3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884bec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 0x564884beed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884bec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50832/0 (socket says 172.21.15.22:50832) 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f7276a00640 1 -- 172.21.15.22:0/2648970662 learned_addr learned my addr 172.21.15.22:0/2648970662 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 -- 172.21.15.22:0/2648970662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 msgr2=0x564884bec410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884bec410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 -- 172.21.15.22:0/2648970662 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884be3450 msgr2=0x564884be3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884be3450 0x564884be3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.612+0000 7f726e000640 1 -- 172.21.15.22:0/2648970662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564884b0be40 con 0x564884bec950 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f726e000640 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 0x564884beed40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7260009bc0 tx=0x7f7260011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=704f71389de85f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f7267e00640 1 -- 172.21.15.22:0/2648970662 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260004230 con 0x564884bec950 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f7267e00640 1 -- 172.21.15.22:0/2648970662 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7260009e30 con 0x564884bec950 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f7267e00640 1 -- 172.21.15.22:0/2648970662 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260004db0 con 0x564884bec950 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f72809e5740 1 -- 172.21.15.22:0/2648970662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 msgr2=0x564884beed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.613+0000 7f72809e5740 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 0x564884beed40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7260009bc0 tx=0x7f7260011810 comp rx=0 tx=0).stop 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 -- 172.21.15.22:0/2648970662 shutdown_connections 2024-03-24T06:49:59.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884bec950 0x564884beed40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884bec410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 --2- 172.21.15.22:0/2648970662 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884be3450 0x564884be3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 -- 172.21.15.22:0/2648970662 >> 172.21.15.22:0/2648970662 conn(0x564884bddb80 msgr2=0x564884bddf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 -- 172.21.15.22:0/2648970662 shutdown_connections 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 -- 172.21.15.22:0/2648970662 wait complete. 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f726fe00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 0x7f725c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f726fe00640 1 -- 172.21.15.22:0/3198509835 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f72680f0c70 con 0x7f7278100530 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f72809e5740 1 -- start start 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.614+0000 7f726e000640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 0x7f725c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 Processor -- start 2024-03-24T06:49:59.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f726e000640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 0x7f725c0a4600 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f726000a010 tx=0x7f7260009fa0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884be9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564884aef060 con 0x564884bec950 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564884aeed00 con 0x564884be3450 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648848b81c0 con 0x564884be3d70 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.615+0000 7f72809e5740 10 client.0 _ll_register_callbacks cb 0x56488485abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49496/0 (socket says 172.21.15.22:49496) 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 -- 172.21.15.22:0/2746383553 learned_addr learned my addr 172.21.15.22:0/2746383553 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f726e000640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884be9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7277400640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 -- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 msgr2=0x564884be9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884be9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 -- 172.21.15.22:0/2746383553 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 msgr2=0x564884bea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 -- 172.21.15.22:0/2746383553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7260009820 con 0x564884be3450 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7276a00640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f72680b9d30 tx=0x7f72680095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfceeb8e545cd4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.616+0000 7f7264c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680bcea0 con 0x564884be3450 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7268032dd0 con 0x564884be3450 2024-03-24T06:49:59.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648848497e0 con 0x564884be3450 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f72680bccc0 con 0x564884be3450 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f72809e5740 10 client.27731 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564884b48a70 con 0x564884be3450 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5648848497e0 con 0x564884be3450 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f72680c1120 con 0x564884be3450 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f72809e5740 10 client.27731 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.617+0000 7f72809e5740 10 client.27731 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.618+0000 7f72809e5740 20 client.27731 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.618+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884bfcbb0 0x564884bfeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.618+0000 7f72809e5740 4 client.27731 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.618+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564884886720 con 0x564884bfcbb0 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.618+0000 7f726e000640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884bfcbb0 0x564884bfeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.619+0000 7f7264c00640 10 client.27731 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.619+0000 7f726e000640 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884bfcbb0 0x564884bfeff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d876188680b358e7 server_cookie=508059ca730ded35 in_seq=0 out_seq=0 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.619+0000 7f7264c00640 1 -- 172.21.15.22:0/2746383553 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564884886720 con 0x564884bfcbb0 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.619+0000 7f7264c00640 10 client.27731 handle_command_reply: tid=0 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.619+0000 7f72809e5740 1 client.27731 shutdown 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 msgr2=0x564884be9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f72680b9d30 tx=0x7f72680095c0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 shutdown_connections 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884bfcbb0 0x564884bfeff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3d70 0x564884be9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f7264c00640 0 client.27731 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 --2- 172.21.15.22:0/2746383553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3450 0x564884be9980 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 >> 172.21.15.22:0/2746383553 conn(0x564884bddb80 msgr2=0x564884bdf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 shutdown_connections 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 1 -- 172.21.15.22:0/2746383553 wait complete. 2024-03-24T06:49:59.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.620+0000 7f72809e5740 20 client.27731 trim_cache size 0 max 16384 2024-03-24T06:49:59.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 Processor -- start 2024-03-24T06:49:59.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 -- start start 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884bf4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bdf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564884bdd440 con 0x564884bec950 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564884bdd9f0 con 0x564884be3d70 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.635+0000 7f72809e5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564884bdeb80 con 0x564884be3450 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.637 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884bf4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50850/0 (socket says 172.21.15.22:50850) 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f7276a00640 1 -- 172.21.15.22:0/2365722290 learned_addr learned my addr 172.21.15.22:0/2365722290 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f7277400640 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bdf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 -- 172.21.15.22:0/2365722290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 msgr2=0x564884c04900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 -- 172.21.15.22:0/2365722290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 msgr2=0x564884bdf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bdf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.638 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 -- 172.21.15.22:0/2365722290 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564884ae4b00 con 0x564884be3d70 2024-03-24T06:49:59.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f7276a00640 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.636+0000 7f726e000640 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884bf4800 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f72600057c0 tx=0x7f7260022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d464de17037895 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f7267400640 1 -- 172.21.15.22:0/2365722290 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726003d070 con 0x564884be3d70 2024-03-24T06:49:59.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f7267400640 1 -- 172.21.15.22:0/2365722290 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7260004500 con 0x564884be3d70 2024-03-24T06:49:59.639 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f7267400640 1 -- 172.21.15.22:0/2365722290 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260032690 con 0x564884be3d70 2024-03-24T06:49:59.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 -- 172.21.15.22:0/2365722290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 msgr2=0x564884bf4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884bf4800 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f72600057c0 tx=0x7f7260022000 comp rx=0 tx=0).stop 2024-03-24T06:49:59.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 -- 172.21.15.22:0/2365722290 shutdown_connections 2024-03-24T06:49:59.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884bdf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884bf4800 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 --2- 172.21.15.22:0/2365722290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884c04900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 -- 172.21.15.22:0/2365722290 >> 172.21.15.22:0/2365722290 conn(0x564884bddb80 msgr2=0x564884bdeed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 -- 172.21.15.22:0/2365722290 shutdown_connections 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.637+0000 7f72809e5740 1 -- 172.21.15.22:0/2365722290 wait complete. 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.638+0000 7f72809e5740 1 -- start start 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.638+0000 7f72809e5740 1 Processor -- start 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.638+0000 7f72809e5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884be6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.644 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884be72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x564884c62120 con 0x564884bec950 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564884c62290 con 0x564884be3d70 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564884c62450 con 0x564884be3450 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f72809e5740 10 client.0 _ll_register_callbacks cb 0x7f72680f43d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f726e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884be6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f726e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49512/0 (socket says 172.21.15.22:49512) 2024-03-24T06:49:59.645 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f726e000640 1 -- 172.21.15.22:0/1772408450 learned_addr learned my addr 172.21.15.22:0/1772408450 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7277400640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884be72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 -- 172.21.15.22:0/1772408450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 msgr2=0x564884be6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 -- 172.21.15.22:0/1772408450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 msgr2=0x564884be72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884be72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f7276a00640 1 -- 172.21.15.22:0/1772408450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564884b48a70 con 0x564884be3450 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.639+0000 7f726e000640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.646 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7276a00640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884be6850 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7268004af0 tx=0x7f72680c1350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9160ed601f7f1e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680c5870 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7268032dd0 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7260009e30 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f72680bccc0 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f72809e5740 10 client.27763 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x7f726001d050 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72680d8020 con 0x564884be3450 2024-03-24T06:49:59.647 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.640+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7268004b40 con 0x564884be3450 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 10 client.27763 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 10 client.27763 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 20 client.27763 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884c63890 0x564884c65c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 4 client.27763 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648848d5890 con 0x564884c63890 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.641+0000 7f726e000640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884c63890 0x564884c65c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f726e000640 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884c63890 0x564884c65c80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=17f7d5683fd883f2 server_cookie=1724cbe704fa7f2a in_seq=0 out_seq=0 2024-03-24T06:49:59.648 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f7264c00640 10 client.27763 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f7264c00640 1 -- 172.21.15.22:0/1772408450 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648848d5890 con 0x564884c63890 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f7264c00640 10 client.27763 handle_command_reply: tid=0 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 client.27763 shutdown 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 msgr2=0x564884be6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884be6850 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7268004af0 tx=0x7f72680c1350 comp rx=0 tx=0).stop 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 shutdown_connections 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x564884c63890 0x564884c65c80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564884bec950 0x564884be72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.649 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564884be3d70 0x564884be6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f7264c00640 0 client.27763 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 --2- 172.21.15.22:0/1772408450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564884be3450 0x564884be6850 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 >> 172.21.15.22:0/1772408450 conn(0x564884bddb80 msgr2=0x564884bdccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 shutdown_connections 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 1 -- 172.21.15.22:0/1772408450 wait complete. 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.642+0000 7f72809e5740 20 client.27763 trim_cache size 0 max 16384 2024-03-24T06:49:59.650 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:59.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 msgr2=0x7f725c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 0x7f725c0a4600 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f726000a010 tx=0x7f7260009fa0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 msgr2=0x7f727819e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.651 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7268005e30 tx=0x7f7268034830 comp rx=0 tx=0).stop 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 shutdown_connections 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f725c0a2140 0x7f725c0a4600 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7278105930 0x7f72781a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7278100e50 0x7f727819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 --2- 172.21.15.22:0/3198509835 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7278100530 0x7f727819e210 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 >> 172.21.15.22:0/3198509835 conn(0x7f72780fbf60 msgr2=0x7f72781032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 shutdown_connections 2024-03-24T06:49:59.652 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.643+0000 7f7277e00640 1 -- 172.21.15.22:0/3198509835 wait complete. 2024-03-24T06:49:59.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:49:59.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:49:59 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1398085784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:49:59.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 -- 172.21.15.22:0/1455354771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 msgr2=0x7fb1ec100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 --2- 172.21.15.22:0/1455354771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec100900 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009900 tx=0x7fb1d402bc40 comp rx=0 tx=0).stop 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 -- 172.21.15.22:0/1455354771 shutdown_connections 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 --2- 172.21.15.22:0/1455354771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec1104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 --2- 172.21.15.22:0/1455354771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec100e40 0x7fb1ec10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 --2- 172.21.15.22:0/1455354771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec100900 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 -- 172.21.15.22:0/1455354771 >> 172.21.15.22:0/1455354771 conn(0x7fb1ec077770 msgr2=0x7fb1ec077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 -- 172.21.15.22:0/1455354771 shutdown_connections 2024-03-24T06:49:59.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.892+0000 7fb1f1000640 1 -- 172.21.15.22:0/1455354771 wait complete. 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.893+0000 7fb1f1000640 1 Processor -- start 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.893+0000 7fb1f1000640 1 -- start start 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec100e40 0x7fb1ec19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec19eeb0 con 0x7fb1ec10e0d0 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec19f020 con 0x7fb1ec100520 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1f1000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec19f1e0 con 0x7fb1ec100e40 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec100e40 0x7fb1ec19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49540/0 (socket says 172.21.15.22:49540) 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9e00640 1 -- 172.21.15.22:0/4165797356 learned_addr learned my addr 172.21.15.22:0/4165797356 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1ea800640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9e00640 1 -- 172.21.15.22:0/4165797356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec100e40 msgr2=0x7fb1ec19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.894+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec100e40 0x7fb1ec19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e9e00640 1 -- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 msgr2=0x7fb1ec19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e9e00640 1 -- 172.21.15.22:0/4165797356 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d4009560 con 0x7fb1ec100520 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1ea800640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000c00 tx=0x7fb1d4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e2a00640 1 -- 172.21.15.22:0/4165797356 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4004500 con 0x7fb1ec100520 2024-03-24T06:49:59.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e2a00640 1 -- 172.21.15.22:0/4165797356 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d40052d0 con 0x7fb1ec100520 2024-03-24T06:49:59.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1e2a00640 1 -- 172.21.15.22:0/4165797356 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4003e50 con 0x7fb1ec100520 2024-03-24T06:49:59.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec19f460 con 0x7fb1ec100520 2024-03-24T06:49:59.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.895+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec1a6d40 con 0x7fb1ec100520 2024-03-24T06:49:59.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.898+0000 7fb1f3db7740 1 Processor -- start 2024-03-24T06:49:59.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.898+0000 7fb1e2a00640 1 -- 172.21.15.22:0/4165797356 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb1d40035f0 con 0x7fb1ec100520 2024-03-24T06:49:59.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 -- start start 2024-03-24T06:49:59.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923259240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f92325bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f9231be6e0 con 0x55f923259780 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f9231bbc10 con 0x55f923250ba0 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f9231b73f0 con 0x55f923250280 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1e9e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923259240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1e9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f92325bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50886/0 (socket says 172.21.15.22:50886) 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.899+0000 7fb1ea800640 1 -- 172.21.15.22:0/1829313745 learned_addr learned my addr 172.21.15.22:0/1829313745 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 -- 172.21.15.22:0/1829313745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 msgr2=0x55f923259240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923259240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 -- 172.21.15.22:0/1829313745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 msgr2=0x55f92325bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f92325bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 -- 172.21.15.22:0/1829313745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f923178e40 con 0x55f923250280 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923259240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400dc00 tx=0x7fb1e400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574d5955d114351 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1e2a00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 0x7fb1d00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1db400640 1 -- 172.21.15.22:0/1829313745 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4004020 con 0x55f923250280 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1db400640 1 -- 172.21.15.22:0/1829313745 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e4004180 con 0x55f923250280 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1db400640 1 -- 172.21.15.22:0/1829313745 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4011660 con 0x55f923250280 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.900+0000 7fb1e2a00640 1 -- 172.21.15.22:0/4165797356 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb1d40f0c50 con 0x7fb1ec100520 2024-03-24T06:49:59.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 0x7fb1d00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1e9e00640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 0x7fb1d00a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55f92324fd50 tx=0x7fb1d40dd040 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1829313745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 msgr2=0x55f923250660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400dc00 tx=0x7fb1e400f730 comp rx=0 tx=0).stop 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1829313745 shutdown_connections 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f92325bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923259240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1829313745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923250660 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.901+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1829313745 >> 172.21.15.22:0/1829313745 conn(0x55f92324a9b0 msgr2=0x55f92324ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.902+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1829313745 shutdown_connections 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.902+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1829313745 wait complete. 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.902+0000 7fb1f3db7740 1 -- start start 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 Processor -- start 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f923256c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f9232571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f92315c060 con 0x55f923250ba0 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f92315bd00 con 0x55f923259780 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.903+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f922f251c0 con 0x55f923250280 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1f3db7740 10 client.0 _ll_register_callbacks cb 0x55f922ec7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50892/0 (socket says 172.21.15.22:50892) 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 -- 172.21.15.22:0/2475825137 learned_addr learned my addr 172.21.15.22:0/2475825137 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f9232571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1e9400640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f923256c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 -- 172.21.15.22:0/2475825137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 msgr2=0x55f9232571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f9232571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 -- 172.21.15.22:0/2475825137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 msgr2=0x55f923256c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f923256c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.904+0000 7fb1ea800640 1 -- 172.21.15.22:0/2475825137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e400d830 con 0x55f923250280 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1ea800640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005780 tx=0x7fb1e4003d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56a0451a8db1e89d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40037f0 con 0x55f923250280 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb1e4003950 con 0x55f923250280 2024-03-24T06:49:59.912 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f922eb67e0 con 0x55f923250280 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb1e4011440 con 0x55f923250280 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1f3db7740 10 client.27775 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f9231b5a70 con 0x55f923250280 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.905+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e4033020 con 0x55f923250280 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb1e402b430 con 0x55f923250280 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 10 client.27775 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 10 client.27775 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 20 client.27775 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.913 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f923269b00 0x55f92326bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 4 client.27775 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f922ef3720 con 0x55f923269b00 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.906+0000 7fb1e9400640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f923269b00 0x55f92326bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.907+0000 7fb1e9400640 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f923269b00 0x55f92326bf20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47ecf2f48ad2a88d server_cookie=89667278a43318d0 in_seq=0 out_seq=0 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.907+0000 7fb1cfe00640 10 client.27775 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1cfe00640 1 -- 172.21.15.22:0/2475825137 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f922ef3720 con 0x55f923269b00 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1cfe00640 10 client.27775 handle_command_reply: tid=0 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 client.27775 shutdown 2024-03-24T06:49:59.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 msgr2=0x55f923256750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005780 tx=0x7fb1e4003d10 comp rx=0 tx=0).stop 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 shutdown_connections 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f923269b00 0x55f92326bf20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f9232571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f923256c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1cfe00640 0 client.27775 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/2475825137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923256750 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 >> 172.21.15.22:0/2475825137 conn(0x55f92324a9b0 msgr2=0x55f92324c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 shutdown_connections 2024-03-24T06:49:59.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.908+0000 7fb1f3db7740 1 -- 172.21.15.22:0/2475825137 wait complete. 2024-03-24T06:49:59.916 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.909+0000 7fb1f3db7740 20 client.27775 trim_cache size 0 max 16384 2024-03-24T06:49:59.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.938+0000 7fb1f3db7740 1 Processor -- start 2024-03-24T06:49:59.938 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.938+0000 7fb1f3db7740 1 -- start start 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923256ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f9232717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f92326c580 con 0x55f923259780 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f92324fee0 con 0x55f923250ba0 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f92324b740 con 0x55f923250280 2024-03-24T06:49:59.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50894/0 (socket says 172.21.15.22:50894) 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1ea800640 1 -- 172.21.15.22:0/1782919606 learned_addr learned my addr 172.21.15.22:0/1782919606 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f9232717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.939+0000 7fb1e9400640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923256ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 -- 172.21.15.22:0/1782919606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 msgr2=0x55f923256ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923256ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 -- 172.21.15.22:0/1782919606 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 msgr2=0x55f9232717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f9232717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 -- 172.21.15.22:0/1782919606 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f923151b00 con 0x55f923250280 2024-03-24T06:49:59.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1e9400640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923256ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:49:59.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1ea800640 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400d7d0 tx=0x7fb1e400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca41250eccce81a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1daa00640 1 -- 172.21.15.22:0/1782919606 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e400fc30 con 0x55f923250280 2024-03-24T06:49:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1daa00640 1 -- 172.21.15.22:0/1782919606 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e4033070 con 0x55f923250280 2024-03-24T06:49:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.940+0000 7fb1daa00640 1 -- 172.21.15.22:0/1782919606 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40054d0 con 0x55f923250280 2024-03-24T06:49:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1782919606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 msgr2=0x55f923265010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400d7d0 tx=0x7fb1e400f730 comp rx=0 tx=0).stop 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1782919606 shutdown_connections 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923259780 0x55f9232717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923250ba0 0x55f923256ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/1782919606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923265010 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1782919606 >> 172.21.15.22:0/1782919606 conn(0x55f92324a9b0 msgr2=0x55f92324bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1782919606 shutdown_connections 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- 172.21.15.22:0/1782919606 wait complete. 2024-03-24T06:49:59.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.941+0000 7fb1f3db7740 1 -- start start 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 Processor -- start 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f9232619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f92325ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55f92325d110 con 0x55f923250ba0 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f92325d280 con 0x55f923259780 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f92325d440 con 0x55f923250280 2024-03-24T06:49:59.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1f3db7740 10 client.0 _ll_register_callbacks cb 0x7fb1e4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1ea800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50900/0 (socket says 172.21.15.22:50900) 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1ea800640 1 -- 172.21.15.22:0/107877655 learned_addr learned my addr 172.21.15.22:0/107877655 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1e9400640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f9232619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.942+0000 7fb1ea800640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f92325ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 -- 172.21.15.22:0/107877655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 msgr2=0x55f92325ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f92325ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 -- 172.21.15.22:0/107877655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 msgr2=0x55f9232619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:49:59.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f9232619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 -- 172.21.15.22:0/107877655 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9231b5a70 con 0x55f923250280 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1e9400640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f9232619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1ea800640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4004150 tx=0x7fb1e40050d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=636228ff755c468b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.943+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e402b590 con 0x55f923250280 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e4004180 con 0x55f923250280 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fb1e4000e40 con 0x55f923250280 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb1e4005380 con 0x55f923250280 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 10 client.27781 fetch_fsmap learned FSMap version 37 2024-03-24T06:49:59.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f9231776a0 con 0x55f923250280 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e4004180 con 0x55f923250280 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fb1e4036070 con 0x55f923250280 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 10 client.27781 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 10 client.27781 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 20 client.27781 populate_metadata read hostname 'smithi022' 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f9232d8470 0x55f9232da890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 4 client.27781 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.944+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f922f42890 con 0x55f9232d8470 2024-03-24T06:49:59.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1e9400640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f9232d8470 0x55f9232da890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1e9400640 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f9232d8470 0x55f9232da890 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2f8a68c4119f48b8 server_cookie=4b385af994d94c60 in_seq=0 out_seq=0 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1cfe00640 10 client.27781 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1cfe00640 1 -- 172.21.15.22:0/107877655 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f922f42890 con 0x55f9232d8470 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1cfe00640 10 client.27781 handle_command_reply: tid=0 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.945+0000 7fb1f3db7740 1 client.27781 shutdown 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 msgr2=0x55f923261470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4004150 tx=0x7fb1e40050d0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 shutdown_connections 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x55f9232d8470 0x55f9232da890 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.952 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f923259780 0x55f92325ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55f923250ba0 0x55f9232619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1cfe00640 0 client.27781 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 --2- 172.21.15.22:0/107877655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f923250280 0x55f923261470 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 >> 172.21.15.22:0/107877655 conn(0x55f92324a9b0 msgr2=0x55f923249a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 shutdown_connections 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 1 -- 172.21.15.22:0/107877655 wait complete. 2024-03-24T06:49:59.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.946+0000 7fb1f3db7740 20 client.27781 trim_cache size 0 max 16384 2024-03-24T06:49:59.954 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:49:59.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 msgr2=0x7fb1d00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 0x7fb1d00a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x55f92324fd50 tx=0x7fb1d40dd040 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 msgr2=0x7fb1ec199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000c00 tx=0x7fb1d4004040 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 shutdown_connections 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1d00a2230 0x7fb1d00a46f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1ec10e0d0 0x7fb1ec19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1ec100e40 0x7fb1ec19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 --2- 172.21.15.22:0/4165797356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1ec100520 0x7fb1ec199ff0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 >> 172.21.15.22:0/4165797356 conn(0x7fb1ec077770 msgr2=0x7fb1ec10f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:49:59.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 shutdown_connections 2024-03-24T06:49:59.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:49:59.947+0000 7fb1f1000640 1 -- 172.21.15.22:0/4165797356 wait complete. 2024-03-24T06:49:59.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:50:00.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 -- 172.21.15.22:0/514455206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30107650 msgr2=0x7f7e30107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 --2- 172.21.15.22:0/514455206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30107650 0x7f7e30107a30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009930 tx=0x7f7e2002bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:00.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 -- 172.21.15.22:0/514455206 shutdown_connections 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 --2- 172.21.15.22:0/514455206 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30101f50 0x7f7e3010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 --2- 172.21.15.22:0/514455206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101590 0x7f7e30101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 --2- 172.21.15.22:0/514455206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30107650 0x7f7e30107a30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 -- 172.21.15.22:0/514455206 >> 172.21.15.22:0/514455206 conn(0x7f7e300fd080 msgr2=0x7f7e300ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 -- 172.21.15.22:0/514455206 shutdown_connections 2024-03-24T06:50:00.169 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.168+0000 7f7e36000640 1 -- 172.21.15.22:0/514455206 wait complete. 2024-03-24T06:50:00.171 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.169+0000 7f7e36000640 1 Processor -- start 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.169+0000 7f7e36000640 1 -- start start 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 0x7f7e3019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30107650 0x7f7e3019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3019ec40 con 0x7f7e30107650 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3019edb0 con 0x7f7e30101f50 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e36000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3019ef70 con 0x7f7e30101590 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50904/0 (socket says 172.21.15.22:50904) 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 -- 172.21.15.22:0/897060241 learned_addr learned my addr 172.21.15.22:0/897060241 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e35600640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30107650 0x7f7e3019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e27e00640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 0x7f7e3019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 -- 172.21.15.22:0/897060241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 msgr2=0x7f7e3019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 0x7f7e3019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 -- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30107650 msgr2=0x7f7e3019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.170+0000 7f7e34c00640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30107650 0x7f7e3019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e34c00640 1 -- 172.21.15.22:0/897060241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c009490 con 0x7f7e30101590 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e27e00640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 0x7f7e3019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e34c00640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e2002c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e2e000640 1 -- 172.21.15.22:0/897060241 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200042d0 con 0x7f7e30101590 2024-03-24T06:50:00.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e2e000640 1 -- 172.21.15.22:0/897060241 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e20004430 con 0x7f7e30101590 2024-03-24T06:50:00.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e28009840 con 0x7f7e30101590 2024-03-24T06:50:00.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e2e000640 1 -- 172.21.15.22:0/897060241 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20003b50 con 0x7f7e30101590 2024-03-24T06:50:00.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.171+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e20009560 con 0x7f7e30101590 2024-03-24T06:50:00.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.175+0000 7f7e2e000640 1 -- 172.21.15.22:0/897060241 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7e20005460 con 0x7f7e30101590 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.176+0000 7f7e3a3c4740 1 Processor -- start 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.176+0000 7f7e2e000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 0x7f7e140a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.176+0000 7f7e3a3c4740 1 -- start start 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e2e000640 1 -- 172.21.15.22:0/897060241 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7e200f09f0 con 0x7f7e30101590 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e35600640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 0x7f7e140a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24921f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e24921f950 0x55e249221d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e2491846e0 con 0x55e24921f950 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e249181c10 con 0x55e249216d70 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e24917d3f0 con 0x55e249216450 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50910/0 (socket says 172.21.15.22:50910) 2024-03-24T06:50:00.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 -- 172.21.15.22:0/107576861 learned_addr learned my addr 172.21.15.22:0/107576861 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e27e00640 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24921f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e35600640 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e24921f950 0x55e249221d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 -- 172.21.15.22:0/107576861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 msgr2=0x55e24921f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24921f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.177+0000 7f7e34c00640 1 -- 172.21.15.22:0/107576861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e24921f950 msgr2=0x55e249221d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e34c00640 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e24921f950 0x55e249221d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e34c00640 1 -- 172.21.15.22:0/107576861 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e24913ee40 con 0x55e249216450 2024-03-24T06:50:00.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e35600640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 0x7f7e140a4770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e28005fd0 tx=0x7f7e28011d70 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e34c00640 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7e200b9ab0 tx=0x7f7e200035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f851a0f00cf72639 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e25c00640 1 -- 172.21.15.22:0/107576861 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200d2070 con 0x55e249216450 2024-03-24T06:50:00.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e25c00640 1 -- 172.21.15.22:0/107576861 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e200bd0e0 con 0x55e249216450 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.178+0000 7f7e25c00640 1 -- 172.21.15.22:0/107576861 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20003cb0 con 0x55e249216450 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/107576861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 msgr2=0x55e249216830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7e200b9ab0 tx=0x7f7e200035f0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/107576861 shutdown_connections 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e24921f950 0x55e249221d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24921f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/107576861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e249216830 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/107576861 >> 172.21.15.22:0/107576861 conn(0x55e249210b80 msgr2=0x55e249210f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/107576861 shutdown_connections 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.179+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/107576861 wait complete. 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 -- start start 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 Processor -- start 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e24921d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e249122060 con 0x55e249216450 2024-03-24T06:50:00.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e249121d00 con 0x55e24921f950 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.180+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e248eeb1c0 con 0x55e249216d70 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e3a3c4740 10 client.0 _ll_register_callbacks cb 0x55e248e8dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50914/0 (socket says 172.21.15.22:50914) 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 -- 172.21.15.22:0/1967497476 learned_addr learned my addr 172.21.15.22:0/1967497476 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e35600640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e24921d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 -- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 msgr2=0x55e24921d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e24921d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 -- 172.21.15.22:0/1967497476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 msgr2=0x55e24921ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 -- 172.21.15.22:0/1967497476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e20009560 con 0x55e249216d70 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e34c00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.181+0000 7f7e27e00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c012d20 tx=0x7f7e1c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac6a40343593e913 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e1a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c017070 con 0x55e249216d70 2024-03-24T06:50:00.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7e1c004030 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e248e7c7e0 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7e1c0041f0 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e3a3c4740 10 client.27793 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e24917ba70 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.182+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e248e7c7e0 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7e1c010430 con 0x55e249216d70 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 10 client.27793 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:00.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 10 client.27793 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 20 client.27793 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e24922fae0 0x55e249231f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 4 client.27793 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e248eb9720 con 0x55e24922fae0 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.183+0000 7f7e34c00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e24922fae0 0x55e249231f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e34c00640 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e24922fae0 0x55e249231f20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9af15466c7f58540 server_cookie=e7c6d4303e11a5c1 in_seq=0 out_seq=0 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e1a000640 10 client.27793 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e1a000640 1 -- 172.21.15.22:0/1967497476 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e248eb9720 con 0x55e24922fae0 2024-03-24T06:50:00.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e1a000640 10 client.27793 handle_command_reply: tid=0 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e3a3c4740 1 client.27793 shutdown 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 msgr2=0x55e24921cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.184+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c012d20 tx=0x7f7e1c00f790 comp rx=0 tx=0).stop 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 shutdown_connections 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e24922fae0 0x55e249231f20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e24921d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216d70 0x55e24921cf70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/1967497476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e1a000640 0 client.27793 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:00.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 >> 172.21.15.22:0/1967497476 conn(0x55e249210b80 msgr2=0x55e2492128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 shutdown_connections 2024-03-24T06:50:00.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/1967497476 wait complete. 2024-03-24T06:50:00.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.185+0000 7f7e3a3c4740 20 client.27793 trim_cache size 0 max 16384 2024-03-24T06:50:00.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 Processor -- start 2024-03-24T06:50:00.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 -- start start 2024-03-24T06:50:00.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24922afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 0x55e249237830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e249218680 con 0x55e249216450 2024-03-24T06:50:00.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e24920c270 con 0x55e249216d70 2024-03-24T06:50:00.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e249211b80 con 0x55e24921f950 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34484/0 (socket says 172.21.15.22:34484) 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e35600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 0x55e249237830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e27e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24922afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.200+0000 7f7e34c00640 1 -- 172.21.15.22:0/4255576873 learned_addr learned my addr 172.21.15.22:0/4255576873 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 -- 172.21.15.22:0/4255576873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 msgr2=0x55e24922afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24922afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 -- 172.21.15.22:0/4255576873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 msgr2=0x55e24921f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 -- 172.21.15.22:0/4255576873 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e249117b00 con 0x55e24921f950 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e34c00640 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e27e00640 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24922afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e35600640 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 0x55e249237830 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55e249211d10 tx=0x7f7e28009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c79f32daab0ddfe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e25200640 1 -- 172.21.15.22:0/4255576873 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800fc80 con 0x55e24921f950 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.201+0000 7f7e25200640 1 -- 172.21.15.22:0/4255576873 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e28025070 con 0x55e24921f950 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e25200640 1 -- 172.21.15.22:0/4255576873 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2802b660 con 0x55e24921f950 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4255576873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 msgr2=0x55e249237830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 0x55e249237830 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x55e249211d10 tx=0x7f7e28009500 comp rx=0 tx=0).stop 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4255576873 shutdown_connections 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e24921f950 0x55e249237830 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e249216d70 0x55e24922afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4255576873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216450 0x55e24921f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4255576873 >> 172.21.15.22:0/4255576873 conn(0x55e249210b80 msgr2=0x55e249217290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4255576873 shutdown_connections 2024-03-24T06:50:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.202+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4255576873 wait complete. 2024-03-24T06:50:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 -- start start 2024-03-24T06:50:00.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 Processor -- start 2024-03-24T06:50:00.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216d70 0x55e249219380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e249219920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55e24921a020 con 0x55e249216d70 2024-03-24T06:50:00.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e24921a190 con 0x55e24921f950 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e249294e60 con 0x55e249216450 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.203+0000 7f7e3a3c4740 10 client.0 _ll_register_callbacks cb 0x7f7e1c009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e35600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e249219920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50922/0 (socket says 172.21.15.22:50922) 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 -- 172.21.15.22:0/4112038563 learned_addr learned my addr 172.21.15.22:0/4112038563 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e27e00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216d70 0x55e249219380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 -- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 msgr2=0x55e249219920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e249219920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 -- 172.21.15.22:0/4112038563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216d70 msgr2=0x55e249219380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216d70 0x55e249219380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 -- 172.21.15.22:0/4112038563 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e24913ee40 con 0x55e249216450 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e35600640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e249219920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e34c00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7e200b9540 tx=0x7f7e20004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=287c17bd34054075 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e1a000640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.204+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200d2040 con 0x55e249216450 2024-03-24T06:50:00.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e200bd0e0 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f7e200c1230 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7e200c5900 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 10 client.27799 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e24917ba70 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e200bd0e0 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f7e2003c070 con 0x55e249216450 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 10 client.27799 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 10 client.27799 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:50:00.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 20 client.27799 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e249296300 0x55e249298720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 4 client.27799 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e248f08890 con 0x55e249296300 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.205+0000 7f7e27e00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e249296300 0x55e249298720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e27e00640 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e249296300 0x55e249298720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1be00b2cb47ba041 server_cookie=7f7e09bed96d593b in_seq=0 out_seq=0 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e1a000640 10 client.27799 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e1a000640 1 -- 172.21.15.22:0/4112038563 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e248f08890 con 0x55e249296300 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e1a000640 10 client.27799 handle_command_reply: tid=0 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 client.27799 shutdown 2024-03-24T06:50:00.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 msgr2=0x55e2492278c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7e200b9540 tx=0x7f7e20004040 comp rx=0 tx=0).stop 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 shutdown_connections 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x55e249296300 0x55e249298720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e24921f950 0x55e249219920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55e249216d70 0x55e249219380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e1a000640 0 client.27799 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 --2- 172.21.15.22:0/4112038563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e249216450 0x55e2492278c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.206+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 >> 172.21.15.22:0/4112038563 conn(0x55e249210b80 msgr2=0x55e24921c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 shutdown_connections 2024-03-24T06:50:00.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e3a3c4740 1 -- 172.21.15.22:0/4112038563 wait complete. 2024-03-24T06:50:00.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e3a3c4740 20 client.27799 trim_cache size 0 max 16384 2024-03-24T06:50:00.215 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 msgr2=0x7f7e140a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 0x7f7e140a4770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e28005fd0 tx=0x7f7e28011d70 comp rx=0 tx=0).stop 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 msgr2=0x7f7e30199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e2002c5f0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 shutdown_connections 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7e140a22b0 0x7f7e140a4770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e30107650 0x7f7e3019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e30101f50 0x7f7e3019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 --2- 172.21.15.22:0/897060241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e30101590 0x7f7e30199d80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.207+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 >> 172.21.15.22:0/897060241 conn(0x7f7e300fd080 msgr2=0x7f7e3010cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.208+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 shutdown_connections 2024-03-24T06:50:00.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.208+0000 7f7e36000640 1 -- 172.21.15.22:0/897060241 wait complete. 2024-03-24T06:50:00.226 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 -- 172.21.15.22:0/3379634571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5810ddb0 msgr2=0x7f3a58110200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 --2- 172.21.15.22:0/3379634571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5810ddb0 0x7f3a58110200 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009ca0 tx=0x7f3a5002c180 comp rx=0 tx=0).stop 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 -- 172.21.15.22:0/3379634571 shutdown_connections 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 --2- 172.21.15.22:0/3379634571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5810ddb0 0x7f3a58110200 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 --2- 172.21.15.22:0/3379634571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58074460 0x7f3a5810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 --2- 172.21.15.22:0/3379634571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a58073b40 0x7f3a58073f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.446+0000 7f3a5d000640 1 -- 172.21.15.22:0/3379634571 >> 172.21.15.22:0/3379634571 conn(0x7f3a580fd080 msgr2=0x7f3a580ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.447+0000 7f3a5d000640 1 -- 172.21.15.22:0/3379634571 shutdown_connections 2024-03-24T06:50:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.447+0000 7f3a5d000640 1 -- 172.21.15.22:0/3379634571 wait complete. 2024-03-24T06:50:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.448+0000 7f3a5d000640 1 Processor -- start 2024-03-24T06:50:00.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.448+0000 7f3a5d000640 1 -- start start 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58073b40 0x7f3a58199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a5810ddb0 0x7f3a5819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5819ecc0 con 0x7f3a5810ddb0 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5819ee30 con 0x7f3a58074460 2024-03-24T06:50:00.450 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a5d000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5819eff0 con 0x7f3a58073b40 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49632/0 (socket says 172.21.15.22:49632) 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 -- 172.21.15.22:0/4031010040 learned_addr learned my addr 172.21.15.22:0/4031010040 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a57400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58073b40 0x7f3a58199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a57e00640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a5810ddb0 0x7f3a5819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 -- 172.21.15.22:0/4031010040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58073b40 msgr2=0x7f3a58199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58073b40 0x7f3a58199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.451 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 -- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a5810ddb0 msgr2=0x7f3a5819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.449+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a5810ddb0 0x7f3a5819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a4ea00640 1 -- 172.21.15.22:0/4031010040 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a50009900 con 0x7f3a58074460 2024-03-24T06:50:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9f0 tx=0x7f3a4000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.452 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a54c00640 1 -- 172.21.15.22:0/4031010040 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40009720 con 0x7f3a58074460 2024-03-24T06:50:00.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a54c00640 1 -- 172.21.15.22:0/4031010040 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4000cd80 con 0x7f3a58074460 2024-03-24T06:50:00.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a54c00640 1 -- 172.21.15.22:0/4031010040 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40010690 con 0x7f3a58074460 2024-03-24T06:50:00.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5819f2d0 con 0x7f3a58074460 2024-03-24T06:50:00.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.450+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a581a6b50 con 0x7f3a58074460 2024-03-24T06:50:00.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 Processor -- start 2024-03-24T06:50:00.455 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 -- start start 2024-03-24T06:50:00.456 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a218410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a21ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562e8a17d6e0 con 0x562e8a20fd70 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562e8a17ac10 con 0x562e8a218950 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a613b4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562e8a1763f0 con 0x562e8a20f450 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a54c00640 1 -- 172.21.15.22:0/4031010040 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f3a400046c0 con 0x7f3a58074460 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.455+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50950/0 (socket says 172.21.15.22:50950) 2024-03-24T06:50:00.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 -- 172.21.15.22:0/1974840507 learned_addr learned my addr 172.21.15.22:0/1974840507 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57400640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a218410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a21ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 -- 172.21.15.22:0/1974840507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 msgr2=0x562e8a21ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a21ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 -- 172.21.15.22:0/1974840507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 msgr2=0x562e8a218410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a218410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57e00640 1 -- 172.21.15.22:0/1974840507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e8a137e40 con 0x562e8a20f450 2024-03-24T06:50:00.458 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.456+0000 7f3a57400640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a218410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a57e00640 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3a5002c790 tx=0x7f3a50005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36050b513240b4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a4cc00640 1 -- 172.21.15.22:0/1974840507 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50034440 con 0x562e8a20f450 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a4cc00640 1 -- 172.21.15.22:0/1974840507 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a50003970 con 0x562e8a20f450 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a4cc00640 1 -- 172.21.15.22:0/1974840507 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50032e30 con 0x562e8a20f450 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a613b4740 1 -- 172.21.15.22:0/1974840507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 msgr2=0x562e8a20f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3a5002c790 tx=0x7f3a50005f80 comp rx=0 tx=0).stop 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.457+0000 7f3a54c00640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 0x7f3a3c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 -- 172.21.15.22:0/1974840507 shutdown_connections 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a21ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a218410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1974840507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a20f830 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 -- 172.21.15.22:0/1974840507 >> 172.21.15.22:0/1974840507 conn(0x562e8a209b80 msgr2=0x562e8a209f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a54c00640 1 -- 172.21.15.22:0/4031010040 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3a40014070 con 0x7f3a58074460 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a57400640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 0x7f3a3c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 -- 172.21.15.22:0/1974840507 shutdown_connections 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 -- 172.21.15.22:0/1974840507 wait complete. 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.458+0000 7f3a613b4740 1 -- start start 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a57400640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 0x7f3a3c0a4770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x562e8a20ef20 tx=0x7f3a48005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 Processor -- start 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20f450 0x562e8a215960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a218950 0x562e8a2163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562e8a11b060 con 0x562e8a20f450 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562e8a11ad00 con 0x562e8a20fd70 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562e89ee41c0 con 0x562e8a218950 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.459+0000 7f3a613b4740 10 client.0 _ll_register_callbacks cb 0x562e89e86be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a57400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20f450 0x562e8a215960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a218950 0x562e8a2163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49650/0 (socket says 172.21.15.22:49650) 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 -- 172.21.15.22:0/1712642188 learned_addr learned my addr 172.21.15.22:0/1712642188 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 -- 172.21.15.22:0/1712642188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a218950 msgr2=0x562e8a2163e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a218950 0x562e8a2163e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 -- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20f450 msgr2=0x562e8a215960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20f450 0x562e8a215960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 -- 172.21.15.22:0/1712642188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a58110ed0 con 0x562e8a20fd70 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3a400cbab0 tx=0x7f3a40009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36bf54f2bbc7a2b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a45600640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:00.466 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a400ce4a0 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3a4009b3f0 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.460+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e89e757e0 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3a4009a830 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 10 client.27785 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562e8a174a70 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562e89e757e0 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3a4001f060 con 0x562e8a20fd70 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 10 client.27785 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:00.467 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 10 client.27785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 20 client.27785 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a228d30 0x562e8a22b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 4 client.27785 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.461+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e89eb2720 con 0x562e8a228d30 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.462+0000 7f3a57400640 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a228d30 0x562e8a22b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.462+0000 7f3a57400640 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a228d30 0x562e8a22b170 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caf811da9cdb5958 server_cookie=b01afed63837edb0 in_seq=0 out_seq=0 2024-03-24T06:50:00.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.462+0000 7f3a45600640 10 client.27785 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:00.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.463+0000 7f3a45600640 1 -- 172.21.15.22:0/1712642188 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e89eb2720 con 0x562e8a228d30 2024-03-24T06:50:00.469 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.463+0000 7f3a45600640 10 client.27785 handle_command_reply: tid=0 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.463+0000 7f3a613b4740 1 client.27785 shutdown 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.463+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 msgr2=0x562e8a215ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.463+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3a400cbab0 tx=0x7f3a40009310 comp rx=0 tx=0).stop 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 shutdown_connections 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a218950 0x562e8a2163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a228d30 0x562e8a22b170 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a215ea0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 --2- 172.21.15.22:0/1712642188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20f450 0x562e8a215960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a45600640 0 client.27785 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:00.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 >> 172.21.15.22:0/1712642188 conn(0x562e8a209b80 msgr2=0x562e8a20b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 shutdown_connections 2024-03-24T06:50:00.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 1 -- 172.21.15.22:0/1712642188 wait complete. 2024-03-24T06:50:00.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.464+0000 7f3a613b4740 20 client.27785 trim_cache size 0 max 16384 2024-03-24T06:50:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 Processor -- start 2024-03-24T06:50:00.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 -- start start 2024-03-24T06:50:00.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a2241d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a230a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562e8a1df190 con 0x562e8a20fd70 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562e8a209440 con 0x562e8a218950 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a613b4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562e8a2107f0 con 0x562e8a20f450 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a57400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a57400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:50974/0 (socket says 172.21.15.22:50974) 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a57400640 1 -- 172.21.15.22:0/3104568759 learned_addr learned my addr 172.21.15.22:0/3104568759 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a57e00640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a230a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.479+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a2241d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 -- 172.21.15.22:0/3104568759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 msgr2=0x562e8a230a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a230a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 -- 172.21.15.22:0/3104568759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 msgr2=0x562e8a2241d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a2241d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 -- 172.21.15.22:0/3104568759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e8a110b00 con 0x562e8a20f450 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a2241d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a57400640 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x562e8a22b9c0 tx=0x7f3a4801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab46c0d0b744cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a47e00640 1 -- 172.21.15.22:0/3104568759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48023070 con 0x562e8a20f450 2024-03-24T06:50:00.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a47e00640 1 -- 172.21.15.22:0/3104568759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a48024040 con 0x562e8a20f450 2024-03-24T06:50:00.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.480+0000 7f3a47e00640 1 -- 172.21.15.22:0/3104568759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48017070 con 0x562e8a20f450 2024-03-24T06:50:00.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 -- 172.21.15.22:0/3104568759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 msgr2=0x562e8a218090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x562e8a22b9c0 tx=0x7f3a4801b000 comp rx=0 tx=0).stop 2024-03-24T06:50:00.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 -- 172.21.15.22:0/3104568759 shutdown_connections 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a218950 0x562e8a230a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a20fd70 0x562e8a2241d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 --2- 172.21.15.22:0/3104568759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a218090 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.481+0000 7f3a613b4740 1 -- 172.21.15.22:0/3104568759 >> 172.21.15.22:0/3104568759 conn(0x562e8a209b80 msgr2=0x562e8a215920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.482+0000 7f3a613b4740 1 -- 172.21.15.22:0/3104568759 shutdown_connections 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.482+0000 7f3a613b4740 1 -- 172.21.15.22:0/3104568759 wait complete. 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.482+0000 7f3a613b4740 1 -- start start 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.482+0000 7f3a613b4740 1 Processor -- start 2024-03-24T06:50:00.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a220800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a218950 0x562e8a21dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x562e8a220f20 con 0x562e8a218950 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562e8a21e250 con 0x562e8a20fd70 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562e8a21e410 con 0x562e8a20f450 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a613b4740 10 client.0 _ll_register_callbacks cb 0x7f3a4000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.489 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a57400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a220800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a4ea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49670/0 (socket says 172.21.15.22:49670) 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.483+0000 7f3a4ea00640 1 -- 172.21.15.22:0/234442767 learned_addr learned my addr 172.21.15.22:0/234442767 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57e00640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a218950 0x562e8a21dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 -- 172.21.15.22:0/234442767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 msgr2=0x562e8a21d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 -- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a218950 msgr2=0x562e8a21dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a218950 0x562e8a21dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 -- 172.21.15.22:0/234442767 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a48009de0 con 0x562e8a20f450 2024-03-24T06:50:00.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a57400640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a220800 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3a48019040 tx=0x7f3a48003e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5985852733e86cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a45600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.484+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48017070 con 0x562e8a20f450 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f3a48005040 con 0x562e8a20f450 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a50003970 con 0x562e8a20f450 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3a48023040 con 0x562e8a20f450 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 10 client.27817 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562e8a1366a0 con 0x562e8a20f450 2024-03-24T06:50:00.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a50003970 con 0x562e8a20f450 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f3a480046c0 con 0x562e8a20f450 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 10 client.27817 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 10 client.27817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 20 client.27817 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a290410 0x562e8a292830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 4 client.27817 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.485+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e89f01890 con 0x562e8a290410 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a290410 0x562e8a292830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:00.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a4ea00640 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a290410 0x562e8a292830 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d86efbfb5b0fdda server_cookie=dd03724ea680eb08 in_seq=0 out_seq=0 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a45600640 10 client.27817 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a45600640 1 -- 172.21.15.22:0/234442767 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e89f01890 con 0x562e8a290410 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a45600640 10 client.27817 handle_command_reply: tid=0 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.486+0000 7f3a613b4740 1 client.27817 shutdown 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 msgr2=0x562e8a220800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a220800 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3a48019040 tx=0x7f3a48003e50 comp rx=0 tx=0).stop 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 shutdown_connections 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x562e8a218950 0x562e8a21dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x562e8a290410 0x562e8a292830 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562e8a20fd70 0x562e8a21d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 --2- 172.21.15.22:0/234442767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562e8a20f450 0x562e8a220800 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 >> 172.21.15.22:0/234442767 conn(0x562e8a209b80 msgr2=0x562e8a208cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a45600640 0 client.27817 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 shutdown_connections 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 1 -- 172.21.15.22:0/234442767 wait complete. 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.487+0000 7f3a613b4740 20 client.27817 trim_cache size 0 max 16384 2024-03-24T06:50:00.494 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 msgr2=0x7f3a3c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 0x7f3a3c0a4770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x562e8a20ef20 tx=0x7f3a48005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 msgr2=0x7f3a5819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:00.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9f0 tx=0x7f3a4000edf0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 shutdown_connections 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f3a3c0a22b0 0x7f3a3c0a4770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3a5810ddb0 0x7f3a5819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a58074460 0x7f3a5819a390 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 --2- 172.21.15.22:0/4031010040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a58073b40 0x7f3a58199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 >> 172.21.15.22:0/4031010040 conn(0x7f3a580fd080 msgr2=0x7f3a580fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 shutdown_connections 2024-03-24T06:50:00.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:00.488+0000 7f3a5d000640 1 -- 172.21.15.22:0/4031010040 wait complete. 2024-03-24T06:50:00.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:00 smithi022 ceph-mon[32003]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:00.504 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:00 smithi022 ceph-mon[32003]: overall HEALTH_OK 2024-03-24T06:50:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:00 smithi143 ceph-mon[36520]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:00.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:00 smithi143 ceph-mon[36520]: overall HEALTH_OK 2024-03-24T06:50:00.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:00 smithi155 ceph-mon[36262]: pgmap v310: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:00.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:00 smithi155 ceph-mon[36262]: overall HEALTH_OK 2024-03-24T06:50:01.509 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T06:50:01.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:50:01.509 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 -- 172.21.15.22:0/1844992947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118106550 msgr2=0x7f2118106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 --2- 172.21.15.22:0/1844992947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118106550 0x7f2118106930 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2108009930 tx=0x7f210802bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 -- 172.21.15.22:0/1844992947 shutdown_connections 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 --2- 172.21.15.22:0/1844992947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211810d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 --2- 172.21.15.22:0/1844992947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118100490 0x7f2118100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 --2- 172.21.15.22:0/1844992947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118106550 0x7f2118106930 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 -- 172.21.15.22:0/1844992947 >> 172.21.15.22:0/1844992947 conn(0x7f21180fbf60 msgr2=0x7f21180fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 -- 172.21.15.22:0/1844992947 shutdown_connections 2024-03-24T06:50:01.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.750+0000 7f211cc00640 1 -- 172.21.15.22:0/1844992947 wait complete. 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.751+0000 7f211cc00640 1 Processor -- start 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 -- start start 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 0x7f211819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f211819ec90 con 0x7f2118106550 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f211819ee00 con 0x7f2118100490 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f211cc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f211819efc0 con 0x7f2118100e50 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f2117400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f2117400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49686/0 (socket says 172.21.15.22:49686) 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f2117400640 1 -- 172.21.15.22:0/1405073251 learned_addr learned my addr 172.21.15.22:0/1405073251 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.752+0000 7f2116a00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117e00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 0x7f211819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 -- 172.21.15.22:0/1405073251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 msgr2=0x7f211819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 -- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 msgr2=0x7f211819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 0x7f211819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 -- 172.21.15.22:0/1405073251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2108009560 con 0x7f2118100490 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2116a00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117e00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 0x7f211819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.753+0000 7f2117400640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2108000c00 tx=0x7f210802c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.754+0000 7f210fe00640 1 -- 172.21.15.22:0/1405073251 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21080042d0 con 0x7f2118100490 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.754+0000 7f210fe00640 1 -- 172.21.15.22:0/1405073251 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2108004430 con 0x7f2118100490 2024-03-24T06:50:01.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.754+0000 7f210fe00640 1 -- 172.21.15.22:0/1405073251 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108003c20 con 0x7f2118100490 2024-03-24T06:50:01.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.754+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f211819f240 con 0x7f2118100490 2024-03-24T06:50:01.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.754+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21181a6b20 con 0x7f2118100490 2024-03-24T06:50:01.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.757+0000 7f210fe00640 1 -- 172.21.15.22:0/1405073251 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f2108004df0 con 0x7f2118100490 2024-03-24T06:50:01.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.757+0000 7f2120fdf740 1 Processor -- start 2024-03-24T06:50:01.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.757+0000 7f2120fdf740 1 -- start start 2024-03-24T06:50:01.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dc8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558405d362d0 con 0x558405dc8950 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558405d33800 con 0x558405dc8030 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2120fdf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558405d2efe0 con 0x558405dd1530 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2117400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2117e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dc8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51028/0 (socket says 172.21.15.22:51028) 2024-03-24T06:50:01.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 -- 172.21.15.22:0/2784944034 learned_addr learned my addr 172.21.15.22:0/2784944034 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 -- 172.21.15.22:0/2784944034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 msgr2=0x558405dc8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dc8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 -- 172.21.15.22:0/2784944034 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 msgr2=0x558405dd0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.758+0000 7f2116a00640 1 -- 172.21.15.22:0/2784944034 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558405cf0e90 con 0x558405dd1530 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f2116a00640 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2100009cf0 tx=0x7f21000117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60523d5ded5f7d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f210fe00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 0x7f20fc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f2107e00640 1 -- 172.21.15.22:0/2784944034 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100012450 con 0x558405dd1530 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f2107e00640 1 -- 172.21.15.22:0/2784944034 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2100012a90 con 0x558405dd1530 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f2107e00640 1 -- 172.21.15.22:0/2784944034 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210000fb80 con 0x558405dd1530 2024-03-24T06:50:01.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f210fe00640 1 -- 172.21.15.22:0/1405073251 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f21080f0a70 con 0x7f2118100490 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.759+0000 7f2117e00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 0x7f20fc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 -- 172.21.15.22:0/2784944034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 msgr2=0x558405dd3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2100009cf0 tx=0x7f21000117b0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 -- 172.21.15.22:0/2784944034 shutdown_connections 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd3920 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2784944034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dc8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 -- 172.21.15.22:0/2784944034 >> 172.21.15.22:0/2784944034 conn(0x558405dc2760 msgr2=0x558405dc2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2117e00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 0x7f20fc0a4770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2110005fd0 tx=0x7f2110009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 -- 172.21.15.22:0/2784944034 shutdown_connections 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.760+0000 7f2120fdf740 1 -- 172.21.15.22:0/2784944034 wait complete. 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 -- start start 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 Processor -- start 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405dcef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558405cd3fc0 con 0x558405dc8030 2024-03-24T06:50:01.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558405cd3c60 con 0x558405dd1530 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.761+0000 7f2120fdf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558405a770c0 con 0x558405dc8950 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2120fdf740 10 client.0 _ll_register_callbacks cb 0x558405cebe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2117e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405dcef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2117400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51040/0 (socket says 172.21.15.22:51040) 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 -- 172.21.15.22:0/1314929175 learned_addr learned my addr 172.21.15.22:0/1314929175 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 -- 172.21.15.22:0/1314929175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 msgr2=0x558405dcef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.767 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405dcef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 -- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 msgr2=0x558405dce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2116a00640 1 -- 172.21.15.22:0/1314929175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2100009920 con 0x558405dc8950 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.762+0000 7f2117e00640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405dcef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2116a00640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f210000f760 tx=0x7f21000097e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d06ce3e1f825b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2104c00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100004310 con 0x558405dc8950 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558405d53e90 con 0x558405dc8950 2024-03-24T06:50:01.768 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f2100004470 con 0x558405dc8950 2024-03-24T06:50:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2100012450 con 0x558405dc8950 2024-03-24T06:50:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2120fdf740 10 client.27829 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:01.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.763+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558405d2d660 con 0x558405dc8950 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558405d53e90 con 0x558405dc8950 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f2100022070 con 0x558405dc8950 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 10 client.27829 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 10 client.27829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 20 client.27829 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405de1870 0x558405de3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 4 client.27829 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558405d25210 con 0x558405de1870 2024-03-24T06:50:01.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.764+0000 7f2117400640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405de1870 0x558405de3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.765+0000 7f2117400640 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405de1870 0x558405de3cd0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa8b6bca44799b2b server_cookie=9c81454b63d699c in_seq=0 out_seq=0 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.765+0000 7f2104c00640 10 client.27829 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2104c00640 1 -- 172.21.15.22:0/1314929175 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558405d25210 con 0x558405de1870 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2104c00640 10 client.27829 handle_command_reply: tid=0 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 client.27829 shutdown 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 msgr2=0x558405dce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f210000f760 tx=0x7f21000097e0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 shutdown_connections 2024-03-24T06:50:01.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405de1870 0x558405de3cd0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405dcef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405dce9c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2104c00640 0 client.27829 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 --2- 172.21.15.22:0/1314929175 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.766+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 >> 172.21.15.22:0/1314929175 conn(0x558405dc2760 msgr2=0x558405dc44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.767+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 shutdown_connections 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.767+0000 7f2120fdf740 1 -- 172.21.15.22:0/1314929175 wait complete. 2024-03-24T06:50:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.767+0000 7f2120fdf740 20 client.27829 trim_cache size 0 max 16384 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.786+0000 7f2120fdf740 1 Processor -- start 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.786+0000 7f2120fdf740 1 -- start start 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.786+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dd0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.786+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405de95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2120fdf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558405ddd330 con 0x558405dc8030 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2120fdf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558405dbdb60 con 0x558405dd1530 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2120fdf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558405dc3760 con 0x558405dc8950 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2117400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dd0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51048/0 (socket says 172.21.15.22:51048) 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 -- 172.21.15.22:0/2970981377 learned_addr learned my addr 172.21.15.22:0/2970981377 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2117e00640 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405de95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 -- 172.21.15.22:0/2970981377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 msgr2=0x558405de95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405de95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 -- 172.21.15.22:0/2970981377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 msgr2=0x558405dd0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dd0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 -- 172.21.15.22:0/2970981377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558405cc9ae0 con 0x558405dc8950 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.787+0000 7f2116a00640 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f210000a010 tx=0x7f21000117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dccf657df4f93d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.788+0000 7f2107400640 1 -- 172.21.15.22:0/2970981377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000035f0 con 0x558405dc8950 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.788+0000 7f2107400640 1 -- 172.21.15.22:0/2970981377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2100009530 con 0x558405dc8950 2024-03-24T06:50:01.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.788+0000 7f2107400640 1 -- 172.21.15.22:0/2970981377 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21000052a0 con 0x558405dc8950 2024-03-24T06:50:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- 172.21.15.22:0/2970981377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 msgr2=0x558405ddcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f210000a010 tx=0x7f21000117b0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- 172.21.15.22:0/2970981377 shutdown_connections 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dd1530 0x558405de95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dc8950 0x558405ddcdf0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 --2- 172.21.15.22:0/2970981377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8030 0x558405dd0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- 172.21.15.22:0/2970981377 >> 172.21.15.22:0/2970981377 conn(0x558405dc2760 msgr2=0x558405dc77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- 172.21.15.22:0/2970981377 shutdown_connections 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- 172.21.15.22:0/2970981377 wait complete. 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.789+0000 7f2120fdf740 1 -- start start 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 Processor -- start 2024-03-24T06:50:01.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x558405e46c10 con 0x558405dc8950 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558405e46d50 con 0x558405dc8030 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558405e46ec0 con 0x558405dd1530 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2120fdf740 10 client.0 _ll_register_callbacks cb 0x7f21080f43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2117400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2117400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49722/0 (socket says 172.21.15.22:49722) 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2117400640 1 -- 172.21.15.22:0/643429178 learned_addr learned my addr 172.21.15.22:0/643429178 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.790+0000 7f2116a00640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117e00640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 -- 172.21.15.22:0/643429178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 msgr2=0x558405dd5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 -- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 msgr2=0x558405dd5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 -- 172.21.15.22:0/643429178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558405d2d660 con 0x558405dc8030 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2117400640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f21080bce60 tx=0x7f210802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e667824b119bc927 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:01.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2104c00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210803c350 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7f21080bd860 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.791+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2100012a90 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f21080bd160 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 10 client.27809 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558405cef6f0 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2100012a90 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f210803c070 con 0x558405dc8030 2024-03-24T06:50:01.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 10 client.27809 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 10 client.27809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 20 client.27809 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405e483b0 0x558405e4a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 4 client.27809 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558405a2e860 con 0x558405e483b0 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.792+0000 7f2116a00640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405e483b0 0x558405e4a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.793+0000 7f2116a00640 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405e483b0 0x558405e4a7a0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35588392b6eafb83 server_cookie=a869708a2313dfd4 in_seq=0 out_seq=0 2024-03-24T06:50:01.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.793+0000 7f2104c00640 10 client.27809 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2104c00640 1 -- 172.21.15.22:0/643429178 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558405a2e860 con 0x558405e483b0 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2104c00640 10 client.27809 handle_command_reply: tid=0 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 client.27809 shutdown 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 msgr2=0x558405dd4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f21080bce60 tx=0x7f210802bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 shutdown_connections 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558405dd1530 0x558405dd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558405dc8950 0x558405dd5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x558405e483b0 0x558405e4a7a0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 --2- 172.21.15.22:0/643429178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558405dc8030 0x558405dd4e00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 >> 172.21.15.22:0/643429178 conn(0x558405dc2760 msgr2=0x558405dc0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.794+0000 7f2104c00640 0 client.27809 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:01.801 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.795+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 shutdown_connections 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.795+0000 7f2120fdf740 1 -- 172.21.15.22:0/643429178 wait complete. 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.795+0000 7f2120fdf740 20 client.27809 trim_cache size 0 max 16384 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stdout: "return_code": 0, 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_tag": "de2257fd-c51d-4ff8-8d7f-fec55c392ac6", 2024-03-24T06:50:01.802 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": "asynchronous" 2024-03-24T06:50:01.803 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:01.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 msgr2=0x7f20fc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.803 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 0x7f20fc0a4770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2110005fd0 tx=0x7f2110009500 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 msgr2=0x7f2118199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2108000c00 tx=0x7f210802c5f0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 shutdown_connections 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f20fc0a22b0 0x7f20fc0a4770 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2118106550 0x7f211819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2118100e50 0x7f211819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 --2- 172.21.15.22:0/1405073251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2118100490 0x7f2118199dd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 >> 172.21.15.22:0/1405073251 conn(0x7f21180fbf60 msgr2=0x7f2118068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:01.804 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 shutdown_connections 2024-03-24T06:50:01.805 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:01.796+0000 7f211cc00640 1 -- 172.21.15.22:0/1405073251 wait complete. 2024-03-24T06:50:01.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:50:01.823 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 -- 172.21.15.22:0/186565840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4107660 msgr2=0x7fe4b4107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 --2- 172.21.15.22:0/186565840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4107660 0x7fe4b4107a40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009930 tx=0x7fe4a402bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 -- 172.21.15.22:0/186565840 shutdown_connections 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 --2- 172.21.15.22:0/186565840 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4101f60 0x7fe4b410e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 --2- 172.21.15.22:0/186565840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b4101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 --2- 172.21.15.22:0/186565840 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4107660 0x7fe4b4107a40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 -- 172.21.15.22:0/186565840 >> 172.21.15.22:0/186565840 conn(0x7fe4b40fd070 msgr2=0x7fe4b40ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 -- 172.21.15.22:0/186565840 shutdown_connections 2024-03-24T06:50:02.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.016+0000 7fe4ba000640 1 -- 172.21.15.22:0/186565840 wait complete. 2024-03-24T06:50:02.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 Processor -- start 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 -- start start 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 0x7fe4b419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b419a690 con 0x7fe4b4107660 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b419a800 con 0x7fe4b41015a0 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.018+0000 7fe4ba000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b419a9c0 con 0x7fe4b4101f60 2024-03-24T06:50:02.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4b8c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 0x7fe4b419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51064/0 (socket says 172.21.15.22:51064) 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4b8c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49726/0 (socket says 172.21.15.22:49726) 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 -- 172.21.15.22:0/284705604 learned_addr learned my addr 172.21.15.22:0/284705604 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 -- 172.21.15.22:0/284705604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 msgr2=0x7fe4b419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 -- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 msgr2=0x7fe4b419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 0x7fe4b419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4abe00640 1 -- 172.21.15.22:0/284705604 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a4009560 con 0x7fe4b4101f60 2024-03-24T06:50:02.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.019+0000 7fe4b9600640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 0x7fe4b419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:02.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4abe00640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00e9f0 tx=0x7fe49c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4b2000640 1 -- 172.21.15.22:0/284705604 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c009700 con 0x7fe4b4101f60 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4b2000640 1 -- 172.21.15.22:0/284705604 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe49c00cd80 con 0x7fe4b4101f60 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4b2000640 1 -- 172.21.15.22:0/284705604 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49c018920 con 0x7fe4b4101f60 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b419ac40 con 0x7fe4b4101f60 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.020+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b41a6aa0 con 0x7fe4b4101f60 2024-03-24T06:50:02.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.023+0000 7fe4b2000640 1 -- 172.21.15.22:0/284705604 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fe49c018a80 con 0x7fe4b4101f60 2024-03-24T06:50:02.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.024+0000 7fe4be383740 1 Processor -- start 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.024+0000 7fe4be383740 1 -- start start 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac20990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b2000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 0x7fe4980a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f1ab8f790 con 0x556f1ac2a2d0 2024-03-24T06:50:02.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f1ab8ccc0 con 0x556f1ac20f60 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4be383740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f1ab884a0 con 0x556f1ac21c40 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b2000640 1 -- 172.21.15.22:0/284705604 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fe49c014070 con 0x7fe4b4101f60 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac20990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b8c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4abe00640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 0x7fe4980a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac20990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:51080/0 (socket says 172.21.15.22:51080) 2024-03-24T06:50:02.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b9600640 1 -- 172.21.15.22:0/564343688 learned_addr learned my addr 172.21.15.22:0/564343688 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b8c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49738/0 (socket says 172.21.15.22:49738) 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.025+0000 7fe4b9600640 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 -- 172.21.15.22:0/564343688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 msgr2=0x556f1ac20990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac20990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 -- 172.21.15.22:0/564343688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 msgr2=0x556f1ac2c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 -- 172.21.15.22:0/564343688 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1ab49c20 con 0x556f1ac20f60 2024-03-24T06:50:02.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b9600640 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:02.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4abe00640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 0x7fe4980a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe49c0972a0 tx=0x7fe49c010e50 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009230 tx=0x7fe4a402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0af2621535981d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4a9c00640 1 -- 172.21.15.22:0/564343688 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4004800 con 0x556f1ac20f60 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4a9c00640 1 -- 172.21.15.22:0/564343688 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a4004960 con 0x556f1ac20f60 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.026+0000 7fe4a9c00640 1 -- 172.21.15.22:0/564343688 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4005030 con 0x556f1ac20f60 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 -- 172.21.15.22:0/564343688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 msgr2=0x556f1ac213e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009230 tx=0x7fe4a402c5d0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 -- 172.21.15.22:0/564343688 shutdown_connections 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac213e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 --2- 172.21.15.22:0/564343688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac20990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 -- 172.21.15.22:0/564343688 >> 172.21.15.22:0/564343688 conn(0x556f1abe7b30 msgr2=0x556f1ac1b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 -- 172.21.15.22:0/564343688 shutdown_connections 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.027+0000 7fe4be383740 1 -- 172.21.15.22:0/564343688 wait complete. 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 -- start start 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 Processor -- start 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac27800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac27d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f1ab2cf20 con 0x556f1ac21c40 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f1ab2cbc0 con 0x556f1ac20f60 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f1a8f61c0 con 0x556f1ac2a2d0 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.028+0000 7fe4be383740 10 client.0 _ll_register_callbacks cb 0x556f1a898be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49748/0 (socket says 172.21.15.22:49748) 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 -- 172.21.15.22:0/906123643 learned_addr learned my addr 172.21.15.22:0/906123643 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:02.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac27d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac27800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 -- 172.21.15.22:0/906123643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 msgr2=0x556f1ac27d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac27d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 -- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 msgr2=0x556f1ac27800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac27800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 -- 172.21.15.22:0/906123643 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b410ed20 con 0x556f1ac20f60 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4b9600640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00fc70 tx=0x7fe4ac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5cf667a0f67cb21 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.029+0000 7fe4a2000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac016040 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f1ab80160 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fe4ac00ebe0 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe4ac00eda0 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4be383740 10 client.27821 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556f1ab86b20 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.030+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac02b080 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe4ac0046d0 con 0x556f1ac20f60 2024-03-24T06:50:02.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 10 client.27821 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 10 client.27821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 20 client.27821 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1ac3a760 0x556f1ac3cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 4 client.27821 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f1a8877e0 con 0x556f1ac3a760 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.031+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1ac3a760 0x556f1ac3cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.032+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1ac3a760 0x556f1ac3cb80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d009978d0b565ba4 server_cookie=81e817b2d13981e4 in_seq=0 out_seq=0 2024-03-24T06:50:02.038 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.032+0000 7fe4a2000640 10 client.27821 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:02.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.032+0000 7fe4a2000640 1 -- 172.21.15.22:0/906123643 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f1a8877e0 con 0x556f1ac3a760 2024-03-24T06:50:02.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.032+0000 7fe4a2000640 10 client.27821 handle_command_reply: tid=0 2024-03-24T06:50:02.039 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 client.27821 shutdown 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 msgr2=0x556f1ac272c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00fc70 tx=0x7fe4ac004ab0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 shutdown_connections 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac27d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac27800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1ac3a760 0x556f1ac3cb80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 --2- 172.21.15.22:0/906123643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac272c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 >> 172.21.15.22:0/906123643 conn(0x556f1abe7b30 msgr2=0x556f1ac1d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4a2000640 0 client.27821 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:02.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 shutdown_connections 2024-03-24T06:50:02.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 1 -- 172.21.15.22:0/906123643 wait complete. 2024-03-24T06:50:02.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.033+0000 7fe4be383740 20 client.27821 trim_cache size 0 max 16384 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.052+0000 7fe4be383740 1 Processor -- start 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.052+0000 7fe4be383740 1 -- start start 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac35bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac1aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac1bf30 con 0x556f1ac2a2d0 2024-03-24T06:50:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac270d0 con 0x556f1ac20f60 2024-03-24T06:50:02.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4be383740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac175f0 con 0x556f1ac21c40 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49762/0 (socket says 172.21.15.22:49762) 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 -- 172.21.15.22:0/2940296848 learned_addr learned my addr 172.21.15.22:0/2940296848 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac35bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac1aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 -- 172.21.15.22:0/2940296848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 msgr2=0x556f1ac35bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac35bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 -- 172.21.15.22:0/2940296848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 msgr2=0x556f1ac1aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.053+0000 7fe4b9600640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac1aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4b9600640 1 -- 172.21.15.22:0/2940296848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1ab229a0 con 0x556f1ac20f60 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac35bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4b9600640 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00f7e0 tx=0x7fe4ac004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f53bb14ceeff74f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4a9200640 1 -- 172.21.15.22:0/2940296848 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac011520 con 0x556f1ac20f60 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4a9200640 1 -- 172.21.15.22:0/2940296848 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556f1ab80160 con 0x556f1ac20f60 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.054+0000 7fe4a9200640 1 -- 172.21.15.22:0/2940296848 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac003970 con 0x556f1ac20f60 2024-03-24T06:50:02.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- 172.21.15.22:0/2940296848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 msgr2=0x556f1ac1b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00f7e0 tx=0x7fe4ac004a40 comp rx=0 tx=0).stop 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- 172.21.15.22:0/2940296848 shutdown_connections 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac2a2d0 0x556f1ac1aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac21c40 0x556f1ac35bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 --2- 172.21.15.22:0/2940296848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac1b4e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- 172.21.15.22:0/2940296848 >> 172.21.15.22:0/2940296848 conn(0x556f1abe7b30 msgr2=0x556f1ac1a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- 172.21.15.22:0/2940296848 shutdown_connections 2024-03-24T06:50:02.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- 172.21.15.22:0/2940296848 wait complete. 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 -- start start 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.055+0000 7fe4be383740 1 Processor -- start 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac2b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac32730 con 0x556f1ac21c40 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac328a0 con 0x556f1ac20f60 2024-03-24T06:50:02.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556f1ac32a60 con 0x556f1ac2a2d0 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4be383740 10 client.0 _ll_register_callbacks cb 0x7fe4ac0119f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4b9600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49770/0 (socket says 172.21.15.22:49770) 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4b9600640 1 -- 172.21.15.22:0/1869217218 learned_addr learned my addr 172.21.15.22:0/1869217218 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4b9600640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.056+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac2b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 -- 172.21.15.22:0/1869217218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 msgr2=0x556f1ac2b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 -- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 msgr2=0x556f1ac2b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:02.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac2b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 -- 172.21.15.22:0/1869217218 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f1ab49c20 con 0x556f1ac20f60 2024-03-24T06:50:02.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4b9600640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00f010 tx=0x7fe4ac00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3131314464a6a6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:02.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4a2000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:02.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac003970 con 0x556f1ac20f60 2024-03-24T06:50:02.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9467+0+0 (secure 0 0 0) 0x7fe4ac031040 con 0x556f1ac20f60 2024-03-24T06:50:02.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4a4004960 con 0x556f1ac20f60 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe4ac003d60 con 0x556f1ac20f60 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4be383740 10 client.27827 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.057+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556f1ab86b20 con 0x556f1ac20f60 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a4004960 con 0x556f1ac20f60 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fe4ac004560 con 0x556f1ac20f60 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 10 client.27827 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 10 client.27827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 20 client.27827 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:02.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1aca1620 0x556f1aca3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 4 client.27827 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556f1a8c4720 con 0x556f1aca1620 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.058+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1aca1620 0x556f1aca3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4b8c00640 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1aca1620 0x556f1aca3a10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62d05e783094644a server_cookie=a2ecdc484d33391f in_seq=0 out_seq=0 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4a2000640 10 client.27827 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4a2000640 1 -- 172.21.15.22:0/1869217218 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556f1a8c4720 con 0x556f1aca1620 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4a2000640 10 client.27827 handle_command_reply: tid=0 2024-03-24T06:50:02.066 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 client.27827 shutdown 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 msgr2=0x556f1ac321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00f010 tx=0x7fe4ac00ed30 comp rx=0 tx=0).stop 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 shutdown_connections 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556f1ac2a2d0 0x556f1ac2b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556f1ac21c40 0x556f1ac2b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x556f1aca1620 0x556f1aca3a10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 --2- 172.21.15.22:0/1869217218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556f1ac20f60 0x556f1ac321f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4a2000640 0 client.27827 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.059+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 >> 172.21.15.22:0/1869217218 conn(0x556f1abe7b30 msgr2=0x556f1ac1abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 shutdown_connections 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4be383740 1 -- 172.21.15.22:0/1869217218 wait complete. 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4be383740 20 client.27827 trim_cache size 0 max 16384 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": { 2024-03-24T06:50:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "de2257fd-c51d-4ff8-8d7f-fec55c392ac6": { 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "path": "/", 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "tag": "de2257fd-c51d-4ff8-8d7f-fec55c392ac6", 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "options": "recursive,force" 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:02.069 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 msgr2=0x7fe4980a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 0x7fe4980a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe49c0972a0 tx=0x7fe49c010e50 comp rx=0 tx=0).stop 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 msgr2=0x7fe4b419ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00e9f0 tx=0x7fe49c00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 shutdown_connections 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fe4980a22b0 0x7fe4980a4770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.070 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe4b4107660 0x7fe4b419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4b4101f60 0x7fe4b419ffd0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.060+0000 7fe4ba000640 1 --2- 172.21.15.22:0/284705604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4b41015a0 0x7fe4b419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:02.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.061+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 >> 172.21.15.22:0/284705604 conn(0x7fe4b40fd070 msgr2=0x7fe4b40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:02.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.061+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 shutdown_connections 2024-03-24T06:50:02.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:02.061+0000 7fe4ba000640 1 -- 172.21.15.22:0/284705604 wait complete. 2024-03-24T06:50:02.080 INFO:tasks.cephfs.filesystem:scrub status for tag:de2257fd-c51d-4ff8-8d7f-fec55c392ac6 - {'path': '/', 'tag': 'de2257fd-c51d-4ff8-8d7f-fec55c392ac6', 'options': 'recursive,force'} 2024-03-24T06:50:02.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:02.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:50:02.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:50:02.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:50:02.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:50:02.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:02 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:50:02.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:02 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: pgmap v311: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:02.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:02 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:03 smithi022 ceph-mon[32003]: scrub queued for path: / 2024-03-24T06:50:03.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:03 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:03 smithi022 ceph-mon[32003]: scrub summary: active paths [/] 2024-03-24T06:50:03.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:03 smithi022 ceph-mon[32003]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:03 smithi143 ceph-mon[36520]: scrub queued for path: / 2024-03-24T06:50:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:03 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:03 smithi143 ceph-mon[36520]: scrub summary: active paths [/] 2024-03-24T06:50:03.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:03 smithi143 ceph-mon[36520]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:03 smithi155 ceph-mon[36262]: scrub queued for path: / 2024-03-24T06:50:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:03 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:03.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:03 smithi155 ceph-mon[36262]: scrub summary: active paths [/] 2024-03-24T06:50:03.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:03 smithi155 ceph-mon[36262]: scrub summary: idle+waiting paths [/] 2024-03-24T06:50:04.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:04 smithi022 ceph-mon[32003]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:50:04.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:04 smithi143 ceph-mon[36520]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:50:04.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:04 smithi155 ceph-mon[36262]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:50:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:06 smithi022 ceph-mon[32003]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:50:06.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:06 smithi143 ceph-mon[36520]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:50:06.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:06 smithi155 ceph-mon[36262]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:50:08.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:08 smithi022 ceph-mon[32003]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:50:08.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:08 smithi143 ceph-mon[36520]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:50:08.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:08 smithi155 ceph-mon[36262]: pgmap v314: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:50:09.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:09 smithi022 ceph-mon[32003]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T06:50:09.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:09 smithi022 ceph-mon[32003]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:09 smithi143 ceph-mon[36520]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T06:50:09.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:09 smithi143 ceph-mon[36520]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:09.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:09 smithi155 ceph-mon[36262]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T06:50:09.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:09 smithi155 ceph-mon[36262]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:12.353 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:12 smithi022 ceph-mon[32003]: pgmap v317: 129 pgs: 8 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 96 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-24T06:50:12.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:12 smithi143 ceph-mon[36520]: pgmap v317: 129 pgs: 8 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 96 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-24T06:50:12.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:12 smithi155 ceph-mon[36262]: pgmap v317: 129 pgs: 8 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 96 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-24T06:50:13.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:50:13.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:50:13.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:50:13.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:13 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:50:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:50:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:50:13.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:50:13.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:13 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:50:13.471 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T06:50:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:50:13.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:50:13.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:50:13.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:13 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:50:13.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 -- 172.21.15.22:0/3986722322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 msgr2=0x7f3954109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 --2- 172.21.15.22:0/3986722322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f3954109cc0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3944009900 tx=0x7f394402bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 -- 172.21.15.22:0/3986722322 shutdown_connections 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 --2- 172.21.15.22:0/3986722322 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3954104290 0x7f39541107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 --2- 172.21.15.22:0/3986722322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39541038d0 0x7f3954103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 --2- 172.21.15.22:0/3986722322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f3954109cc0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:13.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 -- 172.21.15.22:0/3986722322 >> 172.21.15.22:0/3986722322 conn(0x7f39540ff400 msgr2=0x7f3954101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:13.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 -- 172.21.15.22:0/3986722322 shutdown_connections 2024-03-24T06:50:13.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.749+0000 7f3958c00640 1 -- 172.21.15.22:0/3986722322 wait complete. 2024-03-24T06:50:13.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.750+0000 7f3958c00640 1 Processor -- start 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 -- start start 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 0x7f3954199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3954104290 0x7f395419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f395419ed80 con 0x7f39541038d0 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f395419eef0 con 0x7f39541098e0 2024-03-24T06:50:13.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.751+0000 7f3958c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f395419f0b0 con 0x7f3954104290 2024-03-24T06:50:13.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:13.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3951e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 0x7f3954199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:13.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3951e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 0x7f3954199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60302/0 (socket says 172.21.15.22:60302) 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49368/0 (socket says 172.21.15.22:49368) 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 -- 172.21.15.22:0/2754923390 learned_addr learned my addr 172.21.15.22:0/2754923390 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3951400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3954104290 0x7f395419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 -- 172.21.15.22:0/2754923390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3954104290 msgr2=0x7f395419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3954104290 0x7f395419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:13.754 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 -- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 msgr2=0x7f3954199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:13.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 0x7f3954199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:13.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.752+0000 7f3952800640 1 -- 172.21.15.22:0/2754923390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3944009560 con 0x7f39541098e0 2024-03-24T06:50:13.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.753+0000 7f3952800640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f394c00eaa0 tx=0x7f394c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:13.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.753+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c00cdb0 con 0x7f39541098e0 2024-03-24T06:50:13.755 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.753+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f394c005020 con 0x7f39541098e0 2024-03-24T06:50:13.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.753+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c0116b0 con 0x7f39541098e0 2024-03-24T06:50:13.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.754+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395419f390 con 0x7f39541098e0 2024-03-24T06:50:13.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.754+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39541a6c10 con 0x7f39541098e0 2024-03-24T06:50:13.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.755+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f395404f470 con 0x7f39541098e0 2024-03-24T06:50:13.758 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.758+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f394c0054c0 con 0x7f39541098e0 2024-03-24T06:50:13.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.759+0000 7f394aa00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 0x7f39380a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:13.759 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.759+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f394c0ce580 con 0x7f39541098e0 2024-03-24T06:50:13.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.760+0000 7f3951e00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 0x7f39380a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:13.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.761+0000 7f3951e00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 0x7f39380a48e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3944000c00 tx=0x7f3944005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:13.766 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:13.766+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f394c096ed0 con 0x7f39541098e0 2024-03-24T06:50:14.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.063+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3954113bc0 con 0x7f39380a2420 2024-03-24T06:50:14.080 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.080+0000 7f394aa00640 1 -- 172.21.15.22:0/2754923390 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f3954113bc0 con 0x7f39380a2420 2024-03-24T06:50:14.080 INFO:teuthology.orchestra.run.smithi022.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T06:40:10", "first": "2024-03-24T06:41:00", "last": "2024-03-24T06:50:00", "last_pruned": "2024-03-24T06:50:01", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-24T06:50:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 msgr2=0x7f39380a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 0x7f39380a48e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3944000c00 tx=0x7f3944005fd0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 msgr2=0x7f395419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f394c00eaa0 tx=0x7f394c00ef60 comp rx=0 tx=0).stop 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 shutdown_connections 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f39380a2420 0x7f39380a48e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39541098e0 0x7f395419e780 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3954104290 0x7f395419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 --2- 172.21.15.22:0/2754923390 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39541038d0 0x7f3954199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 >> 172.21.15.22:0/2754923390 conn(0x7f39540ff400 msgr2=0x7f39540ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:14.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 shutdown_connections 2024-03-24T06:50:14.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.083+0000 7f3958c00640 1 -- 172.21.15.22:0/2754923390 wait complete. 2024-03-24T06:50:14.099 DEBUG:teuthology.orchestra.run.smithi022:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T06:50:14.279 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.278+0000 7f310f400640 1 -- 172.21.15.22:0/1815940271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108100860 msgr2=0x7f3108100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.278+0000 7f310f400640 1 --2- 172.21.15.22:0/1815940271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108100860 0x7f3108100ce0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009930 tx=0x7f30f402bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 -- 172.21.15.22:0/1815940271 shutdown_connections 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 --2- 172.21.15.22:0/1815940271 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3108101220 0x7f310810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 --2- 172.21.15.22:0/1815940271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108100860 0x7f3108100ce0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 --2- 172.21.15.22:0/1815940271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31081055b0 0x7f3108105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 -- 172.21.15.22:0/1815940271 >> 172.21.15.22:0/1815940271 conn(0x7f3108077460 msgr2=0x7f3108077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:14.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 -- 172.21.15.22:0/1815940271 shutdown_connections 2024-03-24T06:50:14.281 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.279+0000 7f310f400640 1 -- 172.21.15.22:0/1815940271 wait complete. 2024-03-24T06:50:14.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.280+0000 7f310f400640 1 Processor -- start 2024-03-24T06:50:14.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.280+0000 7f310f400640 1 -- start start 2024-03-24T06:50:14.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 0x7f3108197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:14.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 0x7f31081983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:14.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 0x7f310819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f310819cd30 con 0x7f31081055b0 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f310819cea0 con 0x7f3108100860 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310f400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f310819d060 con 0x7f3108101220 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 0x7f3108197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310ca00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 0x7f3108197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:49392/0 (socket says 172.21.15.22:49392) 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f3107e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 0x7f31081983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:14.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310ca00640 1 -- 172.21.15.22:0/2671968759 learned_addr learned my addr 172.21.15.22:0/2671968759 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.281+0000 7f310d400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 0x7f310819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 -- 172.21.15.22:0/2671968759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 msgr2=0x7f3108197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 0x7f3108197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 -- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 msgr2=0x7f310819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 0x7f310819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 -- 172.21.15.22:0/2671968759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f4009560 con 0x7f3108101220 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f310d400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 0x7f310819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:14.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3107e00640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 0x7f31081983b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f30f4036040 tx=0x7f30f4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:14.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.282+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4004440 con 0x7f3108101220 2024-03-24T06:50:14.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.283+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310819d2e0 con 0x7f3108101220 2024-03-24T06:50:14.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.283+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30f40052d0 con 0x7f3108101220 2024-03-24T06:50:14.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.283+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4003d00 con 0x7f3108101220 2024-03-24T06:50:14.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.283+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31081a4bc0 con 0x7f3108101220 2024-03-24T06:50:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.284+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310804f470 con 0x7f3108101220 2024-03-24T06:50:14.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.286+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f30f4004df0 con 0x7f3108101220 2024-03-24T06:50:14.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.288+0000 7f3105600640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 0x7f30f00a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:14.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.288+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9555+0+0 (secure 0 0 0) 0x7f30f40f1190 con 0x7f3108101220 2024-03-24T06:50:14.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.289+0000 7f310ca00640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 0x7f30f00a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:14.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.289+0000 7f310ca00640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 0x7f30f00a4790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f30fc005fd0 tx=0x7f30fc00b4a0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:14.294 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.293+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30f40b9ac0 con 0x7f3108101220 2024-03-24T06:50:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:14 smithi022 ceph-mon[32003]: scrub summary: idle 2024-03-24T06:50:14.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:14 smithi022 ceph-mon[32003]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:14.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:14 smithi155 ceph-mon[36262]: scrub summary: idle 2024-03-24T06:50:14.428 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:14 smithi155 ceph-mon[36262]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:14 smithi143 ceph-mon[36520]: scrub summary: idle 2024-03-24T06:50:14.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:14 smithi143 ceph-mon[36520]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:50:14.607 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.607+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 --> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f31081119b0 con 0x7f30f00a22d0 2024-03-24T06:50:14.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.615+0000 7f3105600640 1 -- 172.21.15.22:0/2671968759 <== mgr.25492 v2:172.21.15.22:6800/1309982651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f31081119b0 con 0x7f30f00a22d0 2024-03-24T06:50:14.616 INFO:teuthology.orchestra.run.smithi022.stdout:/ 1m 6m3h 2024-03-24T06:50:14.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 msgr2=0x7f30f00a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 0x7f30f00a4790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f30fc005fd0 tx=0x7f30fc00b4a0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 msgr2=0x7f31081983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 0x7f31081983b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f30f4036040 tx=0x7f30f4004040 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 shutdown_connections 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f30f00a22d0 0x7f30f00a4790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31081055b0 0x7f310819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108101220 0x7f31081983b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 --2- 172.21.15.22:0/2671968759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3108100860 0x7f3108197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:14.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 >> 172.21.15.22:0/2671968759 conn(0x7f3108077460 msgr2=0x7f31080ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:14.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 shutdown_connections 2024-03-24T06:50:14.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:14.618+0000 7f310f400640 1 -- 172.21.15.22:0/2671968759 wait complete. 2024-03-24T06:50:14.637 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-24T06:50:14.647 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-24T06:50:15.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:15 smithi022 ceph-mon[32003]: from='client.27833 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:15.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:15 smithi143 ceph-mon[36520]: from='client.27833 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:15.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:15 smithi155 ceph-mon[36262]: from='client.27833 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:16 smithi022 ceph-mon[32003]: from='client.27865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:16 smithi022 ceph-mon[32003]: pgmap v319: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T06:50:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:16 smithi143 ceph-mon[36520]: from='client.27865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:16 smithi143 ceph-mon[36520]: pgmap v319: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T06:50:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:16 smithi155 ceph-mon[36262]: from='client.27865 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T06:50:16.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:16 smithi155 ceph-mon[36262]: pgmap v319: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 12 op/s 2024-03-24T06:50:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:18 smithi022 ceph-mon[32003]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T06:50:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:18 smithi143 ceph-mon[36520]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T06:50:18.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:18 smithi155 ceph-mon[36262]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 204 B/s wr, 11 op/s 2024-03-24T06:50:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:20 smithi022 ceph-mon[32003]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 191 B/s wr, 11 op/s 2024-03-24T06:50:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:20 smithi022 ceph-mon[32003]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T06:50:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:20 smithi143 ceph-mon[36520]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 191 B/s wr, 11 op/s 2024-03-24T06:50:20.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:20 smithi143 ceph-mon[36520]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T06:50:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:20 smithi155 ceph-mon[36262]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 191 B/s wr, 11 op/s 2024-03-24T06:50:20.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:20 smithi155 ceph-mon[36262]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T06:50:22.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:22 smithi022 ceph-mon[32003]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T06:50:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:22 smithi143 ceph-mon[36520]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T06:50:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:22 smithi155 ceph-mon[36262]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T06:50:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:24 smithi022 ceph-mon[32003]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T06:50:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:24 smithi143 ceph-mon[36520]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T06:50:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:24 smithi155 ceph-mon[36262]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T06:50:26.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:26 smithi022 ceph-mon[32003]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 102 B/s wr, 11 op/s 2024-03-24T06:50:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:26 smithi143 ceph-mon[36520]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 102 B/s wr, 11 op/s 2024-03-24T06:50:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:26 smithi155 ceph-mon[36262]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 102 B/s wr, 11 op/s 2024-03-24T06:50:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:28 smithi022 ceph-mon[32003]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:28 smithi143 ceph-mon[36520]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:28 smithi155 ceph-mon[36262]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:30.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:30.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:50:30.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.279 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:30 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:30.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:50:30.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:30 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T06:50:30.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T06:50:30.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:30.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:30 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T06:50:32.081 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T06:50:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:31 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:31 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:31 smithi155 ceph-mon[36262]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 -- 172.21.15.22:0/1686507130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c0100da0 msgr2=0x7fa9c010d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/1686507130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c0100da0 0x7fa9c010d9d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009860 tx=0x7fa9a802bb70 comp rx=0 tx=0).stop 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 -- 172.21.15.22:0/1686507130 shutdown_connections 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/1686507130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c010dfa0 0x7fa9c0110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/1686507130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c0100da0 0x7fa9c010d9d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/1686507130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c01003f0 0x7fa9c01007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.292 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 -- 172.21.15.22:0/1686507130 >> 172.21.15.22:0/1686507130 conn(0x7fa9c00fc020 msgr2=0x7fa9c00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.291+0000 7fa9bfe00640 1 -- 172.21.15.22:0/1686507130 shutdown_connections 2024-03-24T06:50:32.293 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.292+0000 7fa9bfe00640 1 -- 172.21.15.22:0/1686507130 wait complete. 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.292+0000 7fa9bfe00640 1 Processor -- start 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 -- start start 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c0100da0 0x7fa9c019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c010dfa0 0x7fa9c019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c019edb0 con 0x7fa9c0100da0 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c019ef20 con 0x7fa9c01003f0 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bfe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c019f0e0 con 0x7fa9c010dfa0 2024-03-24T06:50:32.295 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9b7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c0100da0 0x7fa9c019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bea00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:42920/0 (socket says 172.21.15.22:42920) 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bea00640 1 -- 172.21.15.22:0/2854913547 learned_addr learned my addr 172.21.15.22:0/2854913547 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.293+0000 7fa9bf400640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c010dfa0 0x7fa9c019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 -- 172.21.15.22:0/2854913547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c010dfa0 msgr2=0x7fa9c019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c010dfa0 0x7fa9c019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 -- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c0100da0 msgr2=0x7fa9c019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c0100da0 0x7fa9c019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.296 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 -- 172.21.15.22:0/2854913547 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b0009560 con 0x7fa9c01003f0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bea00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000e9c0 tx=0x7fa9b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9c4a00640 1 -- 172.21.15.22:0/2854913547 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000cde0 con 0x7fa9c01003f0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9c4a00640 1 -- 172.21.15.22:0/2854913547 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9b00035f0 con 0x7fa9c01003f0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9c4a00640 1 -- 172.21.15.22:0/2854913547 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0010670 con 0x7fa9c01003f0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a8009490 con 0x7fa9c01003f0 2024-03-24T06:50:32.297 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.294+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c01a6a60 con 0x7fa9c01003f0 2024-03-24T06:50:32.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c4a00640 1 -- 172.21.15.22:0/2854913547 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fa9b0003ad0 con 0x7fa9c01003f0 2024-03-24T06:50:32.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 Processor -- start 2024-03-24T06:50:32.298 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 -- start start 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8658790 con 0x55d0d86f32d0 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8655cc0 con 0x55d0d86eac40 2024-03-24T06:50:32.300 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.298+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d0d86514a0 con 0x55d0d86e9f60 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bea00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:42924/0 (socket says 172.21.15.22:42924) 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 -- 172.21.15.22:0/1351325310 learned_addr learned my addr 172.21.15.22:0/1351325310 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 -- 172.21.15.22:0/1351325310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 msgr2=0x55d0d86ea3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86ea3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 -- 172.21.15.22:0/1351325310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 msgr2=0x55d0d86f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.301 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 -- 172.21.15.22:0/1351325310 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0d8612c20 con 0x55d0d86eac40 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009f30 tx=0x7fa9b80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b26c57c1843c26a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.299+0000 7fa9c4a00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 0x7fa9a40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9afe00640 1 -- 172.21.15.22:0/1351325310 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8004d20 con 0x55d0d86eac40 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9afe00640 1 -- 172.21.15.22:0/1351325310 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9b8004e80 con 0x55d0d86eac40 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9afe00640 1 -- 172.21.15.22:0/1351325310 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8005280 con 0x55d0d86eac40 2024-03-24T06:50:32.302 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9c4a00640 1 -- 172.21.15.22:0/2854913547 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7fa9b0014070 con 0x7fa9c01003f0 2024-03-24T06:50:32.303 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9bea00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 0x7fa9a40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1351325310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 msgr2=0x55d0d86e9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.300+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009f30 tx=0x7fa9b80118d0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9bea00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 0x7fa9a40a4770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55d0d86eb190 tx=0x7fa9b00d4530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1351325310 shutdown_connections 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86ea3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1351325310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86eac40 0x55d0d86e9990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1351325310 >> 172.21.15.22:0/1351325310 conn(0x55d0d86b0b30 msgr2=0x55d0d86e49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1351325310 shutdown_connections 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1351325310 wait complete. 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.301+0000 7fa9c8ba8740 1 -- start start 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 Processor -- start 2024-03-24T06:50:32.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d0d85f5f20 con 0x55d0d86f32d0 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d0d85f5bc0 con 0x55d0d86e9f60 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.302+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d0d83bf1c0 con 0x55d0d86eac40 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9c8ba8740 10 client.0 _ll_register_callbacks cb 0x55d0d8361be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42116/0 (socket says 172.21.15.22:42116) 2024-03-24T06:50:32.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 -- 172.21.15.22:0/1067831599 learned_addr learned my addr 172.21.15.22:0/1067831599 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 -- 172.21.15.22:0/1067831599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 msgr2=0x55d0d86f02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86f02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 -- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 msgr2=0x55d0d86f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9b7e00640 1 -- 172.21.15.22:0/1067831599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b8009be0 con 0x55d0d86eac40 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.303+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:32.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80057c0 tx=0x7fa9a8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8d93cb5f8e06f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9acc00640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8034640 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7fa9a80347a0 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0d83507e0 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9a8004500 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9c8ba8740 10 client.27877 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.304+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d0d864fb20 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d0d83507e0 con 0x55d0d86eac40 2024-03-24T06:50:32.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa9a8045020 con 0x55d0d86eac40 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 10 client.27877 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 10 client.27877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 20 client.27877 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d87036e0 0x55d0d8705b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 4 client.27877 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0d838d720 con 0x55d0d87036e0 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.305+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d87036e0 0x55d0d8705b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.306+0000 7fa9bf400640 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d87036e0 0x55d0d8705b00 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f71502832c5fd418 server_cookie=34393c49e2660442 in_seq=0 out_seq=0 2024-03-24T06:50:32.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.306+0000 7fa9acc00640 10 client.27877 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:32.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9acc00640 1 -- 172.21.15.22:0/1067831599 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0d838d720 con 0x55d0d87036e0 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9acc00640 10 client.27877 handle_command_reply: tid=0 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 client.27877 shutdown 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 msgr2=0x55d0d86f0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80057c0 tx=0x7fa9a8004040 comp rx=0 tx=0).stop 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 shutdown_connections 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86f32d0 0x55d0d86f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86eac40 0x55d0d86f0820 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d87036e0 0x55d0d8705b00 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/1067831599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86f02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 >> 172.21.15.22:0/1067831599 conn(0x55d0d86b0b30 msgr2=0x55d0d86e6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9acc00640 0 client.27877 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:32.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 shutdown_connections 2024-03-24T06:50:32.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.307+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/1067831599 wait complete. 2024-03-24T06:50:32.314 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.308+0000 7fa9c8ba8740 20 client.27877 trim_cache size 0 max 16384 2024-03-24T06:50:32.327 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 Processor -- start 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 -- start start 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86ef220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86f32d0 0x55d0d870b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d0d86e9800 con 0x55d0d86eac40 2024-03-24T06:50:32.328 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8700330 con 0x55d0d86e9f60 2024-03-24T06:50:32.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.327+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d0d86f2d40 con 0x55d0d86f32d0 2024-03-24T06:50:32.330 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86ef220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:42952/0 (socket says 172.21.15.22:42952) 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9bf400640 1 -- 172.21.15.22:0/917713684 learned_addr learned my addr 172.21.15.22:0/917713684 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60798/0 (socket says 172.21.15.22:60798) 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9bf400640 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86f32d0 0x55d0d870b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 -- 172.21.15.22:0/917713684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86f32d0 msgr2=0x55d0d870b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86f32d0 0x55d0d870b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 -- 172.21.15.22:0/917713684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 msgr2=0x55d0d86ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.328+0000 7fa9b7e00640 1 -- 172.21.15.22:0/917713684 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0d85eb9a0 con 0x55d0d86eac40 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.329+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8003ca0 tx=0x7fa9a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d01aeb89f1a90c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.329+0000 7fa9af400640 1 -- 172.21.15.22:0/917713684 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80037f0 con 0x55d0d86eac40 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.329+0000 7fa9af400640 1 -- 172.21.15.22:0/917713684 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a80040b0 con 0x55d0d86eac40 2024-03-24T06:50:32.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.329+0000 7fa9af400640 1 -- 172.21.15.22:0/917713684 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a802c790 con 0x55d0d86eac40 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/917713684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 msgr2=0x55d0d86fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8003ca0 tx=0x7fa9a8000bc0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/917713684 shutdown_connections 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86f32d0 0x55d0d870b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86fec20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/917713684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86e9f60 0x55d0d86ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.333 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/917713684 >> 172.21.15.22:0/917713684 conn(0x55d0d86b0b30 msgr2=0x55d0d86e6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/917713684 shutdown_connections 2024-03-24T06:50:32.334 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.330+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/917713684 wait complete. 2024-03-24T06:50:32.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.331+0000 7fa9c8ba8740 1 -- start start 2024-03-24T06:50:32.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.331+0000 7fa9c8ba8740 1 Processor -- start 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86f32d0 0x55d0d86f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8768a00 con 0x55d0d86eac40 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8768b40 con 0x55d0d86f32d0 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d0d8768c80 con 0x55d0d86e9f60 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9c8ba8740 10 client.0 _ll_register_callbacks cb 0x7fa9b80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:32.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42126/0 (socket says 172.21.15.22:42126) 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9bf400640 1 -- 172.21.15.22:0/2369929248 learned_addr learned my addr 172.21.15.22:0/2369929248 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9bf400640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86f32d0 0x55d0d86f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 -- 172.21.15.22:0/2369929248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 msgr2=0x55d0d86f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 -- 172.21.15.22:0/2369929248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86f32d0 msgr2=0x55d0d86f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86f32d0 0x55d0d86f75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.332+0000 7fa9b7e00640 1 -- 172.21.15.22:0/2369929248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0d864fb20 con 0x55d0d86eac40 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9b7e00640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86f7090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55d0d86ea5b0 tx=0x7fa9a80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5211fbeced5b7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9acc00640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8005850 con 0x55d0d86eac40 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7fa9a8000e40 con 0x55d0d86eac40 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0d83dc890 con 0x55d0d86eac40 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9a8004040 con 0x55d0d86eac40 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.333+0000 7fa9c8ba8740 10 client.17490 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:32.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d0d8611480 con 0x55d0d86eac40 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d0d83dc890 con 0x55d0d86eac40 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa9a805b070 con 0x55d0d86eac40 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 10 client.17490 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 20 client.17490 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d876a060 0x55d0d876c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 4 client.17490 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T06:50:32.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.334+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0d82fc700 con 0x55d0d876a060 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.335+0000 7fa9bf400640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d876a060 0x55d0d876c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.335+0000 7fa9bf400640 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d876a060 0x55d0d876c450 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=180e3a31bbe4c2c8 server_cookie=7db73910992bfb1d in_seq=0 out_seq=0 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.335+0000 7fa9acc00640 10 client.17490 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9acc00640 1 -- 172.21.15.22:0/2369929248 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d0d82fc700 con 0x55d0d876a060 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9acc00640 10 client.17490 handle_command_reply: tid=0 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 client.17490 shutdown 2024-03-24T06:50:32.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 msgr2=0x55d0d86f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86f7090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55d0d86ea5b0 tx=0x7fa9a80038a0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 shutdown_connections 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x55d0d876a060 0x55d0d876c450 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d0d86f32d0 0x55d0d86f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55d0d86eac40 0x55d0d86f7090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 --2- 172.21.15.22:0/2369929248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d0d86e9f60 0x55d0d86f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 >> 172.21.15.22:0/2369929248 conn(0x55d0d86b0b30 msgr2=0x55d0d86f18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9acc00640 0 client.17490 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:32.343 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 shutdown_connections 2024-03-24T06:50:32.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 1 -- 172.21.15.22:0/2369929248 wait complete. 2024-03-24T06:50:32.344 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.336+0000 7fa9c8ba8740 20 client.17490 trim_cache size 0 max 16384 2024-03-24T06:50:32.344 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:32.344 INFO:teuthology.orchestra.run.smithi022.stdout: "status": "no active scrubs running", 2024-03-24T06:50:32.344 INFO:teuthology.orchestra.run.smithi022.stdout: "scrubs": {} 2024-03-24T06:50:32.345 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:32.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 msgr2=0x7fa9a40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 0x7fa9a40a4770 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55d0d86eb190 tx=0x7fa9b00d4530 comp rx=0 tx=0).stop 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 msgr2=0x7fa9c0199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000e9c0 tx=0x7fa9b000ee80 comp rx=0 tx=0).stop 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 shutdown_connections 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fa9a40a22b0 0x7fa9a40a4770 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9c010dfa0 0x7fa9c019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9c0100da0 0x7fa9c019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 --2- 172.21.15.22:0/2854913547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c01003f0 0x7fa9c0199ef0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.337+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 >> 172.21.15.22:0/2854913547 conn(0x7fa9c00fc020 msgr2=0x7fa9c00fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.338+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 shutdown_connections 2024-03-24T06:50:32.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.338+0000 7fa9bfe00640 1 -- 172.21.15.22:0/2854913547 wait complete. 2024-03-24T06:50:32.354 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:31 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.354 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:31 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.355 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:31 smithi022 ceph-mon[32003]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:50:32.359 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T06:50:32.359 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:50:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:31 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:31 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:32.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:31 smithi143 ceph-mon[36520]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 -- 172.21.15.22:0/2940125678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 msgr2=0x7f2d54101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 --2- 172.21.15.22:0/2940125678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54101e20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2d40009930 tx=0x7f2d4002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 -- 172.21.15.22:0/2940125678 shutdown_connections 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 --2- 172.21.15.22:0/2940125678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d54102360 0x7f2d5410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 --2- 172.21.15.22:0/2940125678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54101e20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 --2- 172.21.15.22:0/2940125678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d541078b0 0x7f2d54107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 -- 172.21.15.22:0/2940125678 >> 172.21.15.22:0/2940125678 conn(0x7f2d54079700 msgr2=0x7f2d54079b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 -- 172.21.15.22:0/2940125678 shutdown_connections 2024-03-24T06:50:32.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.560+0000 7f2d5a600640 1 -- 172.21.15.22:0/2940125678 wait complete. 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.562+0000 7f2d5a600640 1 Processor -- start 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.562+0000 7f2d5a600640 1 -- start start 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d54102360 0x7f2d5419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 0x7f2d5419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5419ee50 con 0x7f2d541078b0 2024-03-24T06:50:32.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5419efc0 con 0x7f2d54102360 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d5a600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5419f180 con 0x7f2d541019a0 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d53400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42142/0 (socket says 172.21.15.22:42142) 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d53400640 1 -- 172.21.15.22:0/374173136 learned_addr learned my addr 172.21.15.22:0/374173136 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d52a00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d54102360 0x7f2d5419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.563+0000 7f2d53e00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 0x7f2d5419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 -- 172.21.15.22:0/374173136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d54102360 msgr2=0x7f2d5419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d54102360 0x7f2d5419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 -- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 msgr2=0x7f2d5419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 0x7f2d5419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 -- 172.21.15.22:0/374173136 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d40009560 con 0x7f2d541019a0 2024-03-24T06:50:32.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53e00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 0x7f2d5419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d53400640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e9f0 tx=0x7f2d4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44009700 con 0x7f2d541019a0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d4400cd80 con 0x7f2d541019a0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5419f460 con 0x7f2d541019a0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44018920 con 0x7f2d541019a0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.564+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d541a6ce0 con 0x7f2d541019a0 2024-03-24T06:50:32.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.566+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d54068500 con 0x7f2d541019a0 2024-03-24T06:50:32.568 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.568+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f2d44018a80 con 0x7f2d541019a0 2024-03-24T06:50:32.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.569+0000 7f2d4be00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 0x7f2d380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:32.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.570+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f2d44014070 con 0x7f2d541019a0 2024-03-24T06:50:32.570 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.570+0000 7f2d52a00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 0x7f2d380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:32.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.571+0000 7f2d52a00640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 0x7f2d380a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d40005ab0 tx=0x7f2d40005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:32.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.573+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2d44053280 con 0x7f2d541019a0 2024-03-24T06:50:32.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.904+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d54111870 con 0x7f2d541019a0 2024-03-24T06:50:32.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.905+0000 7f2d4be00640 1 -- 172.21.15.22:0/374173136 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f2d44097140 con 0x7f2d541019a0 2024-03-24T06:50:32.906 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:50:32.906 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:50:32.907 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:50:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.908+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 msgr2=0x7f2d380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.908+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 0x7f2d380a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d40005ab0 tx=0x7f2d40005fd0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.909 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 msgr2=0x7f2d54199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e9f0 tx=0x7f2d4400eeb0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 shutdown_connections 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f2d380a2230 0x7f2d380a46f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d541078b0 0x7f2d5419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d54102360 0x7f2d5419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 --2- 172.21.15.22:0/374173136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d541019a0 0x7f2d54199f90 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:32.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 >> 172.21.15.22:0/374173136 conn(0x7f2d54079700 msgr2=0x7f2d5410cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:32.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 shutdown_connections 2024-03-24T06:50:32.911 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:32.909+0000 7f2d5a600640 1 -- 172.21.15.22:0/374173136 wait complete. 2024-03-24T06:50:32.930 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 -- 172.21.15.22:0/4104036145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 msgr2=0x7f24b0101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 --2- 172.21.15.22:0/4104036145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0101a20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2498009930 tx=0x7f249802bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 -- 172.21.15.22:0/4104036145 shutdown_connections 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 --2- 172.21.15.22:0/4104036145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 0x7f24b010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 --2- 172.21.15.22:0/4104036145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0101a20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 --2- 172.21.15.22:0/4104036145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 0x7f24b0107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 -- 172.21.15.22:0/4104036145 >> 172.21.15.22:0/4104036145 conn(0x7f24b00fd0b0 msgr2=0x7f24b00ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 -- 172.21.15.22:0/4104036145 shutdown_connections 2024-03-24T06:50:33.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.138+0000 7f24b5400640 1 -- 172.21.15.22:0/4104036145 wait complete. 2024-03-24T06:50:33.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.139+0000 7f24b5400640 1 Processor -- start 2024-03-24T06:50:33.141 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.139+0000 7f24b5400640 1 -- start start 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 0x7f24b019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 0x7f24b019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f24b019ec50 con 0x7f24b0107660 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f24b019edc0 con 0x7f24b0101f60 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24b5400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f24b019ef80 con 0x7f24b01015a0 2024-03-24T06:50:33.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24afe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24afe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42162/0 (socket says 172.21.15.22:42162) 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.140+0000 7f24afe00640 1 -- 172.21.15.22:0/2954875527 learned_addr learned my addr 172.21.15.22:0/2954875527 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 0x7f24b019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 0x7f24b019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 -- 172.21.15.22:0/2954875527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 msgr2=0x7f24b019a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 0x7f24b019a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 -- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 msgr2=0x7f24b019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 0x7f24b019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 -- 172.21.15.22:0/2954875527 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a8009840 con 0x7f24b01015a0 2024-03-24T06:50:33.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.141+0000 7f24afe00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f24a000e9f0 tx=0x7f24a000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.142+0000 7f24ad600640 1 -- 172.21.15.22:0/2954875527 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0009700 con 0x7f24b01015a0 2024-03-24T06:50:33.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.142+0000 7f24ad600640 1 -- 172.21.15.22:0/2954875527 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a000ccc0 con 0x7f24b01015a0 2024-03-24T06:50:33.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.142+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2498009560 con 0x7f24b01015a0 2024-03-24T06:50:33.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.142+0000 7f24ad600640 1 -- 172.21.15.22:0/2954875527 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a0018820 con 0x7f24b01015a0 2024-03-24T06:50:33.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.142+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b01a6900 con 0x7f24b01015a0 2024-03-24T06:50:33.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.145+0000 7f24ad600640 1 -- 172.21.15.22:0/2954875527 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f24a0010430 con 0x7f24b01015a0 2024-03-24T06:50:33.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 Processor -- start 2024-03-24T06:50:33.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 -- start start 2024-03-24T06:50:33.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36bfa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bfcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556c36b5f6e0 con 0x556c36bfa950 2024-03-24T06:50:33.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556c36b5cc10 con 0x556c36bf1450 2024-03-24T06:50:33.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b9837740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c36b583f0 con 0x556c36bf1d70 2024-03-24T06:50:33.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24afe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36bfa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43004/0 (socket says 172.21.15.22:43004) 2024-03-24T06:50:33.149 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.146+0000 7f24b4a00640 1 -- 172.21.15.22:0/2129563598 learned_addr learned my addr 172.21.15.22:0/2129563598 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bfcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 -- 172.21.15.22:0/2129563598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 msgr2=0x556c36bfa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36bfa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 -- 172.21.15.22:0/2129563598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 msgr2=0x556c36bfcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bfcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 -- 172.21.15.22:0/2129563598 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c36b19e40 con 0x556c36bf1450 2024-03-24T06:50:33.150 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bfcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f24a80115b0 tx=0x7f24a8011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d962fbfc7d927bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.147+0000 7f24ad600640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 0x7f24940a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.148+0000 7f24a5200640 1 -- 172.21.15.22:0/2129563598 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8004d20 con 0x556c36bf1450 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.148+0000 7f24a5200640 1 -- 172.21.15.22:0/2129563598 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a8004e80 con 0x556c36bf1450 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.148+0000 7f24a5200640 1 -- 172.21.15.22:0/2129563598 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a800f8c0 con 0x556c36bf1450 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.148+0000 7f24ad600640 1 -- 172.21.15.22:0/2954875527 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f24a0014070 con 0x7f24b01015a0 2024-03-24T06:50:33.151 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.148+0000 7f24afe00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 0x7f24940a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 -- 172.21.15.22:0/2129563598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 msgr2=0x556c36bf1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24afe00640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 0x7f24940a4770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x556c36bf0f20 tx=0x7f24a00d4530 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f24a80115b0 tx=0x7f24a8011a70 comp rx=0 tx=0).stop 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 -- 172.21.15.22:0/2129563598 shutdown_connections 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bfcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36bfa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 --2- 172.21.15.22:0/2129563598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf1830 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 -- 172.21.15.22:0/2129563598 >> 172.21.15.22:0/2129563598 conn(0x556c36bebb80 msgr2=0x556c36bebf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 -- 172.21.15.22:0/2129563598 shutdown_connections 2024-03-24T06:50:33.153 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.149+0000 7f24b9837740 1 -- 172.21.15.22:0/2129563598 wait complete. 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.150+0000 7f24b9837740 1 -- start start 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 Processor -- start 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 0x556c36bf8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556c36afd060 con 0x556c36bf1d70 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556c36afcd00 con 0x556c36bf1450 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c368c61c0 con 0x556c36bfa950 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.151+0000 7f24b9837740 10 client.0 _ll_register_callbacks cb 0x556c36868be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60862/0 (socket says 172.21.15.22:60862) 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 -- 172.21.15.22:0/670802956 learned_addr learned my addr 172.21.15.22:0/670802956 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24b4a00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 0x556c36bf8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 -- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 msgr2=0x556c36bf8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 0x556c36bf8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 -- 172.21.15.22:0/670802956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 msgr2=0x556c36bf79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24a7e00640 1 -- 172.21.15.22:0/670802956 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a8009e20 con 0x556c36bf1d70 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.152+0000 7f24b4a00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 0x556c36bf8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f24a7e00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f249802c230 tx=0x7f249802ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f3cc5c240ff60a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f249d600640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249802dea0 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f249802c5c0 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c36894720 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f249802cbf0 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f24b9837740 10 client.17505 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.153+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556c36b56a70 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556c36894720 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f249803c040 con 0x556c36bf1d70 2024-03-24T06:50:33.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 10 client.17505 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 10 client.17505 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 20 client.17505 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c13570 0x556c36c15990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 4 client.17505 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c368e3890 con 0x556c36c13570 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.154+0000 7f24b4a00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c13570 0x556c36c15990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.155+0000 7f24b4a00640 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c13570 0x556c36c15990 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6cfd145b02942b58 server_cookie=c28b5a070b5cf311 in_seq=0 out_seq=0 2024-03-24T06:50:33.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.155+0000 7f249d600640 10 client.17505 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:50:33.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f249d600640 1 -- 172.21.15.22:0/670802956 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c368e3890 con 0x556c36c13570 2024-03-24T06:50:33.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f249d600640 10 client.17505 handle_command_reply: tid=0 2024-03-24T06:50:33.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 client.17505 shutdown 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 msgr2=0x556c36bf7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f249802c230 tx=0x7f249802ca00 comp rx=0 tx=0).stop 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 shutdown_connections 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c13570 0x556c36c15990 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bfa950 0x556c36bf8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bf1d70 0x556c36bf7ef0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 --2- 172.21.15.22:0/670802956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 >> 172.21.15.22:0/670802956 conn(0x556c36bebb80 msgr2=0x556c36bed8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.156+0000 7f249d600640 0 client.17505 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:50:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.157+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 shutdown_connections 2024-03-24T06:50:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.157+0000 7f24b9837740 1 -- 172.21.15.22:0/670802956 wait complete. 2024-03-24T06:50:33.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.157+0000 7f24b9837740 20 client.17505 trim_cache size 0 max 16384 2024-03-24T06:50:33.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.171+0000 7f24b9837740 1 Processor -- start 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.171+0000 7f24b9837740 1 -- start start 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bf8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556c36c100f0 con 0x556c36bfa950 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556c36becb80 con 0x556c36bf1450 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b9837740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c36bede90 con 0x556c36bf1d70 2024-03-24T06:50:33.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42194/0 (socket says 172.21.15.22:42194) 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 -- 172.21.15.22:0/256165149 learned_addr learned my addr 172.21.15.22:0/256165149 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b4a00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b4a00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bf8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 -- 172.21.15.22:0/256165149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 msgr2=0x556c36bf2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 -- 172.21.15.22:0/256165149 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 msgr2=0x556c36bf8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bf8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24a7e00640 1 -- 172.21.15.22:0/256165149 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c36af2b00 con 0x556c36bf1d70 2024-03-24T06:50:33.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.172+0000 7f24b4a00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bf8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:33.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.173+0000 7f24a7e00640 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2498006010 tx=0x7f2498009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f166fbcb7e22bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.173+0000 7f249fe00640 1 -- 172.21.15.22:0/256165149 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498004ad0 con 0x556c36bf1d70 2024-03-24T06:50:33.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.173+0000 7f249fe00640 1 -- 172.21.15.22:0/256165149 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f249802ce90 con 0x556c36bf1d70 2024-03-24T06:50:33.179 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.173+0000 7f249fe00640 1 -- 172.21.15.22:0/256165149 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498004250 con 0x556c36bf1d70 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- 172.21.15.22:0/256165149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 msgr2=0x556c36c0ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2498006010 tx=0x7f2498009800 comp rx=0 tx=0).stop 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- 172.21.15.22:0/256165149 shutdown_connections 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36bf8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c0ea40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 --2- 172.21.15.22:0/256165149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36bf2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- 172.21.15.22:0/256165149 >> 172.21.15.22:0/256165149 conn(0x556c36bebb80 msgr2=0x556c36beacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- 172.21.15.22:0/256165149 shutdown_connections 2024-03-24T06:50:33.180 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- 172.21.15.22:0/256165149 wait complete. 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 -- start start 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.174+0000 7f24b9837740 1 Processor -- start 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36c07020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36c788b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x556c36c07c80 con 0x556c36bfa950 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556c36c78df0 con 0x556c36bf1450 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c36c78f60 con 0x556c36bf1d70 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b9837740 10 client.0 _ll_register_callbacks cb 0x7f24a80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36c07020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24a7e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42206/0 (socket says 172.21.15.22:42206) 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24a7e00640 1 -- 172.21.15.22:0/2935200371 learned_addr learned my addr 172.21.15.22:0/2935200371 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36c07020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43034/0 (socket says 172.21.15.22:43034) 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.175+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36c788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 -- 172.21.15.22:0/2935200371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 msgr2=0x556c36c07020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36c07020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 -- 172.21.15.22:0/2935200371 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 msgr2=0x556c36c788b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36c788b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 -- 172.21.15.22:0/2935200371 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249804e040 con 0x556c36bf1d70 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24a7e00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f249802c230 tx=0x7f2498004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fe9ab9f2c5f5187 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f249d600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498003c50 con 0x556c36bf1d70 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f2498000e40 con 0x556c36bf1d70 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.176+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c368577e0 con 0x556c36bf1d70 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2498003db0 con 0x556c36bf1d70 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 10 client.27907 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556c36b186a0 con 0x556c36bf1d70 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556c368577e0 con 0x556c36bf1d70 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f249803c070 con 0x556c36bf1d70 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 10 client.27907 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 10 client.27907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 20 client.27907 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c7a250 0x556c36c7c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 4 client.27907 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:33.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.177+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 --> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c36803700 con 0x556c36c7a250 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c7a250 0x556c36c7c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b4a00640 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c7a250 0x556c36c7c640 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=890e906c121f5b59 server_cookie=7a3311a274cb0610 in_seq=0 out_seq=0 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f249d600640 10 client.27907 ms_handle_connect on v2:172.21.15.155:6834/1202752642 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f249d600640 1 -- 172.21.15.22:0/2935200371 <== mds.4 v2:172.21.15.155:6834/1202752642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c36803700 con 0x556c36c7a250 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f249d600640 10 client.27907 handle_command_reply: tid=0 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 client.27907 shutdown 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 msgr2=0x556c36c07560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f249802c230 tx=0x7f2498004b40 comp rx=0 tx=0).stop 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 shutdown_connections 2024-03-24T06:50:33.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:6834/1202752642,v1:172.21.15.155:6835/1202752642] conn(0x556c36c7a250 0x556c36c7c640 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x556c36bfa950 0x556c36c788b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c36bf1d70 0x556c36c07560 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 --2- 172.21.15.22:0/2935200371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556c36bf1450 0x556c36c07020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 >> 172.21.15.22:0/2935200371 conn(0x556c36bebb80 msgr2=0x556c36beccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.178+0000 7f249d600640 0 client.27907 ms_handle_reset on v2:172.21.15.155:6834/1202752642 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 shutdown_connections 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b9837740 1 -- 172.21.15.22:0/2935200371 wait complete. 2024-03-24T06:50:33.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b9837740 20 client.27907 trim_cache size 0 max 16384 2024-03-24T06:50:33.187 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:33.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 msgr2=0x7f24940a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 0x7f24940a4770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x556c36bf0f20 tx=0x7f24a00d4530 comp rx=0 tx=0).stop 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 msgr2=0x7f24b0199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f24a000e9f0 tx=0x7f24a000edf0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 shutdown_connections 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f24940a22b0 0x7f24940a4770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24b0107660 0x7f24b019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b0101f60 0x7f24b019a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 --2- 172.21.15.22:0/2954875527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b01015a0 0x7f24b0199d90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.179+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 >> 172.21.15.22:0/2954875527 conn(0x7f24b00fd0b0 msgr2=0x7f24b00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.180+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 shutdown_connections 2024-03-24T06:50:33.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.180+0000 7f24b5400640 1 -- 172.21.15.22:0/2954875527 wait complete. 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.196 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/374173136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.198 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:33 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.199 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T06:50:33.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 -- 172.21.15.22:0/1663925375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8106550 msgr2=0x7f6fa8106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 --2- 172.21.15.22:0/1663925375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8106550 0x7f6fa8106930 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009930 tx=0x7f6f9802bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:33.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 -- 172.21.15.22:0/1663925375 shutdown_connections 2024-03-24T06:50:33.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 --2- 172.21.15.22:0/1663925375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa810d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 --2- 172.21.15.22:0/1663925375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8100490 0x7f6fa8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 --2- 172.21.15.22:0/1663925375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8106550 0x7f6fa8106930 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 -- 172.21.15.22:0/1663925375 >> 172.21.15.22:0/1663925375 conn(0x7f6fa80fbf60 msgr2=0x7f6fa80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 -- 172.21.15.22:0/1663925375 shutdown_connections 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.396+0000 7f6fad200640 1 -- 172.21.15.22:0/1663925375 wait complete. 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.397+0000 7f6fad200640 1 Processor -- start 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.397+0000 7f6fad200640 1 -- start start 2024-03-24T06:50:33.401 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8106550 0x7f6fa819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa819ec50 con 0x7f6fa8106550 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa819edc0 con 0x7f6fa8100490 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fad200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa819ef80 con 0x7f6fa8100e50 2024-03-24T06:50:33.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42212/0 (socket says 172.21.15.22:42212) 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43054/0 (socket says 172.21.15.22:43054) 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa6a00640 1 -- 172.21.15.22:0/3951405089 learned_addr learned my addr 172.21.15.22:0/3951405089 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7e00640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8106550 0x7f6fa819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 -- 172.21.15.22:0/3951405089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 msgr2=0x7f6fa819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 -- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8106550 msgr2=0x7f6fa819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8106550 0x7f6fa819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa7400640 1 -- 172.21.15.22:0/3951405089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa0009840 con 0x7f6fa8100490 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.398+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6fa7400640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6f98036040 tx=0x7f6f9802c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6f9fe00640 1 -- 172.21.15.22:0/3951405089 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9802dea0 con 0x7f6fa8100490 2024-03-24T06:50:33.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6f9fe00640 1 -- 172.21.15.22:0/3951405089 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f980042d0 con 0x7f6fa8100490 2024-03-24T06:50:33.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f98009560 con 0x7f6fa8100490 2024-03-24T06:50:33.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6f9fe00640 1 -- 172.21.15.22:0/3951405089 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980039b0 con 0x7f6fa8100490 2024-03-24T06:50:33.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.399+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa81a6900 con 0x7f6fa8100490 2024-03-24T06:50:33.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 Processor -- start 2024-03-24T06:50:33.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 -- start start 2024-03-24T06:50:33.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee07830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee10410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.410 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55951ed756e0 con 0x55951ee07450 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55951ed72c10 con 0x55951ee10950 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.403+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55951ed6e3f0 con 0x55951ee07d70 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee07830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43058/0 (socket says 172.21.15.22:43058) 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 -- 172.21.15.22:0/261174400 learned_addr learned my addr 172.21.15.22:0/261174400 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee07830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60924/0 (socket says 172.21.15.22:60924) 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 -- 172.21.15.22:0/261174400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 msgr2=0x55951ee10410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6f9fe00640 1 -- 172.21.15.22:0/3951405089 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f6f98003b10 con 0x7f6fa8100490 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee10410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 -- 172.21.15.22:0/261174400 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 msgr2=0x55951ee07830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee07830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 -- 172.21.15.22:0/261174400 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55951ed2fe40 con 0x55951ee10950 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.404+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6f940115f0 tx=0x7f6f94011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e66375617dda357 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.412 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6f9cc00640 1 -- 172.21.15.22:0/261174400 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94004030 con 0x55951ee10950 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6f9cc00640 1 -- 172.21.15.22:0/261174400 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f94004190 con 0x55951ee10950 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6f9cc00640 1 -- 172.21.15.22:0/261174400 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94003990 con 0x55951ee10950 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 -- 172.21.15.22:0/261174400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 msgr2=0x55951ee12d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6f940115f0 tx=0x7f6f94011ab0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 -- 172.21.15.22:0/261174400 shutdown_connections 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee12d40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee10410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/261174400 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee07830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.413 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.405+0000 7f6fb13f6740 1 -- 172.21.15.22:0/261174400 >> 172.21.15.22:0/261174400 conn(0x55951ee01b80 msgr2=0x55951ee01f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6fb13f6740 1 -- 172.21.15.22:0/261174400 shutdown_connections 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6f9fe00640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 0x7f6f8c0a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6f9fe00640 1 -- 172.21.15.22:0/3951405089 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f6f980f0800 con 0x7f6fa8100490 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6fb13f6740 1 -- 172.21.15.22:0/261174400 wait complete. 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 0x7f6f8c0a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.406+0000 7f6fb13f6740 1 -- start start 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 Processor -- start 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 0x7f6f8c0a4680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6f94010010 tx=0x7f6f94000bc0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.414 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07d70 0x55951ee0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55951ed13060 con 0x55951ee07d70 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55951ed12d00 con 0x55951ee10950 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55951eadc1c0 con 0x55951ee07450 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.407+0000 7f6fb13f6740 10 client.0 _ll_register_callbacks cb 0x55951ea7ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.415 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42216/0 (socket says 172.21.15.22:42216) 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 -- 172.21.15.22:0/2421207397 learned_addr learned my addr 172.21.15.22:0/2421207397 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7e00640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07d70 0x55951ee0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 -- 172.21.15.22:0/2421207397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 msgr2=0x55951ee0e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 -- 172.21.15.22:0/2421207397 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07d70 msgr2=0x55951ee0df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07d70 0x55951ee0df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 -- 172.21.15.22:0/2421207397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f94009d80 con 0x55951ee07450 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6fa7400640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6f980edd10 tx=0x7f6f980bc970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e409df82783e1f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6f91600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.408+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980bcf50 con 0x55951ee07450 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f6f980c10a0 con 0x55951ee07450 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55951ea6d7e0 con 0x55951ee07450 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6f980c56f0 con 0x55951ee07450 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 10 client.27916 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55951ed6ca70 con 0x55951ee07450 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f9809be90 con 0x55951ee07450 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6f980db020 con 0x55951ee07450 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 10 client.27916 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 10 client.27916 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 20 client.27916 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee22fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 4 client.27916 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:33.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.409+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55951eaaa720 con 0x55951ee20b70 2024-03-24T06:50:33.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.410+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee22fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.410+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee22fc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c55067442e19008a server_cookie=7b29b001ef1fad28 in_seq=0 out_seq=0 2024-03-24T06:50:33.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.410+0000 7f6f91600640 10 client.27916 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:50:33.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6f91600640 1 -- 172.21.15.22:0/2421207397 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55951eaaa720 con 0x55951ee20b70 2024-03-24T06:50:33.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6f91600640 10 client.27916 handle_command_reply: tid=0 2024-03-24T06:50:33.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 client.27916 shutdown 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 msgr2=0x55951ee0da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6f980edd10 tx=0x7f6f980bc970 comp rx=0 tx=0).stop 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 shutdown_connections 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee22fc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6f91600640 0 client.27916 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07d70 0x55951ee0df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2421207397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07450 0x55951ee0da30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.421 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.411+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 >> 172.21.15.22:0/2421207397 conn(0x55951ee01b80 msgr2=0x55951ee038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.412+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 shutdown_connections 2024-03-24T06:50:33.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.412+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2421207397 wait complete. 2024-03-24T06:50:33.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.412+0000 7f6fb13f6740 20 client.27916 trim_cache size 0 max 16384 2024-03-24T06:50:33.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 Processor -- start 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 -- start start 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 0x55951ee1c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 0x55951ee288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55951ee06e90 con 0x55951ee10950 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55951ee07bd0 con 0x55951ee07450 2024-03-24T06:50:33.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.427+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55951ee02db0 con 0x55951ee07d70 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 0x55951ee1c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42220/0 (socket says 172.21.15.22:42220) 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 -- 172.21.15.22:0/3865410622 learned_addr learned my addr 172.21.15.22:0/3865410622 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 0x55951ee1c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43086/0 (socket says 172.21.15.22:43086) 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa7e00640 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 0x55951ee288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 -- 172.21.15.22:0/3865410622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 msgr2=0x55951ee1c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 0x55951ee1c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 -- 172.21.15.22:0/3865410622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 msgr2=0x55951ee288a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 0x55951ee288a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 -- 172.21.15.22:0/3865410622 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55951ed08b00 con 0x55951ee07d70 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa7e00640 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 0x55951ee288a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009230 tx=0x7f6f94011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a82211d40a1fb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.428+0000 7f6f93e00640 1 -- 172.21.15.22:0/3865410622 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94009c10 con 0x55951ee07d70 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6f93e00640 1 -- 172.21.15.22:0/3865410622 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f9402d040 con 0x55951ee07d70 2024-03-24T06:50:33.431 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6f93e00640 1 -- 172.21.15.22:0/3865410622 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94004660 con 0x55951ee07d70 2024-03-24T06:50:33.432 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.432 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.432 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.433 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/374173136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.434 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.435 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.435 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:33 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 -- 172.21.15.22:0/3865410622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 msgr2=0x55951ee0e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009230 tx=0x7f6f94011ab0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 -- 172.21.15.22:0/3865410622 shutdown_connections 2024-03-24T06:50:33.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee10950 0x55951ee288a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0e2f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/3865410622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee07450 0x55951ee1c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 -- 172.21.15.22:0/3865410622 >> 172.21.15.22:0/3865410622 conn(0x55951ee01b80 msgr2=0x55951ee03840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 -- 172.21.15.22:0/3865410622 shutdown_connections 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.429+0000 7f6fb13f6740 1 -- 172.21.15.22:0/3865410622 wait complete. 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 -- start start 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 Processor -- start 2024-03-24T06:50:33.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee0a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.430+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x55951ee85f50 con 0x55951ee07450 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55951ee860c0 con 0x55951ee10950 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fb13f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55951ee86280 con 0x55951ee07d70 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fb13f6740 10 client.0 _ll_register_callbacks cb 0x7f6f980bce10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa7400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee0a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42228/0 (socket says 172.21.15.22:42228) 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 -- 172.21.15.22:0/2449753247 learned_addr learned my addr 172.21.15.22:0/2449753247 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa7e00640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 -- 172.21.15.22:0/2449753247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 msgr2=0x55951ee0b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 -- 172.21.15.22:0/2449753247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 msgr2=0x55951ee0a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee0a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 -- 172.21.15.22:0/2449753247 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55951ed2fe40 con 0x55951ee07d70 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6fa6a00640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f9401c040 tx=0x7f6f9400fe80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f4f36fbb647dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.431+0000 7f6f91600640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94035030 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f6f94035190 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f94004190 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6f94005040 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6fb13f6740 10 client.27922 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55951ed2e6a0 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f94004190 con 0x55951ee07d70 2024-03-24T06:50:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.432+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f6f9403d020 con 0x55951ee07d70 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 10 client.27922 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 10 client.27922 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 20 client.27922 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee89850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 4 client.27922 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 --> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55951eaf9890 con 0x55951ee20b70 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fa7400640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee89850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6fa7400640 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee89850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c3f4a33dd44301c4 server_cookie=88e0b9ba9002775b in_seq=0 out_seq=0 2024-03-24T06:50:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.433+0000 7f6f91600640 10 client.27922 ms_handle_connect on v2:172.21.15.155:6836/1157072254 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6f91600640 1 -- 172.21.15.22:0/2449753247 <== mds.3 v2:172.21.15.155:6836/1157072254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55951eaf9890 con 0x55951ee20b70 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6f91600640 10 client.27922 handle_command_reply: tid=0 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 client.27922 shutdown 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 msgr2=0x55951ee0ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f9401c040 tx=0x7f6f9400fe80 comp rx=0 tx=0).stop 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 shutdown_connections 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:6836/1157072254,v1:172.21.15.155:6837/1157072254] conn(0x55951ee20b70 0x55951ee89850 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55951ee10950 0x55951ee0b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55951ee07d70 0x55951ee0ace0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 --2- 172.21.15.22:0/2449753247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55951ee07450 0x55951ee0a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 >> 172.21.15.22:0/2449753247 conn(0x55951ee01b80 msgr2=0x55951ee00430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6f91600640 0 client.27922 ms_handle_reset on v2:172.21.15.155:6836/1157072254 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 shutdown_connections 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 1 -- 172.21.15.22:0/2449753247 wait complete. 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.434+0000 7f6fb13f6740 20 client.27922 trim_cache size 0 max 16384 2024-03-24T06:50:33.445 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 msgr2=0x7f6f8c0a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 0x7f6f8c0a4680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6f94010010 tx=0x7f6f94000bc0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 msgr2=0x7f6fa8199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6f98036040 tx=0x7f6f9802c630 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 shutdown_connections 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f6f8c0a21c0 0x7f6f8c0a4680 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fa8106550 0x7f6fa819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fa8100e50 0x7f6fa819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 --2- 172.21.15.22:0/3951405089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fa8100490 0x7f6fa8199d90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 >> 172.21.15.22:0/3951405089 conn(0x7f6fa80fbf60 msgr2=0x7f6fa80fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 shutdown_connections 2024-03-24T06:50:33.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.435+0000 7f6fad200640 1 -- 172.21.15.22:0/3951405089 wait complete. 2024-03-24T06:50:33.455 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T06:50:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/374173136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:33 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 -- 172.21.15.22:0/3682033324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c1016a0 msgr2=0x7f088c101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 --2- 172.21.15.22:0/3682033324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c1016a0 0x7f088c101b20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0874009900 tx=0x7f087402bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 -- 172.21.15.22:0/3682033324 shutdown_connections 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 --2- 172.21.15.22:0/3682033324 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c102060 0x7f088c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 --2- 172.21.15.22:0/3682033324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c1016a0 0x7f088c101b20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 --2- 172.21.15.22:0/3682033324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 0x7f088c107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 -- 172.21.15.22:0/3682033324 >> 172.21.15.22:0/3682033324 conn(0x7f088c078b10 msgr2=0x7f088c078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 -- 172.21.15.22:0/3682033324 shutdown_connections 2024-03-24T06:50:33.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.656+0000 7f0893c00640 1 -- 172.21.15.22:0/3682033324 wait complete. 2024-03-24T06:50:33.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.657+0000 7f0893c00640 1 Processor -- start 2024-03-24T06:50:33.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.657+0000 7f0893c00640 1 -- start start 2024-03-24T06:50:33.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c1016a0 0x7f088c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.660 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 0x7f088c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19edf0 con 0x7f088c1016a0 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19ef60 con 0x7f088c102060 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0893c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f088c19f120 con 0x7f088c1076b0 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0883e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0883e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43122/0 (socket says 172.21.15.22:43122) 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0883e00640 1 -- 172.21.15.22:0/148233480 learned_addr learned my addr 172.21.15.22:0/148233480 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0891c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 0x7f088c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.658+0000 7f0891200640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c1016a0 0x7f088c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 -- 172.21.15.22:0/148233480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 msgr2=0x7f088c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 0x7f088c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 -- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c1016a0 msgr2=0x7f088c199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c1016a0 0x7f088c199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 -- 172.21.15.22:0/148233480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0874009560 con 0x7f088c102060 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0883e00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0874000c00 tx=0x7f087402c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f088a000640 1 -- 172.21.15.22:0/148233480 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087402dea0 con 0x7f088c102060 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f088a000640 1 -- 172.21.15.22:0/148233480 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0874004410 con 0x7f088c102060 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f088a000640 1 -- 172.21.15.22:0/148233480 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874003c20 con 0x7f088c102060 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088c19f3a0 con 0x7f088c102060 2024-03-24T06:50:33.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.659+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c1a6c80 con 0x7f088c102060 2024-03-24T06:50:33.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.662+0000 7f088a000640 1 -- 172.21.15.22:0/148233480 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f08740054f0 con 0x7f088c102060 2024-03-24T06:50:33.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.664+0000 7f088a000640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 0x7f08700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.664+0000 7f088a000640 1 -- 172.21.15.22:0/148233480 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f08740f08f0 con 0x7f088c102060 2024-03-24T06:50:33.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.664+0000 7f0891200640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 0x7f08700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.664+0000 7f089697a740 1 Processor -- start 2024-03-24T06:50:33.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 -- start start 2024-03-24T06:50:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ee240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f0891200640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 0x7f08700a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3536e0 con 0x5641ca3e5ba0 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5641ca350c10 con 0x5641ca3e5280 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f089697a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5641ca34c3f0 con 0x5641ca3ee780 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.665+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43138/0 (socket says 172.21.15.22:43138) 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 -- 172.21.15.22:0/3470754286 learned_addr learned my addr 172.21.15.22:0/3470754286 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0883e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891c00640 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 -- 172.21.15.22:0/3470754286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 msgr2=0x5641ca3f0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 -- 172.21.15.22:0/3470754286 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 msgr2=0x5641ca3ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 -- 172.21.15.22:0/3470754286 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641ca30de40 con 0x5641ca3e5280 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.666+0000 7f0891200640 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bf0 tx=0x7f087c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9111467cc612b77b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.667+0000 7f0882000640 1 -- 172.21.15.22:0/3470754286 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c026430 con 0x5641ca3e5280 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.667+0000 7f0882000640 1 -- 172.21.15.22:0/3470754286 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f087c021070 con 0x5641ca3e5280 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.667+0000 7f0882000640 1 -- 172.21.15.22:0/3470754286 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c01d8a0 con 0x5641ca3e5280 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.667+0000 7f089697a740 1 -- 172.21.15.22:0/3470754286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 msgr2=0x5641ca3e5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.667+0000 7f089697a740 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bf0 tx=0x7f087c014000 comp rx=0 tx=0).stop 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 -- 172.21.15.22:0/3470754286 shutdown_connections 2024-03-24T06:50:33.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 --2- 172.21.15.22:0/3470754286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3e5660 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 -- 172.21.15.22:0/3470754286 >> 172.21.15.22:0/3470754286 conn(0x5641ca3df9b0 msgr2=0x5641ca3dfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 -- 172.21.15.22:0/3470754286 shutdown_connections 2024-03-24T06:50:33.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 -- 172.21.15.22:0/3470754286 wait complete. 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 -- start start 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.668+0000 7f089697a740 1 Processor -- start 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 0x5641ca3ec170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5641ca2f1060 con 0x5641ca3ee780 2024-03-24T06:50:33.677 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5641ca2f0d00 con 0x5641ca3e5280 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5641ca0ba1c0 con 0x5641ca3e5ba0 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f089697a740 10 client.0 _ll_register_callbacks cb 0x5641ca05cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43140/0 (socket says 172.21.15.22:43140) 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f0883e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f0891200640 1 -- 172.21.15.22:0/46411122 learned_addr learned my addr 172.21.15.22:0/46411122 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.669+0000 7f0891c00640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 0x5641ca3ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 -- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 msgr2=0x5641ca3ebc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 -- 172.21.15.22:0/46411122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 msgr2=0x5641ca3ec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 0x5641ca3ec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 -- 172.21.15.22:0/46411122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087c017050 con 0x5641ca3e5280 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891c00640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 0x5641ca3ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f0891200640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bc0 tx=0x7f087c01cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e04f0d8ccacc8855 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f087aa00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:33.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c033070 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5641ca04b7e0 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.670+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f087c004b00 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f087c004cc0 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 10 client.27911 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5641ca34aa70 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5641ca04b7e0 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f087c036020 con 0x5641ca3e5280 2024-03-24T06:50:33.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 10 client.27911 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 10 client.27911 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 20 client.27911 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca3feae0 0x5641ca400f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 4 client.27911 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.671+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641ca088720 con 0x5641ca3feae0 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.672+0000 7f0883e00640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca3feae0 0x5641ca400f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.672+0000 7f0883e00640 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca3feae0 0x5641ca400f00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7fccffe6d871b2a server_cookie=b284cd8c8f95146b in_seq=0 out_seq=0 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.672+0000 7f087aa00640 10 client.27911 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f087aa00640 1 -- 172.21.15.22:0/46411122 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5641ca088720 con 0x5641ca3feae0 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f087aa00640 10 client.27911 handle_command_reply: tid=0 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 client.27911 shutdown 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 msgr2=0x5641ca3eb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bc0 tx=0x7f087c01cfd0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 shutdown_connections 2024-03-24T06:50:33.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca3feae0 0x5641ca400f00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3ee780 0x5641ca3ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 --2- 172.21.15.22:0/46411122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3eb6f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 >> 172.21.15.22:0/46411122 conn(0x5641ca3df9b0 msgr2=0x5641ca3e1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f087aa00640 0 client.27911 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 shutdown_connections 2024-03-24T06:50:33.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 1 -- 172.21.15.22:0/46411122 wait complete. 2024-03-24T06:50:33.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.673+0000 7f089697a740 20 client.27911 trim_cache size 0 max 16384 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 Processor -- start 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 -- start start 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5280 0x5641ca3fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3ee780 0x5641ca3de8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3db2e0 con 0x5641ca3e5280 2024-03-24T06:50:33.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3f1140 con 0x5641ca3ee780 2024-03-24T06:50:33.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.689+0000 7f089697a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3fb5f0 con 0x5641ca3e5ba0 2024-03-24T06:50:33.692 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0891200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5280 0x5641ca3fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42272/0 (socket says 172.21.15.22:42272) 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 -- 172.21.15.22:0/4229198056 learned_addr learned my addr 172.21.15.22:0/4229198056 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0891c00640 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3ee780 0x5641ca3de8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 -- 172.21.15.22:0/4229198056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3ee780 msgr2=0x5641ca3de8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3ee780 0x5641ca3de8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.693 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 -- 172.21.15.22:0/4229198056 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5280 msgr2=0x5641ca3fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5280 0x5641ca3fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 -- 172.21.15.22:0/4229198056 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641ca2e6b00 con 0x5641ca3e5ba0 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.690+0000 7f0883e00640 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f08740bcf70 tx=0x7f08740036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8989a4fbb138bbd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.691+0000 7f0881600640 1 -- 172.21.15.22:0/4229198056 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740c5780 con 0x5641ca3e5ba0 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.691+0000 7f0881600640 1 -- 172.21.15.22:0/4229198056 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08740d4030 con 0x5641ca3e5ba0 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.691+0000 7f0881600640 1 -- 172.21.15.22:0/4229198056 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740edeb0 con 0x5641ca3e5ba0 2024-03-24T06:50:33.694 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.691+0000 7f089697a740 1 -- 172.21.15.22:0/4229198056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 msgr2=0x5641ca3ebf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.691+0000 7f089697a740 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f08740bcf70 tx=0x7f08740036f0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 -- 172.21.15.22:0/4229198056 shutdown_connections 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3ee780 0x5641ca3de8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3e5ba0 0x5641ca3ebf90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 --2- 172.21.15.22:0/4229198056 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5280 0x5641ca3fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 -- 172.21.15.22:0/4229198056 >> 172.21.15.22:0/4229198056 conn(0x5641ca3df9b0 msgr2=0x5641ca3de1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.695 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 -- 172.21.15.22:0/4229198056 shutdown_connections 2024-03-24T06:50:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 -- 172.21.15.22:0/4229198056 wait complete. 2024-03-24T06:50:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 -- start start 2024-03-24T06:50:33.696 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.692+0000 7f089697a740 1 Processor -- start 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3f24a0 con 0x5641ca3e5ba0 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3f2610 con 0x5641ca3e5280 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5641ca3f27d0 con 0x5641ca3ee780 2024-03-24T06:50:33.700 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f089697a740 10 client.0 _ll_register_callbacks cb 0x7f087c003ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43164/0 (socket says 172.21.15.22:43164) 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 -- 172.21.15.22:0/3571747017 learned_addr learned my addr 172.21.15.22:0/3571747017 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0883e00640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.701 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 -- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 msgr2=0x5641ca3f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 -- 172.21.15.22:0/3571747017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 msgr2=0x5641ca3f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.693+0000 7f0891200640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f0891200640 1 -- 172.21.15.22:0/3571747017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c1a6c80 con 0x5641ca3e5280 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f0891c00640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f0891200640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bf0 tx=0x7f087c0097d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9da3c2185dad1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.702 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f087aa00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c033050 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f087c0056e0 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f087c021070 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f087c0044d0 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f089697a740 10 client.27917 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.694+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5641ca30c6a0 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f087c021070 con 0x5641ca3e5280 2024-03-24T06:50:33.703 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f087c048070 con 0x5641ca3e5280 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 10 client.27917 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 10 client.27917 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 20 client.27917 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca465050 0x5641ca467470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 4 client.27917 mds_command: new command op to 24565 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 --> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641ca0d7890 con 0x5641ca465050 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.695+0000 7f0883e00640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca465050 0x5641ca467470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.704 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f0883e00640 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca465050 0x5641ca467470 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a2186d695b21024 server_cookie=b5de9a937b74f0e1 in_seq=0 out_seq=0 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f087aa00640 10 client.27917 ms_handle_connect on v2:172.21.15.155:6838/2828760693 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f087aa00640 1 -- 172.21.15.22:0/3571747017 <== mds.2 v2:172.21.15.155:6838/2828760693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641ca0d7890 con 0x5641ca465050 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f087aa00640 10 client.27917 handle_command_reply: tid=0 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 client.27917 shutdown 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 msgr2=0x5641ca3f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f087c003bf0 tx=0x7f087c0097d0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 shutdown_connections 2024-03-24T06:50:33.705 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:6838/2828760693,v1:172.21.15.155:6839/2828760693] conn(0x5641ca465050 0x5641ca467470 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5641ca3ee780 0x5641ca3f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5641ca3e5ba0 0x5641ca3f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f087aa00640 0 client.27917 ms_handle_reset on v2:172.21.15.155:6838/2828760693 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 --2- 172.21.15.22:0/3571747017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5641ca3e5280 0x5641ca3f66b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 >> 172.21.15.22:0/3571747017 conn(0x5641ca3df9b0 msgr2=0x5641ca3dea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 shutdown_connections 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.696+0000 7f089697a740 1 -- 172.21.15.22:0/3571747017 wait complete. 2024-03-24T06:50:33.706 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f089697a740 20 client.27917 trim_cache size 0 max 16384 2024-03-24T06:50:33.707 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 msgr2=0x7f08700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 0x7f08700a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c009500 comp rx=0 tx=0).stop 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 msgr2=0x7f088c19a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0874000c00 tx=0x7f087402c6b0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 shutdown_connections 2024-03-24T06:50:33.708 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f08700a2230 0x7f08700a46f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f088c1076b0 0x7f088c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c102060 0x7f088c19a470 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 --2- 172.21.15.22:0/148233480 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f088c1016a0 0x7f088c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.697+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 >> 172.21.15.22:0/148233480 conn(0x7f088c078b10 msgr2=0x7f088c10ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.698+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 shutdown_connections 2024-03-24T06:50:33.709 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.698+0000 7f0893c00640 1 -- 172.21.15.22:0/148233480 wait complete. 2024-03-24T06:50:33.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T06:50:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 -- 172.21.15.22:0/26792436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c81038b0 msgr2=0x7f07c8103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 --2- 172.21.15.22:0/26792436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c81038b0 0x7f07c8103d30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009930 tx=0x7f07b402bc30 comp rx=0 tx=0).stop 2024-03-24T06:50:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 -- 172.21.15.22:0/26792436 shutdown_connections 2024-03-24T06:50:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 --2- 172.21.15.22:0/26792436 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c8104270 0x7f07c8110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 --2- 172.21.15.22:0/26792436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c81038b0 0x7f07c8103d30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 --2- 172.21.15.22:0/26792436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81098c0 0x7f07c8109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 -- 172.21.15.22:0/26792436 >> 172.21.15.22:0/26792436 conn(0x7f07c80ff3c0 msgr2=0x7f07c81017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 -- 172.21.15.22:0/26792436 shutdown_connections 2024-03-24T06:50:33.924 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.922+0000 7f07cd400640 1 -- 172.21.15.22:0/26792436 wait complete. 2024-03-24T06:50:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 Processor -- start 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 -- start start 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81038b0 0x7f07c8075c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 0x7f07c80766a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f07c81aa5e0 con 0x7f07c81098c0 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f07c81aa720 con 0x7f07c81038b0 2024-03-24T06:50:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.923+0000 7f07cd400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f07c81aa860 con 0x7f07c8104270 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42298/0 (socket says 172.21.15.22:42298) 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 -- 172.21.15.22:0/1032004388 learned_addr learned my addr 172.21.15.22:0/1032004388 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c6800640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81038b0 0x7f07c8075c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c7200640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 0x7f07c80766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 -- 172.21.15.22:0/1032004388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81038b0 msgr2=0x7f07c8075c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81038b0 0x7f07c8075c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 -- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 msgr2=0x7f07c80766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 0x7f07c80766a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 -- 172.21.15.22:0/1032004388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b4009560 con 0x7f07c8104270 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c7200640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 0x7f07c80766a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.924+0000 7f07c5e00640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f07b4000c00 tx=0x7f07b402c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.925+0000 7f07bf400640 1 -- 172.21.15.22:0/1032004388 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b40042d0 con 0x7f07c8104270 2024-03-24T06:50:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.925+0000 7f07bf400640 1 -- 172.21.15.22:0/1032004388 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b4004430 con 0x7f07c8104270 2024-03-24T06:50:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.925+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c81aaab0 con 0x7f07c8104270 2024-03-24T06:50:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.925+0000 7f07bf400640 1 -- 172.21.15.22:0/1032004388 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4003c50 con 0x7f07c8104270 2024-03-24T06:50:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.925+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c81ab010 con 0x7f07c8104270 2024-03-24T06:50:33.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.929+0000 7f07bf400640 1 -- 172.21.15.22:0/1032004388 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f07b4004df0 con 0x7f07c8104270 2024-03-24T06:50:33.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 Processor -- start 2024-03-24T06:50:33.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 -- start start 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391efb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601391526e0 con 0x5601391e4280 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56013914fc10 con 0x5601391e4ba0 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.930+0000 7f07d002f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56013914b3f0 con 0x5601391ed780 2024-03-24T06:50:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43190/0 (socket says 172.21.15.22:43190) 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c7200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42310/0 (socket says 172.21.15.22:42310) 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 -- 172.21.15.22:0/4094567652 learned_addr learned my addr 172.21.15.22:0/4094567652 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07bf400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 0x7f07ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 -- 172.21.15.22:0/4094567652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 msgr2=0x5601391efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391efb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 -- 172.21.15.22:0/4094567652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 msgr2=0x5601391e4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 -- 172.21.15.22:0/4094567652 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56013910ce40 con 0x5601391e4ba0 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07bf400640 1 -- 172.21.15.22:0/1032004388 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f07b40f09a0 con 0x7f07c8104270 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.931+0000 7f07c6800640 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009c70 tx=0x7f07b8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdc0d48b06948f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.935 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.932+0000 7f07c7200640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 0x7f07ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.932+0000 7f07b3400640 1 -- 172.21.15.22:0/4094567652 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004230 con 0x5601391e4ba0 2024-03-24T06:50:33.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.932+0000 7f07b3400640 1 -- 172.21.15.22:0/4094567652 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b8004390 con 0x5601391e4ba0 2024-03-24T06:50:33.936 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.932+0000 7f07b3400640 1 -- 172.21.15.22:0/4094567652 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b80197e0 con 0x5601391e4ba0 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 -- 172.21.15.22:0/4094567652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 msgr2=0x5601391ed240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07c7200640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 0x7f07ac0a4770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07c0005fd0 tx=0x7f07c0004a40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009c70 tx=0x7f07b8011810 comp rx=0 tx=0).stop 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 -- 172.21.15.22:0/4094567652 shutdown_connections 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391ed240 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 --2- 172.21.15.22:0/4094567652 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 -- 172.21.15.22:0/4094567652 >> 172.21.15.22:0/4094567652 conn(0x5601391de9b0 msgr2=0x5601391ded90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 -- 172.21.15.22:0/4094567652 shutdown_connections 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.933+0000 7f07d002f740 1 -- 172.21.15.22:0/4094567652 wait complete. 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 -- start start 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 Processor -- start 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391ea740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391eac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391ed780 0x5601391eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601390f0060 con 0x5601391ed780 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601390efd00 con 0x5601391e4ba0 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560138eb91c0 con 0x5601391e4280 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.934+0000 7f07d002f740 10 client.0 _ll_register_callbacks cb 0x560138e5bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c6800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391ea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.943 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391eac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c6800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391ea740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42322/0 (socket says 172.21.15.22:42322) 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c6800640 1 -- 172.21.15.22:0/3542453455 learned_addr learned my addr 172.21.15.22:0/3542453455 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c7200640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391ed780 0x5601391eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 -- 172.21.15.22:0/3542453455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 msgr2=0x5601391ea740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391ea740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 -- 172.21.15.22:0/3542453455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391ed780 msgr2=0x5601391eb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391ed780 0x5601391eb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.944 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 -- 172.21.15.22:0/3542453455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b80098d0 con 0x5601391e4ba0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07c5e00640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391eac80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f07b40eddf0 tx=0x7f07b4005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a2d441b9af21bba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.935+0000 7f07a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b40ee160 con 0x5601391e4ba0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f07b40bd790 con 0x5601391e4ba0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560138e4a7e0 con 0x5601391e4ba0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07b40d3030 con 0x5601391e4ba0 2024-03-24T06:50:33.945 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07d002f740 10 client.27929 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560139149a70 con 0x5601391e4ba0 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560138e4a7e0 con 0x5601391e4ba0 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f07b403c070 con 0x5601391e4ba0 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07d002f740 10 client.27929 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.936+0000 7f07d002f740 10 client.27929 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07d002f740 20 client.27929 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.946 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5601391fd9c0 0x5601391ffdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07d002f740 4 client.27929 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560138e87720 con 0x5601391fd9c0 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07c6800640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5601391fd9c0 0x5601391ffdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07c6800640 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5601391fd9c0 0x5601391ffdb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e773ce832a4b0bd server_cookie=aa0c40ad6ca63645 in_seq=0 out_seq=0 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.937+0000 7f07a7e00640 10 client.27929 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.938+0000 7f07a7e00640 1 -- 172.21.15.22:0/3542453455 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560138e87720 con 0x5601391fd9c0 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.938+0000 7f07a7e00640 10 client.27929 handle_command_reply: tid=0 2024-03-24T06:50:33.947 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 client.27929 shutdown 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 msgr2=0x5601391eac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391eac80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f07b40eddf0 tx=0x7f07b4005bd0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 shutdown_connections 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391ed780 0x5601391eb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x5601391fd9c0 0x5601391ffdb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391eac80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 --2- 172.21.15.22:0/3542453455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391ea740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 >> 172.21.15.22:0/3542453455 conn(0x5601391de9b0 msgr2=0x5601391e0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07a7e00640 0 client.27929 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:33.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 shutdown_connections 2024-03-24T06:50:33.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 1 -- 172.21.15.22:0/3542453455 wait complete. 2024-03-24T06:50:33.949 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.939+0000 7f07d002f740 20 client.27929 trim_cache size 0 max 16384 2024-03-24T06:50:33.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 Processor -- start 2024-03-24T06:50:33.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 -- start start 2024-03-24T06:50:33.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4ba0 0x5601391eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391ed780 0x5601391dd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601391df9b0 con 0x5601391e4ba0 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5601391de7e0 con 0x5601391ed780 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.955+0000 7f07d002f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560139200410 con 0x5601391e4280 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42328/0 (socket says 172.21.15.22:42328) 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 -- 172.21.15.22:0/4111411338 learned_addr learned my addr 172.21.15.22:0/4111411338 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c7200640 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391ed780 0x5601391dd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c5e00640 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4ba0 0x5601391eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 -- 172.21.15.22:0/4111411338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391ed780 msgr2=0x5601391dd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.960 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391ed780 0x5601391dd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 -- 172.21.15.22:0/4111411338 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4ba0 msgr2=0x5601391eafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4ba0 0x5601391eafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 -- 172.21.15.22:0/4111411338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601390e5b00 con 0x5601391e4280 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.956+0000 7f07c6800640 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f07b8005f40 tx=0x7f07b8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65bed965126b36ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.957+0000 7f07b2a00640 1 -- 172.21.15.22:0/4111411338 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004200 con 0x5601391e4280 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.957+0000 7f07b2a00640 1 -- 172.21.15.22:0/4111411338 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07b8009310 con 0x5601391e4280 2024-03-24T06:50:33.961 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.957+0000 7f07b2a00640 1 -- 172.21.15.22:0/4111411338 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b80035f0 con 0x5601391e4280 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- 172.21.15.22:0/4111411338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 msgr2=0x5601391f9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f07b8005f40 tx=0x7f07b8005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- 172.21.15.22:0/4111411338 shutdown_connections 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391ed780 0x5601391dd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4ba0 0x5601391eafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 --2- 172.21.15.22:0/4111411338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391e4280 0x5601391f9070 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- 172.21.15.22:0/4111411338 >> 172.21.15.22:0/4111411338 conn(0x5601391de9b0 msgr2=0x5601391dd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- 172.21.15.22:0/4111411338 shutdown_connections 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- 172.21.15.22:0/4111411338 wait complete. 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.958+0000 7f07d002f740 1 -- start start 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 Processor -- start 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391e77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.963 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5601391f5f10 con 0x5601391e4280 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560139262cb0 con 0x5601391e4ba0 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560139262e40 con 0x5601391ed780 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.959+0000 7f07d002f740 10 client.0 _ll_register_callbacks cb 0x7f07b40035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c6800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391e77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c6800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:32818/0 (socket says 172.21.15.22:32818) 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c6800640 1 -- 172.21.15.22:0/2749340352 learned_addr learned my addr 172.21.15.22:0/2749340352 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:33.964 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c7200640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 -- 172.21.15.22:0/2749340352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 msgr2=0x5601391e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391e7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 -- 172.21.15.22:0/2749340352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 msgr2=0x5601391e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:33.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.965 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.960+0000 7f07c5e00640 1 -- 172.21.15.22:0/2749340352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56013910ce40 con 0x5601391e4ba0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07c5e00640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391e77f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f07b40036a0 tx=0x7f07b40ee100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa473cf7ad06b4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b40d3070 con 0x5601391e4ba0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f07b40d4070 con 0x5601391e4ba0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560138ed6890 con 0x5601391e4ba0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f07b40bce60 con 0x5601391e4ba0 2024-03-24T06:50:33.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07d002f740 10 client.27935 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56013910b6a0 con 0x5601391e4ba0 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.961+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560138ed6890 con 0x5601391e4ba0 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7f07b403c040 con 0x5601391e4ba0 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 10 client.27935 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 10 client.27935 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 20 client.27935 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:33.969 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560139264130 0x560139266520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 4 client.27935 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 --> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560138df6700 con 0x560139264130 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07c6800640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560139264130 0x560139266520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07c6800640 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560139264130 0x560139266520 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c2f02e3ec11ab203 server_cookie=af39cc1bba29d360 in_seq=0 out_seq=0 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.962+0000 7f07a7e00640 10 client.27935 ms_handle_connect on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07a7e00640 1 -- 172.21.15.22:0/2749340352 <== mds.1 v2:172.21.15.143:6838/2325215622 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560138df6700 con 0x560139264130 2024-03-24T06:50:33.970 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07a7e00640 10 client.27935 handle_command_reply: tid=0 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 client.27935 shutdown 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 msgr2=0x5601391e77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391e77f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f07b40036a0 tx=0x7f07b40ee100 comp rx=0 tx=0).stop 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 shutdown_connections 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5601391ed780 0x5601391e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:6838/2325215622,v1:172.21.15.143:6839/2325215622] conn(0x560139264130 0x560139266520 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5601391e4ba0 0x5601391e77f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 --2- 172.21.15.22:0/2749340352 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5601391e4280 0x5601391e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07a7e00640 0 client.27935 ms_handle_reset on v2:172.21.15.143:6838/2325215622 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 >> 172.21.15.22:0/2749340352 conn(0x5601391de9b0 msgr2=0x5601391e05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 shutdown_connections 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 1 -- 172.21.15.22:0/2749340352 wait complete. 2024-03-24T06:50:33.972 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.963+0000 7f07d002f740 20 client.27935 trim_cache size 0 max 16384 2024-03-24T06:50:33.973 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:33.973 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 msgr2=0x7f07ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 0x7f07ac0a4770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f07c0005fd0 tx=0x7f07c0004a40 comp rx=0 tx=0).stop 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 msgr2=0x7f07c8076160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f07b4000c00 tx=0x7f07b402c5f0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 shutdown_connections 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f07ac0a22b0 0x7f07ac0a4770 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07c81098c0 0x7f07c80766a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f07c8104270 0x7f07c8076160 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 --2- 172.21.15.22:0/1032004388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c81038b0 0x7f07c8075c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:33.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.964+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 >> 172.21.15.22:0/1032004388 conn(0x7f07c80ff3c0 msgr2=0x7f07c80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:33.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.965+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 shutdown_connections 2024-03-24T06:50:33.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:33.965+0000 7f07cd400640 1 -- 172.21.15.22:0/1032004388 wait complete. 2024-03-24T06:50:33.983 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T06:50:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 -- 172.21.15.22:0/2630431042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c107650 msgr2=0x7efc0c107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 --2- 172.21.15.22:0/2630431042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c107650 0x7efc0c107a30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009930 tx=0x7efbfc02bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 -- 172.21.15.22:0/2630431042 shutdown_connections 2024-03-24T06:50:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 --2- 172.21.15.22:0/2630431042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c101f50 0x7efc0c10e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 --2- 172.21.15.22:0/2630431042 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101590 0x7efc0c101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 --2- 172.21.15.22:0/2630431042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c107650 0x7efc0c107a30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.187+0000 7efc11c00640 1 -- 172.21.15.22:0/2630431042 >> 172.21.15.22:0/2630431042 conn(0x7efc0c0fd080 msgr2=0x7efc0c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.188+0000 7efc11c00640 1 -- 172.21.15.22:0/2630431042 shutdown_connections 2024-03-24T06:50:34.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.188+0000 7efc11c00640 1 -- 172.21.15.22:0/2630431042 wait complete. 2024-03-24T06:50:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 Processor -- start 2024-03-24T06:50:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 -- start start 2024-03-24T06:50:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 0x7efc0c073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101f50 0x7efc0c073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 0x7efc0c1a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1a6520 con 0x7efc0c101f50 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1a6690 con 0x7efc0c101590 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.189+0000 7efc11c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efc0c1a6850 con 0x7efc0c074490 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc11200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 0x7efc0c1a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc11200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 0x7efc0c1a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42350/0 (socket says 172.21.15.22:42350) 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc11200640 1 -- 172.21.15.22:0/3057485153 learned_addr learned my addr 172.21.15.22:0/3057485153 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 0x7efc0c073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0b400640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101f50 0x7efc0c073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 -- 172.21.15.22:0/3057485153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 msgr2=0x7efc0c1a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 0x7efc0c1a5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 -- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101f50 msgr2=0x7efc0c073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101f50 0x7efc0c073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 -- 172.21.15.22:0/3057485153 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc009560 con 0x7efc0c101590 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.190+0000 7efc0be00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 0x7efc0c073a10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efbfc008000 tx=0x7efbfc004bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.191+0000 7efc08c00640 1 -- 172.21.15.22:0/3057485153 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc02dea0 con 0x7efc0c101590 2024-03-24T06:50:34.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.191+0000 7efc08c00640 1 -- 172.21.15.22:0/3057485153 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbfc004350 con 0x7efc0c101590 2024-03-24T06:50:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.191+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0c1a6ad0 con 0x7efc0c101590 2024-03-24T06:50:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.191+0000 7efc08c00640 1 -- 172.21.15.22:0/3057485153 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0039e0 con 0x7efc0c101590 2024-03-24T06:50:34.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.191+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0c1a6fb0 con 0x7efc0c101590 2024-03-24T06:50:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.194+0000 7efc15e18740 1 Processor -- start 2024-03-24T06:50:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.194+0000 7efc08c00640 1 -- 172.21.15.22:0/3057485153 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7efbfc005360 con 0x7efc0c101590 2024-03-24T06:50:34.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.194+0000 7efc15e18740 1 -- start start 2024-03-24T06:50:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adaa1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaaa950 0x5626adaacd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5626ada0f6e0 con 0x5626adaaa950 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626ada0cc10 con 0x5626adaa1d70 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc15e18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5626ada083f0 con 0x5626adaa1450 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc11200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adaa1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.195+0000 7efc0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43258/0 (socket says 172.21.15.22:43258) 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 -- 172.21.15.22:0/2819546115 learned_addr learned my addr 172.21.15.22:0/2819546115 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0b400640 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaaa950 0x5626adaacd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 -- 172.21.15.22:0/2819546115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 msgr2=0x5626adaa1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adaa1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 -- 172.21.15.22:0/2819546115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaaa950 msgr2=0x5626adaacd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaaa950 0x5626adaacd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.196+0000 7efc0be00640 1 -- 172.21.15.22:0/2819546115 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626ad9c9e40 con 0x5626adaa1d70 2024-03-24T06:50:34.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc08c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 0x7efbf00a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc0be00640 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0edb70 tx=0x7efbfc0f6a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9056defc2d768f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc11200640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 0x7efbf00a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc01600640 1 -- 172.21.15.22:0/2819546115 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0b9170 con 0x5626adaa1d70 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc08c00640 1 -- 172.21.15.22:0/3057485153 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7efbfc0f07e0 con 0x7efc0c101590 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc01600640 1 -- 172.21.15.22:0/2819546115 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbfc0bcca0 con 0x5626adaa1d70 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.197+0000 7efc01600640 1 -- 172.21.15.22:0/2819546115 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc03f400 con 0x5626adaa1d70 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.198+0000 7efc15e18740 1 -- 172.21.15.22:0/2819546115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 msgr2=0x5626adaaa410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.198+0000 7efc15e18740 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0edb70 tx=0x7efbfc0f6a60 comp rx=0 tx=0).stop 2024-03-24T06:50:34.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.198+0000 7efc11200640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 0x7efbf00a47a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc040049d0 tx=0x7efc04009500 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 -- 172.21.15.22:0/2819546115 shutdown_connections 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaaa950 0x5626adaacd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1d70 0x5626adaaa410 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 --2- 172.21.15.22:0/2819546115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adaa1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 -- 172.21.15.22:0/2819546115 >> 172.21.15.22:0/2819546115 conn(0x5626ada9bb80 msgr2=0x5626ada9bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 -- 172.21.15.22:0/2819546115 shutdown_connections 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.199+0000 7efc15e18740 1 -- 172.21.15.22:0/2819546115 wait complete. 2024-03-24T06:50:34.201 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.200+0000 7efc15e18740 1 -- start start 2024-03-24T06:50:34.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.200+0000 7efc15e18740 1 Processor -- start 2024-03-24T06:50:34.204 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.200+0000 7efc15e18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 0x5626adaa7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adaa7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5626ad9ad060 con 0x5626adaa1450 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626ad9acd00 con 0x5626adaa84d0 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5626ad7761c0 con 0x5626adaaa950 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc15e18740 10 client.0 _ll_register_callbacks cb 0x5626ad718be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:34.205 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc0be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 0x5626adaa7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43260/0 (socket says 172.21.15.22:43260) 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 -- 172.21.15.22:0/2961577430 learned_addr learned my addr 172.21.15.22:0/2961577430 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc0b400640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adaa7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 -- 172.21.15.22:0/2961577430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 msgr2=0x5626adaa7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adaa7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 -- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 msgr2=0x5626adaa7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.206 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 0x5626adaa7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc11200640 1 -- 172.21.15.22:0/2961577430 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0c1a6fb0 con 0x5626adaa84d0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.201+0000 7efc0be00640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 0x5626adaa7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efc11200640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efc0400fb00 tx=0x7efc04004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e8f9a641185d89e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efbf6000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04023070 con 0x5626adaa84d0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7efc0401e610 con 0x5626adaa84d0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5626ad7077e0 con 0x5626adaa84d0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efc04017070 con 0x5626adaa84d0 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efc15e18740 10 client.27947 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:34.207 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.202+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5626ada06a70 con 0x5626adaa84d0 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5626ad7077e0 con 0x5626adaa84d0 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7efc04031070 con 0x5626adaa84d0 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 10 client.27947 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 10 client.27947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 20 client.27947 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adabb1c0 0x5626adabd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 4 client.27947 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T06:50:34.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626ad744720 con 0x5626adabb1c0 2024-03-24T06:50:34.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.203+0000 7efc0be00640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adabb1c0 0x5626adabd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.204+0000 7efc0be00640 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adabb1c0 0x5626adabd5e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62230f98cf1f387d server_cookie=a696cc08d2020c06 in_seq=0 out_seq=0 2024-03-24T06:50:34.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.204+0000 7efbf6000640 10 client.27947 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.205+0000 7efbf6000640 1 -- 172.21.15.22:0/2961577430 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626ad744720 con 0x5626adabb1c0 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.205+0000 7efbf6000640 10 client.27947 handle_command_reply: tid=0 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.205+0000 7efc15e18740 1 client.27947 shutdown 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 msgr2=0x5626adab6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efc0400fb00 tx=0x7efc04004ab0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 shutdown_connections 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adabb1c0 0x5626adabd5e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa84d0 0x5626adab6670 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adaa7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 --2- 172.21.15.22:0/2961577430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa1450 0x5626adaa7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efbf6000640 0 client.27947 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.206+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 >> 172.21.15.22:0/2961577430 conn(0x5626ada9bb80 msgr2=0x5626ada9d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.207+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 shutdown_connections 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.207+0000 7efc15e18740 1 -- 172.21.15.22:0/2961577430 wait complete. 2024-03-24T06:50:34.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.207+0000 7efc15e18740 20 client.27947 trim_cache size 0 max 16384 2024-03-24T06:50:34.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.226+0000 7efc15e18740 1 Processor -- start 2024-03-24T06:50:34.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 -- start start 2024-03-24T06:50:34.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adab5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.227 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5626ad744720 con 0x5626adaa84d0 2024-03-24T06:50:34.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626ad7077e0 con 0x5626adaa1450 2024-03-24T06:50:34.228 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc15e18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5626adab9920 con 0x5626adaaa950 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc11200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc11200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42384/0 (socket says 172.21.15.22:42384) 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc0be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:43270/0 (socket says 172.21.15.22:43270) 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc11200640 1 -- 172.21.15.22:0/3483677947 learned_addr learned my addr 172.21.15.22:0/3483677947 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.227+0000 7efc0b400640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adab5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.230 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 -- 172.21.15.22:0/3483677947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 msgr2=0x5626adab64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 -- 172.21.15.22:0/3483677947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 msgr2=0x5626adab5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adab5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 -- 172.21.15.22:0/3483677947 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626ada971f0 con 0x5626adaa1450 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0b400640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adab5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc11200640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc0be00640 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0edb70 tx=0x7efbfc0bc830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dcd1be2b28fa940 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.231 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc00c00640 1 -- 172.21.15.22:0/3483677947 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0b9170 con 0x5626adaa1450 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc00c00640 1 -- 172.21.15.22:0/3483677947 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbfc0bca10 con 0x5626adaa1450 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.228+0000 7efc00c00640 1 -- 172.21.15.22:0/3483677947 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0c10b0 con 0x5626adaa1450 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 -- 172.21.15.22:0/3483677947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 msgr2=0x5626adac2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efbfc0edb70 tx=0x7efbfc0bc830 comp rx=0 tx=0).stop 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 -- 172.21.15.22:0/3483677947 shutdown_connections 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaaa950 0x5626adab64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adab5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 --2- 172.21.15.22:0/3483677947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaa1450 0x5626adac2ed0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.232 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.229+0000 7efc15e18740 1 -- 172.21.15.22:0/3483677947 >> 172.21.15.22:0/3483677947 conn(0x5626ada9bb80 msgr2=0x5626ada9d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.230+0000 7efc15e18740 1 -- 172.21.15.22:0/3483677947 shutdown_connections 2024-03-24T06:50:34.233 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.230+0000 7efc15e18740 1 -- 172.21.15.22:0/3483677947 wait complete. 2024-03-24T06:50:34.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.230+0000 7efc15e18740 1 -- start start 2024-03-24T06:50:34.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 Processor -- start 2024-03-24T06:50:34.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adab25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.236 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adaaf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 0x5626adaafae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x5626adab2cb0 con 0x5626adaa84d0 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626adab2e20 con 0x5626adaaa950 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5626adab00a0 con 0x5626adaa1450 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc15e18740 10 client.0 _ll_register_callbacks cb 0x7efc04009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.231+0000 7efc0be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adab25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc0be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adab25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:42398/0 (socket says 172.21.15.22:42398) 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc0be00640 1 -- 172.21.15.22:0/2139855051 learned_addr learned my addr 172.21.15.22:0/2139855051 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:34.237 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc0b400640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adaaf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 0x5626adaafae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 -- 172.21.15.22:0/2139855051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 msgr2=0x5626adab25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adab25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 -- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 msgr2=0x5626adaaf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adaaf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 -- 172.21.15.22:0/2139855051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626ada06a70 con 0x5626adaaa950 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efc11200640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 0x5626adaafae0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efc04012f70 tx=0x7efc0400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcac41052f147168 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.232+0000 7efbf6000640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04017070 con 0x5626adaaa950 2024-03-24T06:50:34.238 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mon.2 v2:172.21.15.143:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7efc04004b20 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbfc0bcca0 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efc04023040 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efc15e18740 10 client.27953 fetch_fsmap learned FSMap version 37 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5626ad9c9e40 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mon.2 v2:172.21.15.143:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc04042020 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.233+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mon.2 v2:172.21.15.143:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7efc04031040 con 0x5626adaaa950 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 10 client.27953 fetch_fsmap finished waiting for FSMap version 37 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 10 client.27953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-03-24T06:50:34.239 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 20 client.27953 populate_metadata read hostname 'smithi022' 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adb22910 0x5626adb24d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 4 client.27953 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 --> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626ad793890 con 0x5626adb22910 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.234+0000 7efc0be00640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adb22910 0x5626adb24d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc0be00640 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adb22910 0x5626adb24d30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6615ee7a3b610e39 server_cookie=d1bb0228d22d0279 in_seq=0 out_seq=0 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efbf6000640 10 client.27953 ms_handle_connect on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efbf6000640 1 -- 172.21.15.22:0/2139855051 <== mds.0 v2:172.21.15.22:6840/857116957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626ad793890 con 0x5626adb22910 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efbf6000640 10 client.27953 handle_command_reply: tid=0 2024-03-24T06:50:34.240 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 client.27953 shutdown 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 msgr2=0x5626adaafae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 0x5626adaafae0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7efc04012f70 tx=0x7efc0400fa00 comp rx=0 tx=0).stop 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 shutdown_connections 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:6840/857116957,v1:172.21.15.22:6841/857116957] conn(0x5626adb22910 0x5626adb24d30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626adaaa950 0x5626adaafae0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5626adaa84d0 0x5626adaaf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 --2- 172.21.15.22:0/2139855051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5626adaa1450 0x5626adab25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efbf6000640 0 client.27953 ms_handle_reset on v2:172.21.15.22:6840/857116957 2024-03-24T06:50:34.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.235+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 >> 172.21.15.22:0/2139855051 conn(0x5626ada9bb80 msgr2=0x5626ada9cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.236+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 shutdown_connections 2024-03-24T06:50:34.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.236+0000 7efc15e18740 1 -- 172.21.15.22:0/2139855051 wait complete. 2024-03-24T06:50:34.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.236+0000 7efc15e18740 20 client.27953 trim_cache size 0 max 16384 2024-03-24T06:50:34.242 INFO:teuthology.orchestra.run.smithi022.stdout:[] 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 msgr2=0x7efbf00a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 0x7efbf00a47a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc040049d0 tx=0x7efc04009500 comp rx=0 tx=0).stop 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 msgr2=0x7efc0c073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 0x7efc0c073a10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7efbfc008000 tx=0x7efbfc004bf0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 shutdown_connections 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efbf00a22e0 0x7efbf00a47a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc0c074490 0x7efc0c1a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efc0c101f50 0x7efc0c073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 --2- 172.21.15.22:0/3057485153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc0c101590 0x7efc0c073a10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:34.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 >> 172.21.15.22:0/3057485153 conn(0x7efc0c0fd080 msgr2=0x7efc0c0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:34.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 shutdown_connections 2024-03-24T06:50:34.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:34.237+0000 7efc11c00640 1 -- 172.21.15.22:0/3057485153 wait complete. 2024-03-24T06:50:34.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:34 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:50:34.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:34 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:50:34.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:34 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:50:34.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:34 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:50:34.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:34 smithi155 ceph-mon[36262]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:34 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:50:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:34 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:50:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:34 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:50:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:34 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:50:34.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:34 smithi022 ceph-mon[32003]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:34 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi143 to 174.1M 2024-03-24T06:50:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:34 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi143 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-24T06:50:34.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:34 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi155 to 174.1M 2024-03-24T06:50:34.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:34 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi155 to 182634905: error parsing value: Value '182634905' is below minimum 939524096 2024-03-24T06:50:34.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:34 smithi143 ceph-mon[36520]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:35.257 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-24T06:50:35.257 INFO:tasks.fwd_scrub:done joining 2024-03-24T06:50:35.258 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T06:50:35.269 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:50:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 -- 172.21.15.22:0/397265419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 msgr2=0x7f21e41019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 --2- 172.21.15.22:0/397265419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e41019f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009930 tx=0x7f21d002bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 -- 172.21.15.22:0/397265419 shutdown_connections 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 --2- 172.21.15.22:0/397265419 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4101f30 0x7f21e410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 --2- 172.21.15.22:0/397265419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e41019f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 --2- 172.21.15.22:0/397265419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4107630 0x7f21e4107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 -- 172.21.15.22:0/397265419 >> 172.21.15.22:0/397265419 conn(0x7f21e40fd080 msgr2=0x7f21e40ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.485+0000 7f21e3200640 1 -- 172.21.15.22:0/397265419 shutdown_connections 2024-03-24T06:50:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.486+0000 7f21e3200640 1 -- 172.21.15.22:0/397265419 wait complete. 2024-03-24T06:50:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 Processor -- start 2024-03-24T06:50:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 -- start start 2024-03-24T06:50:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4101f30 0x7f21e419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4107630 0x7f21e419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f21e419ec60 con 0x7f21e4107630 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f21e419edd0 con 0x7f21e4101f30 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.487+0000 7f21e3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f21e419ef90 con 0x7f21e4101570 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4101f30 0x7f21e419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43314/0 (socket says 172.21.15.22:43314) 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 -- 172.21.15.22:0/830663730 learned_addr learned my addr 172.21.15.22:0/830663730 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e2800640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4107630 0x7f21e419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 -- 172.21.15.22:0/830663730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4101f30 msgr2=0x7f21e419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4101f30 0x7f21e419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 -- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4107630 msgr2=0x7f21e419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4107630 0x7f21e419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.488+0000 7f21e1e00640 1 -- 172.21.15.22:0/830663730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d4009490 con 0x7f21e4101570 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21e1e00640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f21d400ea60 tx=0x7f21d400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d400cde0 con 0x7f21e4101570 2024-03-24T06:50:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21d40035f0 con 0x7f21e4101570 2024-03-24T06:50:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d0009560 con 0x7f21e4101570 2024-03-24T06:50:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d4010690 con 0x7f21e4101570 2024-03-24T06:50:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.489+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e41a6910 con 0x7f21e4101570 2024-03-24T06:50:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.491+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e4068870 con 0x7f21e4101570 2024-03-24T06:50:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.492+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f21d4003af0 con 0x7f21e4101570 2024-03-24T06:50:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.493+0000 7f21daa00640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 0x7f21c80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.494+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f21d4014070 con 0x7f21e4101570 2024-03-24T06:50:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.494+0000 7f21e1400640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 0x7f21c80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.495+0000 7f21e1400640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 0x7f21c80a4770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009900 tx=0x7f21d0005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:35.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.497+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f21d40971d0 con 0x7f21e4101570 2024-03-24T06:50:35.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:35 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:35 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:35 smithi022 ceph-mon[32003]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:35.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.740+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21e41116c0 con 0x7f21e4101570 2024-03-24T06:50:35.742 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.742+0000 7f21daa00640 1 -- 172.21.15.22:0/830663730 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25271 (secure 0 0 0) 0x7f21d409a890 con 0x7f21e4101570 2024-03-24T06:50:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:50:35.743 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":102,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:50:20.085769+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:50:35.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 msgr2=0x7f21c80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 0x7f21c80a4770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009900 tx=0x7f21d0005a20 comp rx=0 tx=0).stop 2024-03-24T06:50:35.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 msgr2=0x7f21e4199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f21d400ea60 tx=0x7f21d400ef20 comp rx=0 tx=0).stop 2024-03-24T06:50:35.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 shutdown_connections 2024-03-24T06:50:35.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f21c80a22b0 0x7f21c80a4770 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21e4107630 0x7f21e419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.748 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21e4101f30 0x7f21e419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 --2- 172.21.15.22:0/830663730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4101570 0x7f21e4199da0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 >> 172.21.15.22:0/830663730 conn(0x7f21e40fd080 msgr2=0x7f21e40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:35.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 shutdown_connections 2024-03-24T06:50:35.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.746+0000 7f21e3200640 1 -- 172.21.15.22:0/830663730 wait complete. 2024-03-24T06:50:35.768 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:50:35.769 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:50:35.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:35 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:35 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:35 smithi155 ceph-mon[36262]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:35 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:35 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:35.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:35 smithi143 ceph-mon[36520]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:35.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 -- 172.21.15.22:0/1218637160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4410df70 msgr2=0x7f1b44110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1218637160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4410df70 0x7f1b44110360 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009ca0 tx=0x7f1b3c02c180 comp rx=0 tx=0).stop 2024-03-24T06:50:35.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 -- 172.21.15.22:0/1218637160 shutdown_connections 2024-03-24T06:50:35.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1218637160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b4410df70 0x7f1b44110360 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1218637160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b440743e0 0x7f1b4410da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1218637160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b44073ac0 0x7f1b44073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.975+0000 7f1b4b800640 1 -- 172.21.15.22:0/1218637160 >> 172.21.15.22:0/1218637160 conn(0x7f1b440fd140 msgr2=0x7f1b440ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:35.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.976+0000 7f1b4b800640 1 -- 172.21.15.22:0/1218637160 shutdown_connections 2024-03-24T06:50:35.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.976+0000 7f1b4b800640 1 -- 172.21.15.22:0/1218637160 wait complete. 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 Processor -- start 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 -- start start 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b440743e0 0x7f1b4419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b4410df70 0x7f1b4419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4419ee10 con 0x7f1b4410df70 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4419ef80 con 0x7f1b440743e0 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b4b800640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4419f140 con 0x7f1b44073ac0 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b48e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b48e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43324/0 (socket says 172.21.15.22:43324) 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.977+0000 7f1b48e00640 1 -- 172.21.15.22:0/1259628931 learned_addr learned my addr 172.21.15.22:0/1259628931 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:35.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b49800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b4410df70 0x7f1b4419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b3be00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b440743e0 0x7f1b4419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 -- 172.21.15.22:0/1259628931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b440743e0 msgr2=0x7f1b4419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b440743e0 0x7f1b4419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 -- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b4410df70 msgr2=0x7f1b4419e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b4410df70 0x7f1b4419e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:35.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 -- 172.21.15.22:0/1259628931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b34009490 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b48e00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400ea60 tx=0x7f1b3400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.978+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400cde0 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.979+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b340035f0 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.979+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3c009900 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.979+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34010690 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.979+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b441a6ac0 con 0x7f1b44073ac0 2024-03-24T06:50:35.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.980+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b44075170 con 0x7f1b44073ac0 2024-03-24T06:50:35.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.982+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1b34003af0 con 0x7f1b44073ac0 2024-03-24T06:50:35.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.983+0000 7f1b42000640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 0x7f1b280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:35.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.983+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f1b34014070 con 0x7f1b44073ac0 2024-03-24T06:50:35.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.983+0000 7f1b3be00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 0x7f1b280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:35.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.984+0000 7f1b3be00640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 0x7f1b280a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b44100ce0 tx=0x7f1b2c005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:35.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:35.991+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b34097180 con 0x7f1b44073ac0 2024-03-24T06:50:36.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.255+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b44109150 con 0x7f1b44073ac0 2024-03-24T06:50:36.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.256+0000 7f1b42000640 1 -- 172.21.15.22:0/1259628931 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f1b3409a840 con 0x7f1b44073ac0 2024-03-24T06:50:36.257 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:50:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 msgr2=0x7f1b280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 0x7f1b280a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b44100ce0 tx=0x7f1b2c005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 msgr2=0x7f1b44199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.260 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400ea60 tx=0x7f1b3400ef20 comp rx=0 tx=0).stop 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 shutdown_connections 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f1b280a2230 0x7f1b280a46f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1b4410df70 0x7f1b4419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b440743e0 0x7f1b4419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 --2- 172.21.15.22:0/1259628931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b44073ac0 0x7f1b44199f50 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 >> 172.21.15.22:0/1259628931 conn(0x7f1b440fd140 msgr2=0x7f1b440fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 shutdown_connections 2024-03-24T06:50:36.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.259+0000 7f1b4b800640 1 -- 172.21.15.22:0/1259628931 wait complete. 2024-03-24T06:50:36.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:50:36.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 -- 172.21.15.22:0/3116074574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a41097a0 msgr2=0x7fb1a4109b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 --2- 172.21.15.22:0/3116074574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a41097a0 0x7fb1a4109b80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb194009930 tx=0x7fb19402bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:36.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 -- 172.21.15.22:0/3116074574 shutdown_connections 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 --2- 172.21.15.22:0/3116074574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4104150 0x7fb1a4110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 --2- 172.21.15.22:0/3116074574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a4103790 0x7fb1a4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 --2- 172.21.15.22:0/3116074574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a41097a0 0x7fb1a4109b80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 -- 172.21.15.22:0/3116074574 >> 172.21.15.22:0/3116074574 conn(0x7fb1a40ff240 msgr2=0x7fb1a4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 -- 172.21.15.22:0/3116074574 shutdown_connections 2024-03-24T06:50:36.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.499+0000 7fb1a3200640 1 -- 172.21.15.22:0/3116074574 wait complete. 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.500+0000 7fb1a3200640 1 Processor -- start 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 -- start start 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 0x7fb1a4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 0x7fb1a419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a419ed20 con 0x7fb1a4103790 2024-03-24T06:50:36.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a419ee90 con 0x7fb1a41097a0 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a3200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a419f050 con 0x7fb1a4104150 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 0x7fb1a4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a2800640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 0x7fb1a419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a1400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43344/0 (socket says 172.21.15.22:43344) 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a1400640 1 -- 172.21.15.22:0/4160511153 learned_addr learned my addr 172.21.15.22:0/4160511153 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.501+0000 7fb1a1e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 0x7fb1a4199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48946/0 (socket says 172.21.15.22:48946) 2024-03-24T06:50:36.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 -- 172.21.15.22:0/4160511153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 msgr2=0x7fb1a419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 0x7fb1a419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 -- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 msgr2=0x7fb1a4199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 0x7fb1a4199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 -- 172.21.15.22:0/4160511153 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb190009490 con 0x7fb1a4104150 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a2800640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 0x7fb1a419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a1400640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb19000ea60 tx=0x7fb19000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:36.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000cde0 con 0x7fb1a4104150 2024-03-24T06:50:36.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb194009560 con 0x7fb1a4104150 2024-03-24T06:50:36.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1900035f0 con 0x7fb1a4104150 2024-03-24T06:50:36.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190010690 con 0x7fb1a4104150 2024-03-24T06:50:36.506 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.502+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41a69d0 con 0x7fb1a4104150 2024-03-24T06:50:36.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.504+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a4068870 con 0x7fb1a4104150 2024-03-24T06:50:36.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.506+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb190003af0 con 0x7fb1a4104150 2024-03-24T06:50:36.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.507+0000 7fb19aa00640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 0x7fb1880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.507+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7fb190014070 con 0x7fb1a4104150 2024-03-24T06:50:36.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.507+0000 7fb1a1e00640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 0x7fb1880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.509+0000 7fb1a1e00640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 0x7fb1880a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb194009900 tx=0x7fb194005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:36.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.512+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb190097170 con 0x7fb1a4104150 2024-03-24T06:50:36.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/830663730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:50:36.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:50:36.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:50:36.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:50:36.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:36.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:36 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1259628931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:50:36.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.736+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb1a410bd30 con 0x7fb1a4104150 2024-03-24T06:50:36.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.737+0000 7fb19aa00640 1 -- 172.21.15.22:0/4160511153 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7fb19009a830 con 0x7fb1a4104150 2024-03-24T06:50:36.737 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 msgr2=0x7fb1880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 0x7fb1880a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb194009900 tx=0x7fb194005a20 comp rx=0 tx=0).stop 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 msgr2=0x7fb1a419a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb19000ea60 tx=0x7fb19000ef20 comp rx=0 tx=0).stop 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 shutdown_connections 2024-03-24T06:50:36.740 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb1880a2230 0x7fb1880a46f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1a41097a0 0x7fb1a419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a4104150 0x7fb1a419a3a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 --2- 172.21.15.22:0/4160511153 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1a4103790 0x7fb1a4199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 >> 172.21.15.22:0/4160511153 conn(0x7fb1a40ff240 msgr2=0x7fb1a40ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 shutdown_connections 2024-03-24T06:50:36.741 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.740+0000 7fb1a3200640 1 -- 172.21.15.22:0/4160511153 wait complete. 2024-03-24T06:50:36.763 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:50:36.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/830663730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:36.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:36 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1259628931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:50:36.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/830663730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:50:36.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T06:50:36.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: Adjusting osd_memory_target on smithi022 to 177.8M 2024-03-24T06:50:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: Unable to set osd_memory_target on smithi022 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-24T06:50:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T06:50:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T06:50:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='mgr.25492 ' entity='mgr.x' 2024-03-24T06:50:36.930 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:36 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1259628931' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 -- 172.21.15.22:0/119124804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef68074400 msgr2=0x7fef6810da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 --2- 172.21.15.22:0/119124804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef68074400 0x7fef6810da00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fef50009900 tx=0x7fef5002bcd0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 -- 172.21.15.22:0/119124804 shutdown_connections 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 --2- 172.21.15.22:0/119124804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef6810df40 0x7fef68110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 --2- 172.21.15.22:0/119124804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef68074400 0x7fef6810da00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 --2- 172.21.15.22:0/119124804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 0x7fef68073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 -- 172.21.15.22:0/119124804 >> 172.21.15.22:0/119124804 conn(0x7fef680fd1b0 msgr2=0x7fef680ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 -- 172.21.15.22:0/119124804 shutdown_connections 2024-03-24T06:50:36.986 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.985+0000 7fef6e400640 1 -- 172.21.15.22:0/119124804 wait complete. 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.986+0000 7fef6e400640 1 Processor -- start 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 -- start start 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 0x7fef6819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef6810df40 0x7fef6819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fef6819ef30 con 0x7fef68073ae0 2024-03-24T06:50:36.988 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fef6819f0a0 con 0x7fef6810df40 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef6e400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fef6819f260 con 0x7fef68074400 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef67400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 0x7fef6819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef5fe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43350/0 (socket says 172.21.15.22:43350) 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef67e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef6810df40 0x7fef6819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.987+0000 7fef5fe00640 1 -- 172.21.15.22:0/755456759 learned_addr learned my addr 172.21.15.22:0/755456759 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 -- 172.21.15.22:0/755456759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef6810df40 msgr2=0x7fef6819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.989 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef6810df40 0x7fef6819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 -- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 msgr2=0x7fef6819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 0x7fef6819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 -- 172.21.15.22:0/755456759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef50009560 con 0x7fef68074400 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef5fe00640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fef50000c00 tx=0x7fef5002c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5002dbe0 con 0x7fef68074400 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef5002dd40 con 0x7fef68074400 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.988+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef6819f4e0 con 0x7fef68074400 2024-03-24T06:50:36.990 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.989+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50003b20 con 0x7fef68074400 2024-03-24T06:50:36.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.989+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef681a6dc0 con 0x7fef68074400 2024-03-24T06:50:36.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.990+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef68068500 con 0x7fef68074400 2024-03-24T06:50:36.991 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.992+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fef50003c80 con 0x7fef68074400 2024-03-24T06:50:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.993+0000 7fef64c00640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 0x7fef4c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.993+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7fef500f0910 con 0x7fef68074400 2024-03-24T06:50:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.993+0000 7fef67400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 0x7fef4c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:36.993 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.994+0000 7fef67400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 0x7fef4c0a4770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fef58009bd0 tx=0x7fef58005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:36.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:36.996+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fef500b9330 con 0x7fef68074400 2024-03-24T06:50:37.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.198+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fef681090c0 con 0x7fef68074400 2024-03-24T06:50:37.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.199+0000 7fef64c00640 1 -- 172.21.15.22:0/755456759 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7fef50043020 con 0x7fef68074400 2024-03-24T06:50:37.199 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 msgr2=0x7fef4c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 0x7fef4c0a4770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fef58009bd0 tx=0x7fef58005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 msgr2=0x7fef6819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fef50000c00 tx=0x7fef5002c6d0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 shutdown_connections 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fef4c0a22b0 0x7fef4c0a4770 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.202 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef6810df40 0x7fef6819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef68074400 0x7fef6819a5b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 --2- 172.21.15.22:0/755456759 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef68073ae0 0x7fef6819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 >> 172.21.15.22:0/755456759 conn(0x7fef680fd1b0 msgr2=0x7fef680fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 shutdown_connections 2024-03-24T06:50:37.203 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.202+0000 7fef6e400640 1 -- 172.21.15.22:0/755456759 wait complete. 2024-03-24T06:50:37.221 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 -- 172.21.15.22:0/2230047341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d4104150 msgr2=0x7f92d4110660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2230047341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d4104150 0x7f92d4110660 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009cd0 tx=0x7f92cc02c0f0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 -- 172.21.15.22:0/2230047341 shutdown_connections 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2230047341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d4104150 0x7f92d4110660 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2230047341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4103790 0x7f92d4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.441 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.441+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2230047341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d41097a0 0x7f92d4109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.442+0000 7f92d9a00640 1 -- 172.21.15.22:0/2230047341 >> 172.21.15.22:0/2230047341 conn(0x7f92d40ff220 msgr2=0x7f92d4101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:37.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.442+0000 7f92d9a00640 1 -- 172.21.15.22:0/2230047341 shutdown_connections 2024-03-24T06:50:37.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.442+0000 7f92d9a00640 1 -- 172.21.15.22:0/2230047341 wait complete. 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 Processor -- start 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 -- start start 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d4103790 0x7f92d4199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d41097a0 0x7f92d419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f92d419ecb0 con 0x7f92d4103790 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92d419ee20 con 0x7f92d4104150 2024-03-24T06:50:37.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d9a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f92d419efe0 con 0x7f92d41097a0 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.443+0000 7f92d3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d4103790 0x7f92d4199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:44186/0 (socket says 172.21.15.22:44186) 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 -- 172.21.15.22:0/2255778917 learned_addr learned my addr 172.21.15.22:0/2255778917 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d9000640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d41097a0 0x7f92d419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 -- 172.21.15.22:0/2255778917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d41097a0 msgr2=0x7f92d419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d41097a0 0x7f92d419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 -- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d4103790 msgr2=0x7f92d4199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d4103790 0x7f92d4199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 -- 172.21.15.22:0/2255778917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92cc009900 con 0x7f92d4104150 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d3400640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f92c000e9f0 tx=0x7f92c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.444+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0009700 con 0x7f92d4104150 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.445+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92c000cd80 con 0x7f92d4104150 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.445+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d419f2c0 con 0x7f92d4104150 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.445+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0018920 con 0x7f92d4104150 2024-03-24T06:50:37.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.445+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d41a6b40 con 0x7f92d4104150 2024-03-24T06:50:37.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.446+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d4068420 con 0x7f92d4104150 2024-03-24T06:50:37.447 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.448+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f92c0018a80 con 0x7f92d4104150 2024-03-24T06:50:37.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.449+0000 7f92d0c00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 0x7f92b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.450+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f92c0014070 con 0x7f92d4104150 2024-03-24T06:50:37.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.450+0000 7f92d3e00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 0x7f92b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.449 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.451+0000 7f92d3e00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 0x7f92b80a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009b50 tx=0x7f92c4005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:37.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.454+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f92c0053280 con 0x7f92d4104150 2024-03-24T06:50:37.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.671+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f92d410bd00 con 0x7f92d4104150 2024-03-24T06:50:37.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.672+0000 7f92d0c00640 1 -- 172.21.15.22:0/2255778917 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f92c0097140 con 0x7f92d4104150 2024-03-24T06:50:37.672 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:50:37.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 msgr2=0x7f92b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 0x7f92b80a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009b50 tx=0x7f92c4005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 msgr2=0x7f92d419a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f92c000e9f0 tx=0x7f92c000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 shutdown_connections 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f92b80a2230 0x7f92b80a46f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f92d41097a0 0x7f92d419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92d4104150 0x7f92d419a330 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 --2- 172.21.15.22:0/2255778917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f92d4103790 0x7f92d4199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 >> 172.21.15.22:0/2255778917 conn(0x7f92d40ff220 msgr2=0x7f92d40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:37.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 shutdown_connections 2024-03-24T06:50:37.676 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.675+0000 7f92d9a00640 1 -- 172.21.15.22:0/2255778917 wait complete. 2024-03-24T06:50:37.693 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T06:50:37.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:37 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4160511153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:50:37.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:37 smithi022 ceph-mon[32003]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:37.772 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:37 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/755456759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:50:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:37 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4160511153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:50:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:37 smithi155 ceph-mon[36262]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:37.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:37 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/755456759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:50:37.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 -- 172.21.15.22:0/3625665133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 msgr2=0x7f0c88076020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 --2- 172.21.15.22:0/3625665133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c88076020 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0c78009930 tx=0x7f0c7802bc20 comp rx=0 tx=0).stop 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 -- 172.21.15.22:0/3625665133 shutdown_connections 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 --2- 172.21.15.22:0/3625665133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 0x7f0c88114780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 --2- 172.21.15.22:0/3625665133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 0x7f0c88111e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 --2- 172.21.15.22:0/3625665133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c88076020 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 -- 172.21.15.22:0/3625665133 >> 172.21.15.22:0/3625665133 conn(0x7f0c880ff1a0 msgr2=0x7f0c881015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 -- 172.21.15.22:0/3625665133 shutdown_connections 2024-03-24T06:50:37.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.900+0000 7f0c87e00640 1 -- 172.21.15.22:0/3625665133 wait complete. 2024-03-24T06:50:37.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.901+0000 7f0c87e00640 1 Processor -- start 2024-03-24T06:50:37.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.901+0000 7f0c87e00640 1 -- start start 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 0x7f0c8819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 0x7f0c881a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881a30b0 con 0x7f0c88112390 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881a3220 con 0x7f0c88076560 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881a33e0 con 0x7f0c88075c40 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:43378/0 (socket says 172.21.15.22:43378) 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 -- 172.21.15.22:0/1781542745 learned_addr learned my addr 172.21.15.22:0/1781542745 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c87400640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 0x7f0c881a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86000640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 0x7f0c8819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 -- 172.21.15.22:0/1781542745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 msgr2=0x7f0c8819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 0x7f0c8819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 -- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 msgr2=0x7f0c881a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 0x7f0c881a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.902+0000 7f0c86a00640 1 -- 172.21.15.22:0/1781542745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c78009560 con 0x7f0c88075c40 2024-03-24T06:50:37.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c86a00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0c78000c00 tx=0x7f0c7802c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7802dea0 con 0x7f0c88075c40 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c780042d0 con 0x7f0c88075c40 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c881a3660 con 0x7f0c88075c40 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78003a90 con 0x7f0c88075c40 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.903+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c881aaf40 con 0x7f0c88075c40 2024-03-24T06:50:37.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.904+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c88068870 con 0x7f0c88075c40 2024-03-24T06:50:37.906 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.906+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0c780053d0 con 0x7f0c88075c40 2024-03-24T06:50:37.907 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.908+0000 7f0c7f400640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 0x7f0c6c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:50:37.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.908+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9467+0+0 (secure 0 0 0) 0x7f0c780f0880 con 0x7f0c88075c40 2024-03-24T06:50:37.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.909+0000 7f0c86000640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 0x7f0c6c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:50:37.908 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.909+0000 7f0c86000640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 0x7f0c6c0a4770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0c88101670 tx=0x7f0c74005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:50:37.910 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:37.911+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c780b9210 con 0x7f0c88075c40 2024-03-24T06:50:37.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:37 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4160511153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:50:37.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:37 smithi143 ceph-mon[36520]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:37.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:37 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/755456759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:50:38.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.179+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f0c8819f540 con 0x7f0c88075c40 2024-03-24T06:50:38.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.190+0000 7f0c7f400640 1 -- 172.21.15.22:0/1781542745 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+736927 (secure 0 0 0) 0x7f0c780bc8d0 con 0x7f0c88075c40 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "epoch": 27, 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "active_gid": 25492, 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "active_name": "x", 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "active_addrs": { 2024-03-24T06:50:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:6800", 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 1309982651 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v1", 2024-03-24T06:50:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:6801", 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 1309982651 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "active_addr": "172.21.15.22:6801/1309982651", 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "active_change": "2024-03-24T06:40:00.113370+0000", 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "available": true, 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "standbys": [ 2024-03-24T06:50:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "gid": 25442, 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "y", 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr_features": 4540701547738038271, 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "available_modules": [ 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "alerts", 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_destination": { 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_destination", 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email address to send alerts to", 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_from_name": { 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_from_name", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "Ceph", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email From: name", 2024-03-24T06:50:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_host": { 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_host", 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP server", 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_password": { 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_password", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Password to authenticate with", 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_port": { 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_port", 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "465", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP port", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_sender": { 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_sender", 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP envelope sender", 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_ssl": { 2024-03-24T06:50:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_ssl", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_user": { 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_user", 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "User to authenticate as", 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "balancer", 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "active": { 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "active", 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_time": { 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_time", 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0000", 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_weekday": { 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_weekday", 2024-03-24T06:50:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_max_iterations": { 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_max_iterations", 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "25", 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "250", 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_metrics": { 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_metrics", 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_step": { 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_step", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.5", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0.001", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "0.999", 2024-03-24T06:50:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "aggressiveness of optimization", 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "end_time": { 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_time", 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2359", 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "end_weekday": { 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_weekday", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "min_score": { 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_score", 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "upmap", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "crush-compat", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "none", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "read", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap-read" 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Balancer mode", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_ids": { 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_ids", 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_deviation": { 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_deviation", 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_optimizations": { 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_optimizations", 2024-03-24T06:50:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm", 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_down_multiplier": { 2024-03-24T06:50:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_down_multiplier", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T06:50:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_refresh_rate": { 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_refresh_rate", 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "20", 2024-03-24T06:50:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_starting_port": { 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_starting_port", 2024-03-24T06:50:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4721", 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_ptrace": { 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_ptrace", 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_interval": { 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_interval", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T06:50:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_memory_target_ratio": { 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.7", 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm_log_destination": { 2024-03-24T06:50:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm_log_destination", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "file", 2024-03-24T06:50:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "file,syslog", 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "syslog" 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "cgroups_split": { 2024-03-24T06:50:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cgroups_split", 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "config_checks_enabled": { 2024-03-24T06:50:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_checks_enabled", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "config_dashboard": { 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_dashboard", 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_alertmanager": { 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_alertmanager", 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_base": { 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_base", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Container image name, without the tag", 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_elasticsearch": { 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_elasticsearch", 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "elasticsearch container image", 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_grafana": { 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_grafana", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_haproxy": { 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_haproxy", 2024-03-24T06:50:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "HAproxy container image", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_agent": { 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_agent", 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger agent container image", 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_collector": { 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_collector", 2024-03-24T06:50:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger collector container image", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_query": { 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_query", 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T06:50:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger query container image", 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_keepalived": { 2024-03-24T06:50:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_keepalived", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Keepalived container image", 2024-03-24T06:50:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_loki": { 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_loki", 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T06:50:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Loki container image", 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_node_exporter": { 2024-03-24T06:50:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_node_exporter", 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_nvmeof": { 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_nvmeof", 2024-03-24T06:50:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Nvme-of container image", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_prometheus": { 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_prometheus", 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_promtail": { 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_promtail", 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Promtail container image", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_snmp_gateway": { 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_snmp_gateway", 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SNMP Gateway container image", 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "container_init": { 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_init", 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "daemon_cache_timeout": { 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "daemon_cache_timeout", 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "default_cephadm_command_timeout": { 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "900", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "default_registry": { 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_registry", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "device_cache_timeout": { 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_cache_timeout", 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1800", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache device inventory", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "device_enhanced_scan": { 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_enhanced_scan", 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "facts_cache_timeout": { 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "facts_cache_timeout", 2024-03-24T06:50:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache host facts data", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "host_check_interval": { 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "host_check_interval", 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to perform a host check", 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "hw_monitoring": { 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hw_monitoring", 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "inventory_list_all": { 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "inventory_list_all", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "log_refresh_metadata": { 2024-03-24T06:50:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_refresh_metadata", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T06:50:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "*", 2024-03-24T06:50:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "max_count_per_host": { 2024-03-24T06:50:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_count_per_host", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of daemons per service per host", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "max_osd_draining_count": { 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_osd_draining_count", 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "migration_current": { 2024-03-24T06:50:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "migration_current", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "internal - do not modify", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "root", 2024-03-24T06:50:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm-package", 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "root" 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "oob_default_addr": { 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "oob_default_addr", 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "169.254.1.1", 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "prometheus_alerts_path": { 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus_alerts_path", 2024-03-24T06:50:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_insecure": { 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_insecure", 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_password": { 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_password", 2024-03-24T06:50:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_url": { 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_url", 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_username": { 2024-03-24T06:50:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_username", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T06:50:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "secure_monitoring_stack": { 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "secure_monitoring_stack", 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "service_discovery_port": { 2024-03-24T06:50:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "service_discovery_port", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8765", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "cephadm service discovery port", 2024-03-24T06:50:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "ssh_config_file": { 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssh_config_file", 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "use_agent": { 2024-03-24T06:50:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_agent", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T06:50:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "use_repo_digest": { 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_repo_digest", 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_failed_host_check": { 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_failed_host_check", 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_daemons": { 2024-03-24T06:50:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_daemons", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_hosts": { 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_hosts", 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crash", 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "retain_interval": { 2024-03-24T06:50:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retain_interval", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "31536000", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T06:50:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_recent_interval": { 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_recent_interval", 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1209600", 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dashboard", 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T06:50:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_ENABLED": { 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T06:50:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_URL": { 2024-03-24T06:50:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_URL", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T06:50:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8", 2024-03-24T06:50:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T06:50:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "45", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T06:50:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T06:50:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T06:50:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "cross_origin_url": { 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cross_origin_url", 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "crt_file": { 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crt_file", 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "debug": { 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "debug", 2024-03-24T06:50:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable/disable debug options", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "jwt_token_ttl": { 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "jwt_token_ttl", 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "28800", 2024-03-24T06:50:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "motd": { 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "motd", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "The message of the day", 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect_resolve_ip_addr": { 2024-03-24T06:50:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8080", 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl_server_port": { 2024-03-24T06:50:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl_server_port", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8443", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "redirect", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect" 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "url_prefix": { 2024-03-24T06:50:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url_prefix", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "devicehealth", 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_monitoring": { 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_monitoring", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "monitor device health metrics", 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "mark_out_threshold": { 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mark_out_threshold", 2024-03-24T06:50:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2419200", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_name": { 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_name", 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "device_health_metrics", 2024-03-24T06:50:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "retention_period": { 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retention_period", 2024-03-24T06:50:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15552000", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain device health metrics", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_frequency": { 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_frequency", 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "self_heal": { 2024-03-24T06:50:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "self_heal", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_threshold": { 2024-03-24T06:50:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_threshold", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7257600", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T06:50:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "diskprediction_local", 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.492 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.493 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "predict_interval": { 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predict_interval", 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.494 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.497 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "predictor_model": { 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predictor_model", 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.498 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "prophetstor", 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.499 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.500 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "influx", 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": false, 2024-03-24T06:50:38.501 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "influxdb python module not found", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "batch_size": { 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "batch_size", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5000", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.502 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "database": { 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "database", 2024-03-24T06:50:38.503 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "ceph", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.504 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "hostname": { 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hostname", 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.505 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server hostname", 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.506 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "30", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "5", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T06:50:38.507 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.508 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.509 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.510 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.511 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.512 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.513 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.514 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.515 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "password": { 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "password", 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.516 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "password of InfluxDB server user", 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.517 INFO:teuthology.orchestra.run.smithi022.stdout: "port": { 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "port", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8086", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server port", 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.519 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "false", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.520 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "threads": { 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threads", 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.521 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "32", 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "username": { 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "username", 2024-03-24T06:50:38.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "username of InfluxDB server user", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.523 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "verify_ssl": { 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "verify_ssl", 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "true", 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.524 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "insights", 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.526 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.527 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.528 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.529 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.531 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.533 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "iostat", 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.534 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.535 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.536 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.537 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.538 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.539 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.540 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.541 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "k8sevents", 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_event_retention_days": { 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ceph_event_retention_days", 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.543 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "config_check_secs": { 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_check_secs", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "10", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.545 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.546 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.547 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.548 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.549 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.550 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.551 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.552 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.553 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "localpool", 2024-03-24T06:50:38.554 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "failure_domain": { 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "failure_domain", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "host", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.555 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "failure domain for any created local pool", 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.556 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.557 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.559 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.560 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.561 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.562 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.563 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "min_size": { 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_size", 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.564 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default min_size for any created local pool", 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "num_rep": { 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "num_rep", 2024-03-24T06:50:38.565 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default replica count for any created local pool", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_num": { 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_num", 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "128", 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.567 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "prefix": { 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prefix", 2024-03-24T06:50:38.568 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.569 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name prefix for any created local pool", 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "subtree": { 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "subtree", 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.570 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "rack", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T06:50:38.571 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mds_autoscaler", 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.572 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.573 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.574 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.575 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.576 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.577 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.578 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.579 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.580 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirroring", 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.581 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.582 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.584 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.585 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.586 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.587 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.588 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "nfs", 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.590 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.591 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.592 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.593 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.594 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.595 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.596 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "fail_fs": { 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "fail_fs", 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.598 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.600 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.601 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.602 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.603 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.604 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.605 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.606 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator": { 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.607 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "rook", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "test_orchestrator" 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Orchestrator backend", 2024-03-24T06:50:38.608 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_perf_query", 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.609 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.610 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.611 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.612 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.614 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.615 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.616 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_support", 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.617 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.618 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.619 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.620 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.621 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.622 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.623 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.624 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.625 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_autoscaler", 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.626 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.627 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.628 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.630 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.631 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.632 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.633 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.634 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "threshold": { 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threshold", 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:38.635 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1.0", 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "scaling threshold", 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "progress", 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_pg_recovery_event": { 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_pg_recovery_event", 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.637 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.638 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.640 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.641 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.642 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.643 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.644 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.645 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.646 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "max_completed_events": { 2024-03-24T06:50:38.647 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_completed_events", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "50", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "number of past completed events to remember", 2024-03-24T06:50:38.648 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.649 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long the module is going to sleep", 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus", 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "cache": { 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cache", 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "exclude_perf_counters": { 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "exclude_perf_counters", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.653 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.654 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.655 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.658 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.659 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.660 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools": { 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools", 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.662 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T06:50:38.663 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "300", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.664 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_interval": { 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_interval", 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15.0", 2024-03-24T06:50:38.665 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:38.666 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.667 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "9283", 2024-03-24T06:50:38.668 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "stale_cache_strategy": { 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stale_cache_strategy", 2024-03-24T06:50:38.669 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "log", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.670 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "default", 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.671 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "default", 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "error" 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.672 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.673 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_support", 2024-03-24T06:50:38.674 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.675 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.676 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.677 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.678 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.679 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.680 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.681 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_snap_create": { 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_snap_create", 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.683 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "mirror_snapshot_schedule": { 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.684 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.685 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "trash_purge_schedule": { 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "trash_purge_schedule", 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.686 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "restful", 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_auth": { 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_auth", 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.688 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.691 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.692 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.693 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.694 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.695 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.696 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.697 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.698 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.699 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:38.700 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.701 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rgw", 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.702 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.703 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.704 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.705 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.706 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.707 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.709 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rook", 2024-03-24T06:50:38.710 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.711 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.712 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.713 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.714 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.715 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.717 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "storage_class": { 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "storage_class", 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.718 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "local", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.719 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "selftest", 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.720 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.721 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.722 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.723 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.724 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.725 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.726 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.727 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "roption1": { 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption1", 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.728 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.729 INFO:teuthology.orchestra.run.smithi022.stdout: "roption2": { 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption2", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "xyz", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.730 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption1": { 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption1", 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.731 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.732 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption2": { 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption2", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.733 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption3": { 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption3", 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.735 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption4": { 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption4", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.736 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption5": { 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption5", 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.737 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.738 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption6": { 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption6", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.739 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption7": { 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption7", 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "42", 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.741 INFO:teuthology.orchestra.run.smithi022.stdout: "testkey": { 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testkey", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.742 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "testlkey": { 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testlkey", 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.743 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.744 INFO:teuthology.orchestra.run.smithi022.stdout: "testnewline": { 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testnewline", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.745 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snap_schedule", 2024-03-24T06:50:38.746 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_m_granularity": { 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_m_granularity", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.747 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "dump_on_update": { 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dump_on_update", 2024-03-24T06:50:38.748 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "dump database to debug log on update", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.749 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.750 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.751 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.752 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.754 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.755 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.756 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.757 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stats", 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.758 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.759 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.760 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.761 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.762 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.763 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.764 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.765 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "status", 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.767 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.768 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.769 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.770 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.771 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.773 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.774 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telegraf", 2024-03-24T06:50:38.775 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "address": { 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "address", 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T06:50:38.776 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.777 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.778 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.779 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.780 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.782 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.783 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.784 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.785 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.786 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telemetry", 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.787 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_basic": { 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_basic", 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.788 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_crash": { 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_crash", 2024-03-24T06:50:38.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.790 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_device": { 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_device", 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.791 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_ident": { 2024-03-24T06:50:38.792 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_ident", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.793 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_perf": { 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_perf", 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.794 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "contact": { 2024-03-24T06:50:38.795 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "contact", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.796 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": { 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "description", 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "device_url": { 2024-03-24T06:50:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_url", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.799 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.800 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.801 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "24", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "8", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.802 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "last_opt_revision": { 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "last_opt_revision", 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1", 2024-03-24T06:50:38.803 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard": { 2024-03-24T06:50:38.804 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.805 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard_description": { 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard_description", 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.808 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.809 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.810 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.811 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.812 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.813 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.814 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "organization": { 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "organization", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.815 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "proxy": { 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "proxy", 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.816 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "url": { 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.818 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.819 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "test_orchestrator", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.820 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.821 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.822 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.824 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.825 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.826 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "volumes", 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_clones": { 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_clones", 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4", 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "periodic_async_work": { 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "periodic_async_work", 2024-03-24T06:50:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Periodically check for async work", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_delay": { 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_delay", 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_no_wait": { 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T06:50:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix", 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "discovery_interval": { 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "discovery_interval", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "100", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "identifier": { 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "identifier", 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_host": { 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_host", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_port": { 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_port", 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10051", 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_sender": { 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_sender", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "gid": 25489, 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "z", 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr_features": 4540701547738038271, 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "available_modules": [ 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "alerts", 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_destination": { 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_destination", 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email address to send alerts to", 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_from_name": { 2024-03-24T06:50:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_from_name", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "Ceph", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email From: name", 2024-03-24T06:50:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_host": { 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_host", 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP server", 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_password": { 2024-03-24T06:50:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_password", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Password to authenticate with", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_port": { 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_port", 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "465", 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP port", 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_sender": { 2024-03-24T06:50:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_sender", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP envelope sender", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_ssl": { 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_ssl", 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_user": { 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_user", 2024-03-24T06:50:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "User to authenticate as", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "balancer", 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "active": { 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "active", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_time": { 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_time", 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0000", 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_weekday": { 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_weekday", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_max_iterations": { 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_max_iterations", 2024-03-24T06:50:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "25", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "250", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_metrics": { 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_metrics", 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_step": { 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_step", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.5", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0.001", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "0.999", 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "aggressiveness of optimization", 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "end_time": { 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_time", 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2359", 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "end_weekday": { 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_weekday", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "min_score": { 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_score", 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "upmap", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "crush-compat", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "none", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "read", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap-read" 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Balancer mode", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_ids": { 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_ids", 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_deviation": { 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_deviation", 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_optimizations": { 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_optimizations", 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm", 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_down_multiplier": { 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_down_multiplier", 2024-03-24T06:50:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T06:50:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_refresh_rate": { 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_refresh_rate", 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "20", 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_starting_port": { 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_starting_port", 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4721", 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_ptrace": { 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_ptrace", 2024-03-24T06:50:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T06:50:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_interval": { 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_interval", 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_memory_target_ratio": { 2024-03-24T06:50:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.7", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T06:50:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm_log_destination": { 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm_log_destination", 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "file", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "file,syslog", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "syslog" 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T06:50:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "cgroups_split": { 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cgroups_split", 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "config_checks_enabled": { 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_checks_enabled", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "config_dashboard": { 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_dashboard", 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_alertmanager": { 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_alertmanager", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_base": { 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_base", 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Container image name, without the tag", 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_elasticsearch": { 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_elasticsearch", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "elasticsearch container image", 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_grafana": { 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_grafana", 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_haproxy": { 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_haproxy", 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "HAproxy container image", 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_agent": { 2024-03-24T06:50:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_agent", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger agent container image", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_collector": { 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_collector", 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T06:50:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger collector container image", 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_query": { 2024-03-24T06:50:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_query", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger query container image", 2024-03-24T06:50:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_keepalived": { 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_keepalived", 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Keepalived container image", 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_loki": { 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_loki", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Loki container image", 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_node_exporter": { 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_node_exporter", 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_nvmeof": { 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_nvmeof", 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T06:50:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Nvme-of container image", 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_prometheus": { 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_prometheus", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_promtail": { 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_promtail", 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Promtail container image", 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_snmp_gateway": { 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_snmp_gateway", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SNMP Gateway container image", 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "container_init": { 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_init", 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "daemon_cache_timeout": { 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "daemon_cache_timeout", 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "default_cephadm_command_timeout": { 2024-03-24T06:50:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "900", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "default_registry": { 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_registry", 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io", 2024-03-24T06:50:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "device_cache_timeout": { 2024-03-24T06:50:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_cache_timeout", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1800", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache device inventory", 2024-03-24T06:50:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "device_enhanced_scan": { 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_enhanced_scan", 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "facts_cache_timeout": { 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "facts_cache_timeout", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache host facts data", 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "host_check_interval": { 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "host_check_interval", 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to perform a host check", 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "hw_monitoring": { 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hw_monitoring", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "inventory_list_all": { 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "inventory_list_all", 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "log_refresh_metadata": { 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_refresh_metadata", 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T06:50:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "*", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "max_count_per_host": { 2024-03-24T06:50:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_count_per_host", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of daemons per service per host", 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "max_osd_draining_count": { 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_osd_draining_count", 2024-03-24T06:50:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T06:50:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "migration_current": { 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "migration_current", 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "internal - do not modify", 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "root", 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm-package", 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "root" 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T06:50:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "oob_default_addr": { 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "oob_default_addr", 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "169.254.1.1", 2024-03-24T06:50:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "prometheus_alerts_path": { 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus_alerts_path", 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_insecure": { 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_insecure", 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_password": { 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_password", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T06:50:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_url": { 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_url", 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_username": { 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_username", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "secure_monitoring_stack": { 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "secure_monitoring_stack", 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "service_discovery_port": { 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "service_discovery_port", 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8765", 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "cephadm service discovery port", 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "ssh_config_file": { 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssh_config_file", 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "use_agent": { 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_agent", 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "use_repo_digest": { 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_repo_digest", 2024-03-24T06:50:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_failed_host_check": { 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_failed_host_check", 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_daemons": { 2024-03-24T06:50:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_daemons", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_hosts": { 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_hosts", 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crash", 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "retain_interval": { 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retain_interval", 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "31536000", 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_recent_interval": { 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_recent_interval", 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1209600", 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dashboard", 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T06:50:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_ENABLED": { 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T06:50:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T06:50:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_URL": { 2024-03-24T06:50:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_URL", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T06:50:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T06:50:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T06:50:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T06:50:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T06:50:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "45", 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "cross_origin_url": { 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cross_origin_url", 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "crt_file": { 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crt_file", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "debug": { 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "debug", 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable/disable debug options", 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "jwt_token_ttl": { 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "jwt_token_ttl", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "28800", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "motd": { 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "motd", 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "The message of the day", 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect_resolve_ip_addr": { 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8080", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl_server_port": { 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl_server_port", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8443", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "redirect", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect" 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "url_prefix": { 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url_prefix", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "devicehealth", 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_monitoring": { 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_monitoring", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "monitor device health metrics", 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "mark_out_threshold": { 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mark_out_threshold", 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2419200", 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_name": { 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_name", 2024-03-24T06:50:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "device_health_metrics", 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "retention_period": { 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retention_period", 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15552000", 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain device health metrics", 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_frequency": { 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_frequency", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "self_heal": { 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "self_heal", 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_threshold": { 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_threshold", 2024-03-24T06:50:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7257600", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T06:50:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "diskprediction_local", 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "predict_interval": { 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predict_interval", 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "predictor_model": { 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predictor_model", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "prophetstor", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "influx", 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": false, 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "influxdb python module not found", 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "batch_size": { 2024-03-24T06:50:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "batch_size", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5000", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "database": { 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "database", 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "ceph", 2024-03-24T06:50:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "hostname": { 2024-03-24T06:50:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hostname", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server hostname", 2024-03-24T06:50:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "30", 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "5", 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "password": { 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "password", 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "password of InfluxDB server user", 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "port": { 2024-03-24T06:50:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "port", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8086", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server port", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "false", 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "threads": { 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threads", 2024-03-24T06:50:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "32", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "username": { 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "username", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "username of InfluxDB server user", 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "verify_ssl": { 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "verify_ssl", 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "true", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "insights", 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "iostat", 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.190 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.191 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.192 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.193 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.194 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "k8sevents", 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.195 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_event_retention_days": { 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ceph_event_retention_days", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.196 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "config_check_secs": { 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_check_secs", 2024-03-24T06:50:39.197 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "10", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.198 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.199 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.200 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.201 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.202 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.203 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.204 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.205 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.206 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "localpool", 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "failure_domain": { 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "failure_domain", 2024-03-24T06:50:39.207 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "host", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "failure domain for any created local pool", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T06:50:39.208 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.209 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.210 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.211 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.213 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.214 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.215 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "min_size": { 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_size", 2024-03-24T06:50:39.216 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default min_size for any created local pool", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.217 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "num_rep": { 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "num_rep", 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3", 2024-03-24T06:50:39.218 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default replica count for any created local pool", 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_num": { 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_num", 2024-03-24T06:50:39.219 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "128", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.220 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "prefix": { 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prefix", 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.221 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name prefix for any created local pool", 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "subtree": { 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "subtree", 2024-03-24T06:50:39.222 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "rack", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T06:50:39.223 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mds_autoscaler", 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.224 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.225 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.226 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.227 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.229 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.230 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.231 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.232 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirroring", 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.233 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.234 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.235 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.236 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.237 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.238 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.239 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.240 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "nfs", 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.241 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.242 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.243 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.244 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.245 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.246 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.247 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.248 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "fail_fs": { 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "fail_fs", 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.250 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.251 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.253 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.254 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.255 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.256 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.257 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.258 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator": { 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.259 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "rook", 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: "test_orchestrator" 2024-03-24T06:50:39.260 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Orchestrator backend", 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_perf_query", 2024-03-24T06:50:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.262 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.263 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.265 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.266 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.267 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.268 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_support", 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_autoscaler", 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "threshold": { 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threshold", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1.0", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "scaling threshold", 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "progress", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_pg_recovery_event": { 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_pg_recovery_event", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "max_completed_events": { 2024-03-24T06:50:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_completed_events", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "50", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "number of past completed events to remember", 2024-03-24T06:50:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long the module is going to sleep", 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus", 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "cache": { 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cache", 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "exclude_perf_counters": { 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "exclude_perf_counters", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools": { 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "300", 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_interval": { 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_interval", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15.0", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "9283", 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "stale_cache_strategy": { 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stale_cache_strategy", 2024-03-24T06:50:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "log", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "default", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "default", 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "error" 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_support", 2024-03-24T06:50:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_snap_create": { 2024-03-24T06:50:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_snap_create", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "mirror_snapshot_schedule": { 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "trash_purge_schedule": { 2024-03-24T06:50:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "trash_purge_schedule", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "restful", 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_auth": { 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_auth", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rgw", 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rook", 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "storage_class": { 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "storage_class", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "local", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "selftest", 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "roption1": { 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption1", 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "roption2": { 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption2", 2024-03-24T06:50:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "xyz", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption1": { 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption1", 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption2": { 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption2", 2024-03-24T06:50:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption3": { 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption3", 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption4": { 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption4", 2024-03-24T06:50:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption5": { 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption5", 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption6": { 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption6", 2024-03-24T06:50:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption7": { 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption7", 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "42", 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "testkey": { 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testkey", 2024-03-24T06:50:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "testlkey": { 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testlkey", 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "testnewline": { 2024-03-24T06:50:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testnewline", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snap_schedule", 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_m_granularity": { 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_m_granularity", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "dump_on_update": { 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dump_on_update", 2024-03-24T06:50:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "dump database to debug log on update", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stats", 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "status", 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telegraf", 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "address": { 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "address", 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telemetry", 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.440 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_basic": { 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_basic", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.441 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_crash": { 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_crash", 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_device": { 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_device", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.444 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_ident": { 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_ident", 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.445 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.446 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_perf": { 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_perf", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.447 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "contact": { 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "contact", 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.448 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.449 INFO:teuthology.orchestra.run.smithi022.stdout: "description": { 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "description", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.450 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "device_url": { 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_url", 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.451 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.453 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.454 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "24", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "8", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "last_opt_revision": { 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "last_opt_revision", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.456 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard": { 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard", 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.458 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard_description": { 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard_description", 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "organization": { 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "organization", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "proxy": { 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "proxy", 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "url": { 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "test_orchestrator", 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "volumes", 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_clones": { 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_clones", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "periodic_async_work": { 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "periodic_async_work", 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Periodically check for async work", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_delay": { 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_delay", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_no_wait": { 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix", 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "discovery_interval": { 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "discovery_interval", 2024-03-24T06:50:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "100", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "identifier": { 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "identifier", 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_host": { 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_host", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_port": { 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_port", 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10051", 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_sender": { 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_sender", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "modules": [ 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm", 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "dashboard", 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "iostat", 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "nfs", 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "restful", 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "snap_schedule" 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "available_modules": [ 2024-03-24T06:50:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "alerts", 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T06:50:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_destination": { 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_destination", 2024-03-24T06:50:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email address to send alerts to", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_from_name": { 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_from_name", 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "Ceph", 2024-03-24T06:50:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Email From: name", 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_host": { 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_host", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP server", 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_password": { 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_password", 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Password to authenticate with", 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_port": { 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_port", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "465", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP port", 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_sender": { 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_sender", 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SMTP envelope sender", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_ssl": { 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_ssl", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "smtp_user": { 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "smtp_user", 2024-03-24T06:50:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "User to authenticate as", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "balancer", 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "active": { 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "active", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_time": { 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_time", 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0000", 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "begin_weekday": { 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "begin_weekday", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_max_iterations": { 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_max_iterations", 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "25", 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "250", 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_metrics": { 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_metrics", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "crush_compat_step": { 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crush_compat_step", 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.5", 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0.001", 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "0.999", 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "aggressiveness of optimization", 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "end_time": { 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_time", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2359", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "end_weekday": { 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "end_weekday", 2024-03-24T06:50:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "0", 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "6", 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "min_score": { 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_score", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "upmap", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "crush-compat", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "none", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "read", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap", 2024-03-24T06:50:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap-read" 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Balancer mode", 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_ids": { 2024-03-24T06:50:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_ids", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_deviation": { 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_deviation", 2024-03-24T06:50:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T06:50:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "upmap_max_optimizations": { 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "upmap_max_optimizations", 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm", 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_down_multiplier": { 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_down_multiplier", 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_refresh_rate": { 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_refresh_rate", 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "20", 2024-03-24T06:50:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "agent_starting_port": { 2024-03-24T06:50:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "agent_starting_port", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4721", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_ptrace": { 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_ptrace", 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_interval": { 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_interval", 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "autotune_memory_target_ratio": { 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T06:50:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0.7", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T06:50:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm_log_destination": { 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cephadm_log_destination", 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "file", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "file,syslog", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "syslog" 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T06:50:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "cgroups_split": { 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cgroups_split", 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "config_checks_enabled": { 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_checks_enabled", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "config_dashboard": { 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_dashboard", 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_alertmanager": { 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_alertmanager", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_base": { 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_base", 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Container image name, without the tag", 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_elasticsearch": { 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_elasticsearch", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "elasticsearch container image", 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_grafana": { 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_grafana", 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_haproxy": { 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_haproxy", 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "HAproxy container image", 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_agent": { 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_agent", 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger agent container image", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_collector": { 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_collector", 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger collector container image", 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_jaeger_query": { 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_jaeger_query", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Jaeger query container image", 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_keepalived": { 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_keepalived", 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Keepalived container image", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_loki": { 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_loki", 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Loki container image", 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_node_exporter": { 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_node_exporter", 2024-03-24T06:50:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_nvmeof": { 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_nvmeof", 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Nvme-of container image", 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_prometheus": { 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_prometheus", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Prometheus container image", 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_promtail": { 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_promtail", 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Promtail container image", 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "container_image_snmp_gateway": { 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_image_snmp_gateway", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "SNMP Gateway container image", 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "container_init": { 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "container_init", 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "daemon_cache_timeout": { 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "daemon_cache_timeout", 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "default_cephadm_command_timeout": { 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T06:50:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "900", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "default_registry": { 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "default_registry", 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "docker.io", 2024-03-24T06:50:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "device_cache_timeout": { 2024-03-24T06:50:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_cache_timeout", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1800", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache device inventory", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "device_enhanced_scan": { 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_enhanced_scan", 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "facts_cache_timeout": { 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "facts_cache_timeout", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "seconds to cache host facts data", 2024-03-24T06:50:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "host_check_interval": { 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "host_check_interval", 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to perform a host check", 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "hw_monitoring": { 2024-03-24T06:50:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hw_monitoring", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T06:50:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "inventory_list_all": { 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "inventory_list_all", 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "log_refresh_metadata": { 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_refresh_metadata", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "*", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "max_count_per_host": { 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_count_per_host", 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of daemons per service per host", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "max_osd_draining_count": { 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_osd_draining_count", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "migration_current": { 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "migration_current", 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "internal - do not modify", 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "mode": { 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mode", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "root", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm-package", 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "root" 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "oob_default_addr": { 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "oob_default_addr", 2024-03-24T06:50:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "169.254.1.1", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "prometheus_alerts_path": { 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus_alerts_path", 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_insecure": { 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_insecure", 2024-03-24T06:50:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_password": { 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_password", 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_url": { 2024-03-24T06:50:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_url", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T06:50:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "registry_username": { 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "registry_username", 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "secure_monitoring_stack": { 2024-03-24T06:50:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "secure_monitoring_stack", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T06:50:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "service_discovery_port": { 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "service_discovery_port", 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8765", 2024-03-24T06:50:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "cephadm service discovery port", 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "ssh_config_file": { 2024-03-24T06:50:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssh_config_file", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T06:50:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "use_agent": { 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_agent", 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "use_repo_digest": { 2024-03-24T06:50:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "use_repo_digest", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_failed_host_check": { 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_failed_host_check", 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_daemons": { 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_daemons", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_on_stray_hosts": { 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_on_stray_hosts", 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crash", 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "retain_interval": { 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retain_interval", 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "31536000", 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_recent_interval": { 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_recent_interval", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1209600", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dashboard", 2024-03-24T06:50:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T06:50:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_ENABLED": { 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T06:50:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T06:50:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T06:50:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T06:50:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_URL": { 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_URL", 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T06:50:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T06:50:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T06:50:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T06:50:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T06:50:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T06:50:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T06:50:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T06:50:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T06:50:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "45", 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "admin", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T06:50:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T06:50:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "cross_origin_url": { 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cross_origin_url", 2024-03-24T06:50:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "crt_file": { 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "crt_file", 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "debug": { 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "debug", 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Enable/disable debug options", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "jwt_token_ttl": { 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "jwt_token_ttl", 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "28800", 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "motd": { 2024-03-24T06:50:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "motd", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "The message of the day", 2024-03-24T06:50:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect_resolve_ip_addr": { 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8080", 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl_server_port": { 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl_server_port", 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8443", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "redirect", 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "redirect" 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "url_prefix": { 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url_prefix", 2024-03-24T06:50:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "devicehealth", 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_monitoring": { 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_monitoring", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "monitor device health metrics", 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "mark_out_threshold": { 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mark_out_threshold", 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "2419200", 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "pool_name": { 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pool_name", 2024-03-24T06:50:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "device_health_metrics", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "retention_period": { 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "retention_period", 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15552000", 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long to retain device health metrics", 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_frequency": { 2024-03-24T06:50:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_frequency", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "self_heal": { 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "self_heal", 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "warn_threshold": { 2024-03-24T06:50:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "warn_threshold", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7257600", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "diskprediction_local", 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "predict_interval": { 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predict_interval", 2024-03-24T06:50:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "86400", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "predictor_model": { 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "predictor_model", 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "prophetstor", 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "600", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "influx", 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": false, 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "influxdb python module not found", 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "batch_size": { 2024-03-24T06:50:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "batch_size", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5000", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T06:50:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "database": { 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "database", 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "ceph", 2024-03-24T06:50:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "hostname": { 2024-03-24T06:50:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "hostname", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server hostname", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "30", 2024-03-24T06:50:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "5", 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "password": { 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "password", 2024-03-24T06:50:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "password of InfluxDB server user", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "port": { 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "port", 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "8086", 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "InfluxDB server port", 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "ssl": { 2024-03-24T06:50:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ssl", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "false", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "threads": { 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threads", 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "32", 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "username": { 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "username", 2024-03-24T06:50:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "username of InfluxDB server user", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "verify_ssl": { 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "verify_ssl", 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "true", 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "insights", 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "iostat", 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "k8sevents", 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_event_retention_days": { 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "ceph_event_retention_days", 2024-03-24T06:50:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "7", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "config_check_secs": { 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "config_check_secs", 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "10", 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "localpool", 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "failure_domain": { 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "failure_domain", 2024-03-24T06:50:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "host", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "failure domain for any created local pool", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "min_size": { 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "min_size", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default min_size for any created local pool", 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "num_rep": { 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "num_rep", 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3", 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default replica count for any created local pool", 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_num": { 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_num", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "128", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "prefix": { 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prefix", 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "name prefix for any created local pool", 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "subtree": { 2024-03-24T06:50:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "subtree", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "rack", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T06:50:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mds_autoscaler", 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirroring", 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "nfs", 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "fail_fs": { 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "fail_fs", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator": { 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "orchestrator", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "cephadm", 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "rook", 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "test_orchestrator" 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Orchestrator backend", 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_perf_query", 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "osd_support", 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "pg_autoscaler", 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "threshold": { 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "threshold", 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "3.0", 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1.0", 2024-03-24T06:50:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "scaling threshold", 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "progress", 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_pg_recovery_event": { 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_pg_recovery_event", 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "max_completed_events": { 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_completed_events", 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "50", 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "number of past completed events to remember", 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "sleep_interval": { 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "sleep_interval", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "5", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "how long the module is going to sleep", 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "prometheus", 2024-03-24T06:50:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "cache": { 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "cache", 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "exclude_perf_counters": { 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "exclude_perf_counters", 2024-03-24T06:50:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools": { 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools", 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "300", 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "scrape_interval": { 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "scrape_interval", 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15.0", 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "::", 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "9283", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "stale_cache_strategy": { 2024-03-24T06:50:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stale_cache_strategy", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "log", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_behaviour": { 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_behaviour", 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "default", 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "default", 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "error" 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "standby_error_status_code": { 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "standby_error_status_code", 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "500", 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "400", 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "599", 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_support", 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_snap_create": { 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_snap_create", 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10", 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "mirror_snapshot_schedule": { 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "trash_purge_schedule": { 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "trash_purge_schedule", 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "restful", 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "enable_auth": { 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enable_auth", 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "key_file": { 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "key_file", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "server_addr": { 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_addr", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "server_port": { 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "server_port", 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rgw", 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rook", 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "storage_class": { 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "storage_class", 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "local", 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "selftest", 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "roption1": { 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption1", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "roption2": { 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "roption2", 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "xyz", 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption1": { 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption1", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption2": { 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption2", 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption3": { 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption3", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "float", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption4": { 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption4", 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption5": { 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption5", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption6": { 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption6", 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "rwoption7": { 2024-03-24T06:50:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rwoption7", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "1", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "42", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "testkey": { 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testkey", 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "testlkey": { 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testlkey", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "testnewline": { 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "testnewline", 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snap_schedule", 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "allow_m_granularity": { 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "allow_m_granularity", 2024-03-24T06:50:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "dump_on_update": { 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "dump_on_update", 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "dump database to debug log on update", 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "stats", 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "status", 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telegraf", 2024-03-24T06:50:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "address": { 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "address", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "15", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "telemetry", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_basic": { 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_basic", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_crash": { 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_crash", 2024-03-24T06:50:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_device": { 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_device", 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_ident": { 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_ident", 2024-03-24T06:50:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "channel_perf": { 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "channel_perf", 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "contact": { 2024-03-24T06:50:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "contact", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "description": { 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "description", 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "device_url": { 2024-03-24T06:50:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "device_url", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "enabled": { 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "enabled", 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "24", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "8", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "last_opt_revision": { 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "last_opt_revision", 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "1", 2024-03-24T06:50:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard": { 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard", 2024-03-24T06:50:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "leaderboard_description": { 2024-03-24T06:50:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "leaderboard_description", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "organization": { 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "organization", 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "proxy": { 2024-03-24T06:50:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "proxy", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "url": { 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "url", 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.123 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "test_orchestrator", 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.124 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.125 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.126 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.129 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.130 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.131 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.132 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.133 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.134 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "volumes", 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.135 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.136 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.137 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.138 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.140 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.141 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.142 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.143 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "max_concurrent_clones": { 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "max_concurrent_clones", 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "4", 2024-03-24T06:50:40.144 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "periodic_async_work": { 2024-03-24T06:50:40.145 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "periodic_async_work", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.146 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Periodically check for async work", 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_delay": { 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_delay", 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "0", 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.148 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "snapshot_clone_no_wait": { 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "True", 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.149 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix", 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "can_run": true, 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "error_string": "", 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "module_options": { 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "discovery_interval": { 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "discovery_interval", 2024-03-24T06:50:40.151 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "uint", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "100", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.152 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "identifier": { 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "identifier", 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.153 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "interval": { 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "interval", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "secs", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "60", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.155 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "log_level": { 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_level", 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.156 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.157 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.158 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster": { 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.159 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_cluster_level": { 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_cluster_level", 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.160 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "info", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [ 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "critical", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "debug", 2024-03-24T06:50:40.161 INFO:teuthology.orchestra.run.smithi022.stdout: "error", 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "info", 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "warning" 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.162 INFO:teuthology.orchestra.run.smithi022.stdout: "log_to_file": { 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "log_to_file", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "bool", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 1, 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "False", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.163 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_host": { 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_host", 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.164 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "", 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_port": { 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_port", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "int", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "10051", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.166 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "zabbix_sender": { 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "zabbix_sender", 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "str", 2024-03-24T06:50:40.167 INFO:teuthology.orchestra.run.smithi022.stdout: "level": "advanced", 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "flags": 0, 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "min": "", 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "max": "", 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "enum_allowed": [], 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "desc": "", 2024-03-24T06:50:40.168 INFO:teuthology.orchestra.run.smithi022.stdout: "long_desc": "", 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tags": [], 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: "see_also": [] 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: "services": { 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: "dashboard": "https://172.21.15.22:8443/" 2024-03-24T06:50:40.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "always_on_modules": { 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "octopus": [ 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "balancer", 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "devicehealth", 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator", 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_autoscaler", 2024-03-24T06:50:40.170 INFO:teuthology.orchestra.run.smithi022.stdout: "progress", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_support", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "status", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "telemetry", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "volumes" 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "pacific": [ 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "balancer", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T06:50:40.171 INFO:teuthology.orchestra.run.smithi022.stdout: "devicehealth", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_autoscaler", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "progress", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_support", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "status", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "telemetry", 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: "volumes" 2024-03-24T06:50:40.172 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "quincy": [ 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "balancer", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "devicehealth", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_autoscaler", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "progress", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_support", 2024-03-24T06:50:40.173 INFO:teuthology.orchestra.run.smithi022.stdout: "status", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "telemetry", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "volumes" 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "reef": [ 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "balancer", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "devicehealth", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator", 2024-03-24T06:50:40.174 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_autoscaler", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "progress", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_support", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "status", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "telemetry", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "volumes" 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "squid": [ 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "balancer", 2024-03-24T06:50:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: "crash", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "devicehealth", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "orchestrator", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "pg_autoscaler", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "progress", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "rbd_support", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "status", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "telemetry", 2024-03-24T06:50:40.176 INFO:teuthology.orchestra.run.smithi022.stdout: "volumes" 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: "last_failure_osd_epoch": 94, 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: "active_clients": [ 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "devicehealth", 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.177 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 239652573 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "libcephsqlite", 2024-03-24T06:50:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 2794834311 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.179 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "rbd_support", 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 3828001843 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snap_schedule", 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 586254357 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "snap_schedule", 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 751412043 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.182 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: "name": "volumes", 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: "addrvec": [ 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: "type": "v2", 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: "addr": "172.21.15.22:0", 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: "nonce": 4133023816 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.183 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.184 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:40.184 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T06:50:40.184 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:38 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2255778917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:38 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1781542745' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:39 smithi143 ceph-mon[36520]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:38 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2255778917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:38 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1781542745' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T06:50:40.186 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:39 smithi155 ceph-mon[36262]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 msgr2=0x7f0c6c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 0x7f0c6c0a4770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0c88101670 tx=0x7f0c74005f90 comp rx=0 tx=0).stop 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 msgr2=0x7f0c8819e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0c78000c00 tx=0x7f0c7802c5e0 comp rx=0 tx=0).stop 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 shutdown_connections 2024-03-24T06:50:40.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0c6c0a22b0 0x7f0c6c0a4770 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0c88112390 0x7f0c881a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c88076560 0x7f0c8819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 --2- 172.21.15.22:0/1781542745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0c88075c40 0x7f0c8819e240 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 >> 172.21.15.22:0/1781542745 conn(0x7f0c880ff1a0 msgr2=0x7f0c88113a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 shutdown_connections 2024-03-24T06:50:40.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:50:38.201+0000 7f0c87e00640 1 -- 172.21.15.22:0/1781542745 wait complete. 2024-03-24T06:50:40.196 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T06:50:40.196 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T06:50:40.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:38 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2255778917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:50:40.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:38 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1781542745' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T06:50:40.197 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:39 smithi022 ceph-mon[32003]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:40.660 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:50:42.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:42 smithi022 ceph-mon[32003]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:42.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:42 smithi143 ceph-mon[36520]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:42.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:42 smithi155 ceph-mon[36262]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:43.072 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:50:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 3481, 2024-03-24T06:50:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 4216, 2024-03-24T06:50:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 2860938, 2024-03-24T06:50:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 5991044, 2024-03-24T06:50:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 12, 2024-03-24T06:50:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 4, 2024-03-24T06:50:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.709506549, 2024-03-24T06:50:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.284430984, 2024-03-24T06:50:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.324916277, 2024-03-24T06:50:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.103375325, 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 4216, 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 1.525667170, 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000361875 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 4968, 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.001485041, 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000298 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 1846016, 2024-03-24T06:50:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 494192 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 7873, 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 9754, 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 2725907, 2024-03-24T06:50:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 7637729, 2024-03-24T06:50:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 31, 2024-03-24T06:50:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 5, 2024-03-24T06:50:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 2.710908712, 2024-03-24T06:50:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 1.937270741, 2024-03-24T06:50:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.712675973, 2024-03-24T06:50:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.190752348, 2024-03-24T06:50:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 9751, 2024-03-24T06:50:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 4.582176703, 2024-03-24T06:50:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000469918 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 7619, 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.003133106, 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000411 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 404272, 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 270112 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 4285, 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 4480, 2024-03-24T06:50:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 1688204, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 7132155, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 130, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 9, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 1.005787028, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.309235755, 2024-03-24T06:50:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.482341357, 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.279242241, 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 4478, 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 1.092003050, 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000243859 2024-03-24T06:50:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 6191, 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.001767284, 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000285 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 42368, 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 976 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "cct": { 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "total_workers": 1, 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "unhealthy_workers": 0 2024-03-24T06:50:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "finisher-MDSRank": { 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "queue_len": 0, 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "complete_latency": { 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 1740, 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 4.619060242, 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.002654632 2024-03-24T06:50:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "finisher-PurgeQueue": { 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "queue_len": 0, 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "complete_latency": { 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 847, 2024-03-24T06:50:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.148240331, 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000175018 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "request": 2038, 2024-03-24T06:50:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "reply": 2038, 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "reply_latency": { 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 2038, 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 23.738264290, 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.011647823 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "slow_reply": 0, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "forward": 0, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_fetch_complete": 2, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_fetch_keys": 0, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_commit": 12, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_split": 0, 2024-03-24T06:50:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_merge": 0, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes": 204, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_top": 4, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_bottom": 98, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_pin_tail": 102, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_pinned": 198, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_expired": 746, 2024-03-24T06:50:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_with_caps": 12, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "caps": 12, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "subtrees": 10, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse": 2730, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_hit": 2563, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_forward": 0, 2024-03-24T06:50:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_discover": 2, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_remote_ino": 0, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_lock": 45, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "load_cent": 0, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "q": 0, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "exported": 6, 2024-03-24T06:50:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "exported_inodes": 725, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "imported": 6, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "imported_inodes": 308, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_dir_fetch": 0, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_backtrace_fetch": 1052, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_peer_discover": 0, 2024-03-24T06:50:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rfiles": 324, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rbytes": 13424234227, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rsnaps": 6, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_backtrace_fetch": 1052, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_set_tag": 0, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:50:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dir_inodes": 225, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dir_base_inodes": 16, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dirfrag_rstats": 209, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_file_inodes": 827, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_revoke": 226, 2024-03-24T06:50:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_grant": 1708, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_trunc": 6, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_caps": 3000, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_caps_dirty": 2375, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_cap_release": 146, 2024-03-24T06:50:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "process_request_cap_release": 917 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_cache": { 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_update": 319, 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_update_receipt": 12, 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_try_discover": 6, 2024-03-24T06:50:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_send_discover": 12, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_handle_discover": 637, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays": 0, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays_delayed": 0, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_created": 530, 2024-03-24T06:50:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_enqueued": 469, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_reintegrated": 61, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_migrated": 0, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_processing": 0, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "recovery_started": 0, 2024-03-24T06:50:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "recovery_completed": 0, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_enqueue_scrub": 16, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_exportdir": 6, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_flush": 0, 2024-03-24T06:50:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_fragstats": 0, 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_inodestats": 0 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_log": { 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "evlrg": 0, 2024-03-24T06:50:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "evadd": 4906, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "evex": 0, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "evtrm": 0, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "ev": 4906, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "evexg": 0, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "evexd": 0, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "segadd": 8, 2024-03-24T06:50:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "segex": 0, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "segtrm": 0, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "seg": 8, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "segmjr": 2, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "segexg": 0, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "segexd": 0, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "expos": 4194304, 2024-03-24T06:50:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "wrpos": 14113743, 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "rdpos": 0, 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "jlat": { 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 564, 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 18.251304044, 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.032360468 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "replayed": 0 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_mem": { 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "ino": 146, 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "ino+": 1116, 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "ino-": 970, 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "dir": 27, 2024-03-24T06:50:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "dir+": 198, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "dir-": 171, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "dn": 204, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "dn+": 1493, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "dn-": 1289, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "cap": 12, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "cap+": 976, 2024-03-24T06:50:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "cap-": 964, 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "rss": 65728, 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "heap": 231804 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_server": { 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "dispatch_client_request": 2459, 2024-03-24T06:50:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "dispatch_server_request": 5, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_request": 2038, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_session": 40, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_peer_request": 14, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "req_create_latency": { 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 643, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 1.223684679, 2024-03-24T06:50:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.001903086 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getattr_latency": { 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 174, 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.056830789, 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000326613 2024-03-24T06:50:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getfilelock_latency": { 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "req_link_latency": { 2024-03-24T06:50:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 61, 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.103571048, 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.001697886 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookup_latency": { 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 139, 2024-03-24T06:50:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.128368640, 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000923515 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookuphash_latency": { 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupino_latency": { 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupname_latency": { 2024-03-24T06:50:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupparent_latency": { 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupsnap_latency": { 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lssnap_latency": { 2024-03-24T06:50:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 10, 2024-03-24T06:50:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.004903929, 2024-03-24T06:50:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000490392 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mkdir_latency": { 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 105, 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.219287691, 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.002088454 2024-03-24T06:50:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mknod_latency": { 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mksnap_latency": { 2024-03-24T06:50:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 10, 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.128800485, 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.012880048 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "req_open_latency": { 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 89, 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.169185322, 2024-03-24T06:50:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.001900958 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "req_readdir_latency": { 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 160, 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 1.551656251, 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.009697851 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rename_latency": { 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 113, 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 18.411091361, 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.162930012 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "req_renamesnap_latency": { 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "req_snapdiff_latency": { 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmdir_latency": { 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 16, 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.366989703, 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.022936856 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmsnap_latency": { 2024-03-24T06:50:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 4, 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.026331566, 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.006582891 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmxattr_latency": { 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setattr_latency": { 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 8, 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.072963404, 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.009120425 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setdirlayout_latency": { 2024-03-24T06:50:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setfilelock_latency": { 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 4, 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.002028586, 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000507146 2024-03-24T06:50:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setlayout_latency": { 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setxattr_latency": { 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 8, 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.032758739, 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.004094842 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "req_symlink_latency": { 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 13, 2024-03-24T06:50:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.018899200, 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.001453784 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "req_unlink_latency": { 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 481, 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 1.220912897, 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.002538280 2024-03-24T06:50:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getvxattr_latency": { 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_sessions": { 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "session_count": 2, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "session_add": 6, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "session_remove": 4, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "sessions_open": 2, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "sessions_stale": 0, 2024-03-24T06:50:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "total_load": 32, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "average_load": 16, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "avg_session_uptime": 613, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "mdthresh_evicted": 0 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "mempool": { 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_items": 0, 2024-03-24T06:50:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:50:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_items": 0, 2024-03-24T06:50:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_items": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:50:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_items": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_items": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_bytes": 0, 2024-03-24T06:50:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_items": 0, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_bytes": 330584, 2024-03-24T06:50:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_items": 192, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_bytes": 1144, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_items": 13, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_bytes": 0, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_items": 0, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_items": 0, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:50:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_items": 0, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_bytes": 9416, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_items": 180, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_bytes": 0, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_items": 0, 2024-03-24T06:50:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_bytes": 536966, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_items": 4353, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_bytes": 0, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_items": 0, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_bytes": 0, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_items": 0 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "objecter": { 2024-03-24T06:50:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "op_active": 0, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_laggy": 0, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_send": 3308, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_send_bytes": 10093313, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_resend": 37, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_reply": 3308, 2024-03-24T06:50:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "op_latency": { 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 3308, 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 127.679298923, 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.038597127 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "op_inflight": 0, 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "oplen_avg": { 2024-03-24T06:50:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 3308, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 3451 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "op": 3308, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "op_r": 1112, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "op_w": 2196, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "op_rmw": 0, 2024-03-24T06:50:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "op_pg": 0, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_stat": 0, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_create": 11, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_read": 42, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_write": 590, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_writefull": 51, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_writesame": 0, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_append": 0, 2024-03-24T06:50:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_zero": 0, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_truncate": 0, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_delete": 1423, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_mapext": 0, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_sparse_read": 0, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_clonerange": 0, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_getxattr": 1068, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_setxattr": 22, 2024-03-24T06:50:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_rmxattr": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_call": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_watch": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_notify": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_pgls": 0, 2024-03-24T06:50:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_other": 6, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_active": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_send": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_resend": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_ping": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_active": 0, 2024-03-24T06:50:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_send": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_resend": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_active": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_send": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_resend": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_active": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_send": 0, 2024-03-24T06:50:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_resend": 0, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "command_active": 0, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "command_send": 0, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "command_resend": 0, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "map_epoch": 102, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "map_full": 0, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "map_inc": 18, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_sessions": 12, 2024-03-24T06:50:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_session_open": 12, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_session_close": 0, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_laggy": 0, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_wr": 171, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_rd": 4, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_del": 63 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "oft": { 2024-03-24T06:50:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_objs": 1, 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_updates": 0, 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_removes": 0 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "osd-slow-ops": { 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "slow_ops_count": 0 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "purge_queue": { 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_ops": 0, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing": 0, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_high_water": 7, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executed_ops": 909, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executed": 469, 2024-03-24T06:50:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_item_in_journal": 0 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 15635, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 5890922, 2024-03-24T06:50:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 15635, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 15635, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 5890922, 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-objecter_bytes": { 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 3308, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 10144170, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 3308, 2024-03-24T06:50:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 10144170, 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-objecter_ops": { 2024-03-24T06:50:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 1024, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 3308, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 3308, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 3308, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 3308, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 3758096384, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 469, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 50889, 2024-03-24T06:50:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 469, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 26, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 50889, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-write_buf_throttle-0x557c80136500": { 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 3758096384, 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 4906, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 9919439, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 4906, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 564, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 9919439, 2024-03-24T06:50:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:43.165 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:44.131 INFO:tasks.check_counter:Daemon mds.a mds.exported=6 2024-03-24T06:50:44.131 INFO:tasks.check_counter:Daemon mds.a mds.imported=6 2024-03-24T06:50:44.131 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=319 2024-03-24T06:50:44.131 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=12 2024-03-24T06:50:44.132 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-03-24T06:50:44.132 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=10 2024-03-24T06:50:44.132 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=4 2024-03-24T06:50:44.132 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-24T06:50:44.132 DEBUG:tasks.check_counter:Getting stats from d 2024-03-24T06:50:44.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-24T06:50:44.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:44 smithi022 ceph-mon[32003]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:44.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:44 smithi143 ceph-mon[36520]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:44.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:44 smithi155 ceph-mon[36262]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:44.599 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:50:46.359 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:46 smithi022 ceph-mon[32003]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:46.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:46 smithi143 ceph-mon[36520]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:46.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:46 smithi155 ceph-mon[36262]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:46.971 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:46.971 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:50:46.971 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 0, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 2, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 0, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 0, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 4, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 0, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.004363847, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.000963397, 2024-03-24T06:50:46.972 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:46.973 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 242, 2024-03-24T06:50:46.974 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 230, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 1711040, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 141984, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 2, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 1, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.089441437, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.031445293, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.065606272, 2024-03-24T06:50:46.975 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.009550143, 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 229, 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.025438567, 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000111085 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:46.976 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 1711040, 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 141984 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 6, 2024-03-24T06:50:46.977 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 179, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 3344, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 186976, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 5, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 3, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.032267958, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.023104893, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.004199733, 2024-03-24T06:50:46.978 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 177, 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.032714695, 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000184828 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:46.979 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 186976 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "cct": { 2024-03-24T06:50:46.980 INFO:teuthology.orchestra.run.smithi022.stdout: "total_workers": 0, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "unhealthy_workers": 0 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "mempool": { 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_items": 0, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:50:46.981 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:50:46.982 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_items": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_items": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:50:46.983 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_items": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:50:46.984 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_items": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_bytes": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_items": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:50:46.985 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_bytes": 148714, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_items": 30, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_items": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_bytes": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_items": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_items": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:50:46.986 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_items": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_bytes": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_items": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_bytes": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_items": 0, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_bytes": 928, 2024-03-24T06:50:46.987 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_items": 2, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_bytes": 0, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_items": 0, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_bytes": 0, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_items": 0 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:46.988 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 244, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 1676895, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 244, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 244, 2024-03-24T06:50:46.989 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 1676895, 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:46.990 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-24T06:50:47.875 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-24T06:50:47.876 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-24T06:50:47.876 DEBUG:tasks.check_counter:Getting stats from g 2024-03-24T06:50:47.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-24T06:50:48.383 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:50:48.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:48 smithi022 ceph-mon[32003]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:48.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:48 smithi143 ceph-mon[36520]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:48.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:48 smithi155 ceph-mon[36262]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:50.372 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:50 smithi022 ceph-mon[32003]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:50.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:50 smithi143 ceph-mon[36520]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:50.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:50 smithi155 ceph-mon[36262]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:50:50.688 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 242, 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 228, 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 1712976, 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 139872, 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 4, 2024-03-24T06:50:50.689 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 0, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.102240132, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.037538088, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.068154854, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.011059930, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 228, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.056399686, 2024-03-24T06:50:50.690 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000247367 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 1712976, 2024-03-24T06:50:50.691 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 139872 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 2, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 173, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 320, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 185824, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 5, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 3, 2024-03-24T06:50:50.692 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.030892966, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.021467781, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.004082630, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 170, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.027899131, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000164112 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.693 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 185824 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:50:50.694 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 0, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 2, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 0, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 0, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 2, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 1, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.000954850, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.000745638, 2024-03-24T06:50:50.695 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:50.696 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "cct": { 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "total_workers": 0, 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "unhealthy_workers": 0 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: "mempool": { 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_items": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:50:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_items": 0, 2024-03-24T06:50:50.699 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_items": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:50:50.700 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_items": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_items": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_bytes": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_items": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:50:50.701 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_bytes": 148714, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_items": 30, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_items": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_bytes": 0, 2024-03-24T06:50:50.702 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_items": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_items": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_items": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_bytes": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_items": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:50:50.703 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_bytes": 0, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_items": 0, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_bytes": 928, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_items": 2, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_bytes": 0, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_items": 0, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_bytes": 0, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_items": 0 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:50.704 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 240, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 1676375, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 240, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:50.705 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 240, 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 1676375, 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:50.706 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:50.707 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:51.671 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-24T06:50:51.671 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-24T06:50:51.672 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-24T06:50:51.672 DEBUG:tasks.check_counter:Getting stats from j 2024-03-24T06:50:51.672 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-24T06:50:52.162 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T06:50:52.398 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:52 smithi022 ceph-mon[32003]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:52.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:52 smithi143 ceph-mon[36520]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:52.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:52 smithi155 ceph-mon[36262]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:54.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:54 smithi022 ceph-mon[32003]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:54.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:54 smithi143 ceph-mon[36520]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 273, 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 252, 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 1799698, 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 147325, 2024-03-24T06:50:54.499 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 5, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 1, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.100430945, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.023237071, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.093641853, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.011026355, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 251, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.034325021, 2024-03-24T06:50:54.500 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000136753 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 1798864, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 147008 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.501 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 589, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 763, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 146784, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 642358, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 7, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 5, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.159714050, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.075135570, 2024-03-24T06:50:54.502 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.030161734, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.039608132, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 761, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.109227224, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000143531 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.503 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 461040 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_messages": 1659, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages": 1661, 2024-03-24T06:50:54.504 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_bytes": 2306368, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_bytes": 516007, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_created_connections": 4, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_active_connections": 3, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_total_time": 0.345098394, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_send_time": 0.131579743, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_recv_time": 0.081065997, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_running_fast_dispatch_time": 0.099895472, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 1659, 2024-03-24T06:50:54.505 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.215438971, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000129860 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:50:54.506 INFO:teuthology.orchestra.run.smithi022.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "cct": { 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "total_workers": 1, 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "unhealthy_workers": 0 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "finisher-MDSRank": { 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "queue_len": 0, 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "complete_latency": { 2024-03-24T06:50:54.507 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 3869, 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.184238689, 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000047619 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "finisher-PurgeQueue": { 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "queue_len": 0, 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "complete_latency": { 2024-03-24T06:50:54.508 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "request": 0, 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "reply": 0, 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "reply_latency": { 2024-03-24T06:50:54.509 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "slow_reply": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "forward": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_fetch_complete": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_fetch_keys": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_commit": 0, 2024-03-24T06:50:54.510 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_split": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_merge": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes": 96, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_top": 4, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_bottom": 92, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_pin_tail": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_pinned": 91, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_expired": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "inodes_with_caps": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "caps": 0, 2024-03-24T06:50:54.511 INFO:teuthology.orchestra.run.smithi022.stdout: "subtrees": 4, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_hit": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_forward": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_discover": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_remote_ino": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "traverse_lock": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "load_cent": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "q": 0, 2024-03-24T06:50:54.512 INFO:teuthology.orchestra.run.smithi022.stdout: "exported": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "exported_inodes": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "imported": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "imported_inodes": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_dir_fetch": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_backtrace_fetch": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "openino_peer_discover": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rfiles": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rbytes": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "root_rsnaps": 0, 2024-03-24T06:50:54.513 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_set_tag": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "scrub_file_inodes": 0, 2024-03-24T06:50:54.514 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_grant": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_caps": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_caps_dirty": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_cap_release": 0, 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: "process_request_cap_release": 0 2024-03-24T06:50:54.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_cache": { 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_update": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_update_receipt": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_try_discover": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_send_discover": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "dir_handle_discover": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays_delayed": 0, 2024-03-24T06:50:54.516 INFO:teuthology.orchestra.run.smithi022.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_created": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_enqueued": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_reintegrated": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "strays_migrated": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_processing": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "recovery_started": 0, 2024-03-24T06:50:54.517 INFO:teuthology.orchestra.run.smithi022.stdout: "recovery_completed": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_exportdir": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_flush": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_fragstats": 0, 2024-03-24T06:50:54.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ireq_inodestats": 0 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_log": { 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evlrg": 0, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evadd": 0, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evex": 0, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evtrm": 0, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "ev": 1475, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evexg": 0, 2024-03-24T06:50:54.519 INFO:teuthology.orchestra.run.smithi022.stdout: "evexd": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segadd": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segex": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segtrm": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "seg": 3, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segmjr": 2, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segexg": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "segexd": 0, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "expos": 4194304, 2024-03-24T06:50:54.520 INFO:teuthology.orchestra.run.smithi022.stdout: "wrpos": 6008931, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "rdpos": 6008357, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "jlat": { 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "replayed": 1475 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.521 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_mem": { 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "ino": 99, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "ino+": 99, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "ino-": 0, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dir": 11, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dir+": 11, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dir-": 0, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dn": 96, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dn+": 96, 2024-03-24T06:50:54.522 INFO:teuthology.orchestra.run.smithi022.stdout: "dn-": 0, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "cap": 0, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "cap+": 0, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "cap-": 0, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "rss": 50928, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "heap": 215420 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_server": { 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "dispatch_client_request": 0, 2024-03-24T06:50:54.523 INFO:teuthology.orchestra.run.smithi022.stdout: "dispatch_server_request": 0, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_request": 0, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_client_session": 0, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "handle_peer_request": 0, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "req_create_latency": { 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getattr_latency": { 2024-03-24T06:50:54.524 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getfilelock_latency": { 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.525 INFO:teuthology.orchestra.run.smithi022.stdout: "req_link_latency": { 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookup_latency": { 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookuphash_latency": { 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupino_latency": { 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.527 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupname_latency": { 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.528 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupparent_latency": { 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lookupsnap_latency": { 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.529 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "req_lssnap_latency": { 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mkdir_latency": { 2024-03-24T06:50:54.530 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.531 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:54 smithi155 ceph-mon[36262]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:54.531 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mknod_latency": { 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.532 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "req_mksnap_latency": { 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "req_open_latency": { 2024-03-24T06:50:54.533 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: "req_readdir_latency": { 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.534 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rename_latency": { 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.535 INFO:teuthology.orchestra.run.smithi022.stdout: "req_renamesnap_latency": { 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "req_snapdiff_latency": { 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmdir_latency": { 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmsnap_latency": { 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "req_rmxattr_latency": { 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setattr_latency": { 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.538 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setdirlayout_latency": { 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.539 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setfilelock_latency": { 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setlayout_latency": { 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.540 INFO:teuthology.orchestra.run.smithi022.stdout: "req_setxattr_latency": { 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "req_symlink_latency": { 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "req_unlink_latency": { 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "req_getvxattr_latency": { 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.542 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_sessions": { 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "session_count": 2, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "session_add": 2, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "session_remove": 0, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "sessions_open": 2, 2024-03-24T06:50:54.543 INFO:teuthology.orchestra.run.smithi022.stdout: "sessions_stale": 0, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "total_load": 0, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "average_load": 0, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "avg_session_uptime": 1988, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "mdthresh_evicted": 0 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "mempool": { 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "bloom_filter_items": 0, 2024-03-24T06:50:54.544 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:50:54.545 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_extent_items": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_blob_items": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:50:54.546 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_txc_items": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:50:54.547 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluestore_writing_items": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_bytes": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_items": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_bytes": 195717, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_anon_items": 149, 2024-03-24T06:50:54.548 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "buffer_meta_items": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_bytes": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_items": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_mapbl_items": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_pglog_items": 0, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_bytes": 9416, 2024-03-24T06:50:54.549 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_items": 180, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_bytes": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "pgmap_items": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_bytes": 312603, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "mds_co_items": 2654, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_bytes": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_1_items": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_bytes": 0, 2024-03-24T06:50:54.550 INFO:teuthology.orchestra.run.smithi022.stdout: "unittest_2_items": 0 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "objecter": { 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_active": 0, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_laggy": 0, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_send": 2243, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_send_bytes": 21, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_resend": 3, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_reply": 2243, 2024-03-24T06:50:54.551 INFO:teuthology.orchestra.run.smithi022.stdout: "op_latency": { 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 2243, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 13.716818062, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.006115389 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "op_inflight": 0, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "oplen_avg": { 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 2243, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 2244 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: "op": 2243, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "op_r": 2243, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "op_w": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "op_rmw": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "op_pg": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_stat": 1369, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_create": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_read": 872, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_write": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_writefull": 0, 2024-03-24T06:50:54.553 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_writesame": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_append": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_zero": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_truncate": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_delete": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_mapext": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_sparse_read": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_clonerange": 0, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_getxattr": 1, 2024-03-24T06:50:54.554 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_setxattr": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_rmxattr": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_call": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_watch": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_notify": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_pgls": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:50:54.555 INFO:teuthology.orchestra.run.smithi022.stdout: "osdop_other": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_active": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_send": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_resend": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "linger_ping": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_active": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_send": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolop_resend": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_active": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_send": 0, 2024-03-24T06:50:54.556 INFO:teuthology.orchestra.run.smithi022.stdout: "poolstat_resend": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_active": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_send": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "statfs_resend": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "command_active": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "command_send": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "command_resend": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "map_epoch": 102, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "map_full": 0, 2024-03-24T06:50:54.557 INFO:teuthology.orchestra.run.smithi022.stdout: "map_inc": 13, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_sessions": 5, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_session_open": 5, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_session_close": 0, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "osd_laggy": 0, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_wr": 0, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_rd": 2, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_del": 0 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.558 INFO:teuthology.orchestra.run.smithi022.stdout: "oft": { 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_objs": 0, 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_updates": 0, 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "omap_total_removes": 0 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "osd-slow-ops": { 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "slow_ops_count": 0 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.559 INFO:teuthology.orchestra.run.smithi022.stdout: "purge_queue": { 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_ops": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executing_high_water": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executed_ops": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_executed": 0, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "pq_item_in_journal": 0 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:50:54.560 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 2517, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 4009358, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 2517, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 2517, 2024-03-24T06:50:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 4009358, 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-objecter_bytes": { 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:54.562 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 104857600, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 2243, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 1814632, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 2243, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 1814632, 2024-03-24T06:50:54.563 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-objecter_ops": { 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 1024, 2024-03-24T06:50:54.564 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 2243, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 2243, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 2243, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 2243, 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:54.565 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 3758096384, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:54.566 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.567 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "throttle-write_buf_throttle-0x56361f80a780": { 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "val": 0, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "max": 3758096384, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "get_started": 0, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "get": 0, 2024-03-24T06:50:54.568 INFO:teuthology.orchestra.run.smithi022.stdout: "get_sum": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "take": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "take_sum": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "put": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "put_sum": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "wait": { 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "avgcount": 0, 2024-03-24T06:50:54.569 INFO:teuthology.orchestra.run.smithi022.stdout: "sum": 0.000000000, 2024-03-24T06:50:54.570 INFO:teuthology.orchestra.run.smithi022.stdout: "avgtime": 0.000000000 2024-03-24T06:50:54.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T06:50:54.570 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T06:50:55.493 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-03-24T06:50:55.493 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update_receipt=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds.root_rsnaps=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:50:55.494 INFO:tasks.check_counter:Daemon mds.j mds.dir_split=0 2024-03-24T06:50:55.494 DEBUG:tasks.check_counter:Getting stats from b 2024-03-24T06:50:55.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-24T06:50:55.944 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:50:56.337 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:56 smithi143 ceph-mon[36520]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:56.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:56 smithi022 ceph-mon[32003]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:56.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:56 smithi155 ceph-mon[36262]: pgmap v340: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:50:57.819 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T06:50:57.820 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:50:57.820 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 4827, 2024-03-24T06:50:57.820 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 7013, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 1540306, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 11368026, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 20, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 6, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.331756419, 2024-03-24T06:50:57.821 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.140614161, 2024-03-24T06:50:57.822 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.184762804, 2024-03-24T06:50:57.822 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.038306426, 2024-03-24T06:50:57.822 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:57.822 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 7007, 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.387564494, 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000055311 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4140, 2024-03-24T06:50:57.823 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.001429793, 2024-03-24T06:50:57.824 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000345 2024-03-24T06:50:57.824 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.824 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:50:57.824 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:50:57.824 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 2704, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 2954, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 3291649, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 1534807, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 9, 2024-03-24T06:50:57.825 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 6, 2024-03-24T06:50:57.826 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.178301643, 2024-03-24T06:50:57.826 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.065326668, 2024-03-24T06:50:57.826 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.125400280, 2024-03-24T06:50:57.826 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.014123557, 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2957, 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.124158175, 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000041987 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.827 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4008, 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000688709, 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000171 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1829008, 2024-03-24T06:50:57.828 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 159664 2024-03-24T06:50:57.829 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.829 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:50:57.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 843, 2024-03-24T06:50:57.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 1243, 2024-03-24T06:50:57.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 300522, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 1402781, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 41, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 10, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.076304389, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.026282785, 2024-03-24T06:50:57.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.034213500, 2024-03-24T06:50:57.831 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.013386303, 2024-03-24T06:50:57.831 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:50:57.831 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1240, 2024-03-24T06:50:57.831 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.094855744, 2024-03-24T06:50:57.831 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000076496 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 766, 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000463985, 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000605 2024-03-24T06:50:57.832 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 4432, 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 543744 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T06:50:57.833 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3076, 2024-03-24T06:50:57.834 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.232080767, 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000400546 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:50:57.835 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2, 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000007808, 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000003904 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.836 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 216, 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 192, 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 192, 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.211823137, 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001103245 2024-03-24T06:50:57.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.838 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T06:50:57.838 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 24, 2024-03-24T06:50:57.838 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 1, 2024-03-24T06:50:57.838 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T06:50:57.838 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 20, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 1, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 216, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 59, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 94, 2024-03-24T06:50:57.839 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 63, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 132, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 749, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 11, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 11, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 11, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 447, 2024-03-24T06:50:57.840 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 329, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 24, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 94, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T06:50:57.841 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 12, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 811, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 6, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 725, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T06:50:57.842 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 882, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 324, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 13424234227, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 6, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 882, 2024-03-24T06:50:57.843 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 216, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 216, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 666, 2024-03-24T06:50:57.844 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 4, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 51, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 1095, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 2686, 2024-03-24T06:50:57.845 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 2515, 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 7, 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 11, 2024-03-24T06:50:57.846 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 249, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 170, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 159, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 46, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:50:57.847 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:50:57.848 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:50:57.849 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 13, 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 1, 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.850 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T06:50:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T06:50:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 2936, 2024-03-24T06:50:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T06:50:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T06:50:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 2936, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 5, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 5, 2024-03-24T06:50:57.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 2, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 13226649, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 0, 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T06:50:57.853 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1127, 2024-03-24T06:50:57.854 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 36.058012169, 2024-03-24T06:50:57.854 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.031994686 2024-03-24T06:50:57.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.854 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 0 2024-03-24T06:50:57.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 218, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 723, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 505, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 100, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 201, 2024-03-24T06:50:57.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 101, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 216, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 965, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 749, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 11, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 542, 2024-03-24T06:50:57.856 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 531, 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 58740, 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 223612 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 216, 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 16, 2024-03-24T06:50:57.857 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 216, 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 25, 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 36, 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 109, 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.114490394, 2024-03-24T06:50:57.858 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001050370 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.859 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T06:50:57.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 60, 2024-03-24T06:50:57.861 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.056520120, 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000942002 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1, 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000166116, 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000166116 2024-03-24T06:50:57.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T06:50:57.864 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.866 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T06:50:57.866 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.867 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T06:50:57.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 22, 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.040646507, 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001847568 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.872 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T06:50:57.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.875 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T06:50:57.876 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.877 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.878 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.879 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.880 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 2, 2024-03-24T06:50:57.881 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 1, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 1, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 42, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 42, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 657, 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T06:50:57.882 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:50:57.883 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:50:57.884 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:50:57.885 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:50:57.886 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:50:57.886 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:50:57.886 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:50:57.886 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:50:57.886 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T06:50:57.887 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 295622, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 262, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 880, 2024-03-24T06:50:57.888 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 10, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T06:50:57.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9416, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 180, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 838874, 2024-03-24T06:50:57.890 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 9041, 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T06:50:57.891 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 2189, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 9255061, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 27, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 2189, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2189, 2024-03-24T06:50:57.892 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 181.756298822, 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.083031657 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2189, 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2369 2024-03-24T06:50:57.893 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 2189, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 883, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 1306, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 0, 2024-03-24T06:50:57.894 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 10, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 0, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 1127, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 27, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T06:50:57.895 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 1, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 8, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 882, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 20, 2024-03-24T06:50:57.896 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:50:57.897 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T06:50:57.898 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T06:50:57.899 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T06:50:57.900 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 12, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 12, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 210, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T06:50:57.901 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 82 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 1, 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T06:50:57.902 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:50:57.903 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:50:57.904 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 8370, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 4407226, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 8370, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:50:57.905 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 8370, 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 4407226, 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.906 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:57.907 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2189, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 9255045, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2189, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 9255045, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.908 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:50:57.909 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2189, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2189, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2189, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2189, 2024-03-24T06:50:57.910 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:50:57.911 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:50:57.912 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55d5f13b6500": { 2024-03-24T06:50:57.913 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2936, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 9032345, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2936, 2024-03-24T06:50:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 1127, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 9032345, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:50:57.915 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:50:57.916 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.916 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:50:57.916 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T06:50:58.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:50:58 smithi022 ceph-mon[32003]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:58.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:50:58 smithi143 ceph-mon[36520]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:58.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:50:58 smithi155 ceph-mon[36262]: pgmap v341: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-24T06:50:58.702 INFO:tasks.check_counter:Daemon mds.b mds.exported=12 2024-03-24T06:50:58.702 INFO:tasks.check_counter:Daemon mds.b mds.imported=6 2024-03-24T06:50:58.702 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=11 2024-03-24T06:50:58.702 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=249 2024-03-24T06:50:58.703 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-24T06:50:58.703 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:50:58.703 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:50:58.703 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-03-24T06:50:58.703 DEBUG:tasks.check_counter:Getting stats from e 2024-03-24T06:50:58.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-24T06:50:58.952 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:51:00.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:00 smithi022 ceph-mon[32003]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:51:00.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:00 smithi143 ceph-mon[36520]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:51:00.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:00 smithi155 ceph-mon[36262]: pgmap v342: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T06:51:00.764 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 2021, 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 2022, 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 9613726, 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 628227, 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 6, 2024-03-24T06:51:00.765 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 2, 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.131933388, 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.037573225, 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.040485591, 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.044041588, 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:00.766 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2020, 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.050091552, 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000024797 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.767 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:00.768 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 343, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 520, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 84184, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 646138, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 8, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 5, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.031881768, 2024-03-24T06:51:00.769 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.011934791, 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.007607308, 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.008920763, 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 516, 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.038460900, 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000074536 2024-03-24T06:51:00.770 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-24T06:51:00.771 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 540336 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 278, 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 255, 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 1803052, 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 148119, 2024-03-24T06:51:00.772 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 3, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 2, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.041872172, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.010124847, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.040540905, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.004287835, 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:00.773 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 255, 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.011668495, 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000045758 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.774 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1802784, 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 147808 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T06:51:00.775 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4062, 2024-03-24T06:51:00.776 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.078225909, 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000019257 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:00.777 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.778 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 0, 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 0, 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.779 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T06:51:00.780 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 114, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 0, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 114, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 114, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 0, 2024-03-24T06:51:00.781 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 11, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:00.782 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 0, 2024-03-24T06:51:00.783 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 0, 2024-03-24T06:51:00.784 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:00.785 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:00.786 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 0, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 0, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 0, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 0, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 0, 2024-03-24T06:51:00.787 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:00.788 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:00.789 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:00.790 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 0, 2024-03-24T06:51:00.791 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 2936, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T06:51:00.792 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 5, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 2, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 13226649, 2024-03-24T06:51:00.793 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 13222724, 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 2936 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.794 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 115, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 695, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 580, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 21, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 177, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 156, 2024-03-24T06:51:00.795 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 114, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 930, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 816, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 0, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 0, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 0, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 50572, 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 215420 2024-03-24T06:51:00.796 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 0, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 0, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 0, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 0, 2024-03-24T06:51:00.797 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.798 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.799 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.800 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.801 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.802 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.803 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.804 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.805 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.805 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.805 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.806 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.807 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.808 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.809 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.810 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.811 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.812 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.813 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:00.814 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.815 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T06:51:00.816 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.817 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.818 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 2, 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 1, 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 2, 2024-03-24T06:51:00.819 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 0, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 0, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 4398, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:00.820 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:00.821 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:00.822 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:00.823 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:00.824 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 195588, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 180, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T06:51:00.825 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9416, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 180, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:00.826 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 406319, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 3546, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:00.827 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 2361, 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 21, 2024-03-24T06:51:00.828 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 3, 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 2361, 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2361, 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 31.291193925, 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.013253364 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.829 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2361, 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2362 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 2361, 2024-03-24T06:51:00.830 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 2361, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 1420, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 939, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 0, 2024-03-24T06:51:00.831 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:00.832 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 1, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T06:51:00.833 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T06:51:00.834 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T06:51:00.835 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T06:51:00.836 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 5, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 5, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 0, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 0 2024-03-24T06:51:00.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 0, 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.838 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:00.839 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2638, 2024-03-24T06:51:00.840 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 11246498, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2638, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2638, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 11246498, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.841 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:00.842 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2361, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 9032542, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2361, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 9032542, 2024-03-24T06:51:00.843 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:00.844 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2361, 2024-03-24T06:51:00.845 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2361, 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2361, 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2361, 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.846 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:00.847 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.848 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x5647ec6f46e0": { 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:00.849 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:00.850 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:00.851 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:01.682 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-24T06:51:01.682 DEBUG:tasks.check_counter:Getting stats from h 2024-03-24T06:51:01.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-24T06:51:01.929 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:51:02.345 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:02 smithi143 ceph-mon[36520]: pgmap v343: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:51:02.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:02 smithi022 ceph-mon[32003]: pgmap v343: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:51:02.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:02 smithi155 ceph-mon[36262]: pgmap v343: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T06:51:03.828 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T06:51:03.828 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 947, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 949, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 4492805, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 294527, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 6, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 2, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.066325954, 2024-03-24T06:51:03.829 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.022720390, 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.017010371, 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.019908836, 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 947, 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.024638699, 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000026017 2024-03-24T06:51:03.830 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:51:03.831 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 864, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 1038, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 1756815, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 808817, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 8, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 5, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.057668572, 2024-03-24T06:51:03.832 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.016816079, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.016027703, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.019871632, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1035, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.042815194, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000041367 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.833 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 542224 2024-03-24T06:51:03.834 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 637, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 615, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 6085161, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 259420, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 3, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 2, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.065689831, 2024-03-24T06:51:03.835 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.016402035, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.048027848, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.012648233, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 614, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.015630131, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000025456 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.836 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1798480, 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 145904 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.837 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3633, 2024-03-24T06:51:03.838 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.069663165, 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000019175 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.839 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 0, 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 0, 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T06:51:03.840 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 0, 2024-03-24T06:51:03.841 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 665, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 4, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 661, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 660, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 0, 2024-03-24T06:51:03.842 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 5, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:03.843 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 0, 2024-03-24T06:51:03.844 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 0, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 324, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 13424234227, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 6, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:03.845 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:03.846 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 0, 2024-03-24T06:51:03.847 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 0, 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 0, 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 0, 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 0, 2024-03-24T06:51:03.848 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T06:51:03.849 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:03.850 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T06:51:03.851 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 4913, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T06:51:03.852 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 8, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 2, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 14118841, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 14118757, 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T06:51:03.853 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 4913 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 138, 2024-03-24T06:51:03.854 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 1088, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 950, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 14, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 185, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 171, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 665, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 1424, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 759, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 0, 2024-03-24T06:51:03.855 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 0, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 0, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 51708, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 215420 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 0, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:03.856 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 0, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 0, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 0, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T06:51:03.857 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.858 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.859 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T06:51:03.860 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.861 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.862 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.863 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T06:51:03.864 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.864 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.864 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.864 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.864 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.865 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T06:51:03.866 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.867 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.868 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:03.869 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.870 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.871 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.872 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.873 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.874 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.875 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 2, 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 6, 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 4, 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 4, 2024-03-24T06:51:03.876 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 0, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 0, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 4402, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:03.877 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:03.878 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:03.879 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:03.880 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:03.881 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 195082, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 185, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:03.882 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9416, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 180, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T06:51:03.883 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 822850, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 5924, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T06:51:03.884 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 2169, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 26, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 4, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 2169, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2169, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 132.253587236, 2024-03-24T06:51:03.885 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.060974452 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2169, 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2170 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 2169, 2024-03-24T06:51:03.886 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 2169, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 0, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 1358, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 0, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 808, 2024-03-24T06:51:03.887 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:03.888 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 2, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T06:51:03.889 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T06:51:03.890 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T06:51:03.891 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 5, 2024-03-24T06:51:03.892 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 5, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 0, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 0 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 0, 2024-03-24T06:51:03.893 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T06:51:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:03.895 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2444, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 12097874, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2444, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:03.896 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2444, 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 12097874, 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.897 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:03.898 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2169, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 9925125, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2169, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 9925125, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.899 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:03.900 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2169, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2169, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2169, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2169, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.901 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:03.902 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:03.903 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55bd6af266e0": { 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:03.904 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:03.905 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:03.906 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T06:51:04.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:04 smithi022 ceph-mon[32003]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:51:04.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:04 smithi143 ceph-mon[36520]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:51:04.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:04 smithi155 ceph-mon[36262]: pgmap v344: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-24T06:51:04.638 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-24T06:51:04.638 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:04.639 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-24T06:51:04.639 DEBUG:tasks.check_counter:Getting stats from k 2024-03-24T06:51:04.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-24T06:51:04.879 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T06:51:06.391 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:06 smithi143 ceph-mon[36520]: pgmap v345: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:51:06.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:06 smithi155 ceph-mon[36262]: pgmap v345: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:51:06.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:06 smithi022 ceph-mon[32003]: pgmap v345: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-03-24T06:51:06.669 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T06:51:06.669 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 939, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 918, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 2022720, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 355777, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 5, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 1, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.093388199, 2024-03-24T06:51:06.670 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.024174167, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.055968575, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.020592231, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 917, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.029174064, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000031814 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:06.671 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1799952, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 149584 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 1236, 2024-03-24T06:51:06.672 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 1414, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 2662589, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 849179, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 7, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 5, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.073173303, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.026081225, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.018237866, 2024-03-24T06:51:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.022621601, 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1412, 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.058399662, 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000041359 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.674 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 466592 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 4, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 6, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 1629, 2024-03-24T06:51:06.675 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 1256, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 5, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 4, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.001463667, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.000062097, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.001052862, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.000274891, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4, 2024-03-24T06:51:06.676 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.002528306, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000632076 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:51:06.677 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:06.678 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3265, 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.061454674, 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000018822 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.679 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 0, 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 0, 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.680 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T06:51:06.681 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 133, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 6, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 127, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 125, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 0, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 0, 2024-03-24T06:51:06.682 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 4, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:06.683 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 0, 2024-03-24T06:51:06.684 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:06.685 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 0, 2024-03-24T06:51:06.686 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 0, 2024-03-24T06:51:06.687 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:06.688 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T06:51:06.689 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T06:51:06.690 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 1903, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 3, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 2, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T06:51:06.691 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 6540297, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 6539723, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 1903 2024-03-24T06:51:06.692 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 136, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 136, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 0, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 11, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 11, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 0, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 133, 2024-03-24T06:51:06.693 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 133, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 0, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 0, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 0, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 0, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 47828, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 215420 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T06:51:06.694 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.695 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.696 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.697 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.698 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.699 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.700 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T06:51:06.701 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.702 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.703 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.704 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.705 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T06:51:06.706 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.707 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.708 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T06:51:06.710 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.711 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.712 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 3, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 2, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 3, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T06:51:06.713 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 6279, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:06.714 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:06.715 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:06.716 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T06:51:06.717 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 203106, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 188, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T06:51:06.718 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9416, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 180, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:06.719 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 423520, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 3652, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.720 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 1897, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 21, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 3, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 1897, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1897, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 150.280119863, 2024-03-24T06:51:06.721 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.079219883 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1897, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1898 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 1897, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 1897, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 0, 2024-03-24T06:51:06.722 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 1178, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 717, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T06:51:06.723 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 1, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:06.724 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T06:51:06.725 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T06:51:06.726 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 6, 2024-03-24T06:51:06.727 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 6, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 0, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 0 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T06:51:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 0, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:06.729 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:06.730 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2175, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 4474018, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2175, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2175, 2024-03-24T06:51:06.731 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 4474018, 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:06.733 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 1897, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2345998, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 1897, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2345998, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.734 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:06.735 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 1897, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 1897, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 1897, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 1897, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.736 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:06.737 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.738 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55aa4ae766e0": { 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:06.739 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T06:51:06.740 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.741 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T06:51:06.741 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T06:51:07.693 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-24T06:51:07.693 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-24T06:51:07.693 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-24T06:51:07.693 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-24T06:51:07.694 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-24T06:51:07.694 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:07.694 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:07.694 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-24T06:51:07.694 DEBUG:tasks.check_counter:Getting stats from c 2024-03-24T06:51:07.694 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-24T06:51:08.076 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:51:08.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:08 smithi143 ceph-mon[36520]: pgmap v346: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:51:08.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:08 smithi155 ceph-mon[36262]: pgmap v346: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:51:08.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:08 smithi022 ceph-mon[32003]: pgmap v346: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-03-24T06:51:09.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:09 smithi143 ceph-mon[36520]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T06:51:09.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:09 smithi155 ceph-mon[36262]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T06:51:09.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:09 smithi022 ceph-mon[32003]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T06:51:10.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:10 smithi155 ceph-mon[36262]: pgmap v348: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:10.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:10 smithi143 ceph-mon[36520]: pgmap v348: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:10.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:10 smithi022 ceph-mon[32003]: pgmap v348: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:10.997 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T06:51:10.998 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:10.998 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2785, 2024-03-24T06:51:10.998 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 2512, 2024-03-24T06:51:10.998 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 2633093, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 2135702, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 31, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.347916444, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.075295904, 2024-03-24T06:51:10.999 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.133939322, 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.163732418, 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2517, 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.116963017, 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000046469 2024-03-24T06:51:11.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2771, 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000486547, 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000175 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1823504, 2024-03-24T06:51:11.001 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 162784 2024-03-24T06:51:11.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.002 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:11.002 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1953, 2024-03-24T06:51:11.002 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 2417, 2024-03-24T06:51:11.002 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 576993, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 1294882, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 12, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 8, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.272812278, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.052888668, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.050218004, 2024-03-24T06:51:11.003 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.163422510, 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2413, 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.658940359, 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000273079 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 453, 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000213704, 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000471 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 468768 2024-03-24T06:51:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1366, 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1248, 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 795329, 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 604588, 2024-03-24T06:51:11.006 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 26, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.089748505, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.026785149, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.048675452, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.015981780, 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:11.007 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1242, 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.130572806, 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000105131 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1100, 2024-03-24T06:51:11.008 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000478875, 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000435 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 5872 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-24T06:51:11.009 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:11.010 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1950, 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.825748427, 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000423460 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:11.011 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000006775, 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000003387 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.012 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 20, 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 14, 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 14, 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.006394489, 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000456749 2024-03-24T06:51:11.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 6, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 11, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-24T06:51:11.014 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 477, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 8, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 469, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 466, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 16, 2024-03-24T06:51:11.015 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 157, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 157, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 12, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 216, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 115, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 6, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 95, 2024-03-24T06:51:11.016 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 1000, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 1, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-24T06:51:11.017 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 2, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 439, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 2895, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 324, 2024-03-24T06:51:11.018 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 13424234227, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 6, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 2895, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 829, 2024-03-24T06:51:11.019 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 829, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 2066, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 9, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 11, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:11.020 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 399, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 316, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 5, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.021 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 273, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 182, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 182, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-24T06:51:11.022 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:11.024 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-24T06:51:11.025 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 448, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 448, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 2, 2024-03-24T06:51:11.026 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-24T06:51:11.027 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 5083311, 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 0, 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 94, 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 10.858878076, 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.115519979 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 0 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 338, 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 346, 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 8, 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 109, 2024-03-24T06:51:11.029 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 125, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 16, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 477, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 493, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 16, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 157, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 247, 2024-03-24T06:51:11.030 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 90, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 57096, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 223612 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 20, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 20, 2024-03-24T06:51:11.031 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 18, 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-24T06:51:11.032 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.033 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 12, 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.005599263, 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000466605 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:11.035 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.036 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.037 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.038 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.039 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.039 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-24T06:51:11.039 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.039 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.040 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.040 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-24T06:51:11.040 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-24T06:51:11.042 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000795226, 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000397613 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:11.044 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.046 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.048 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.050 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 2, 2024-03-24T06:51:11.053 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 0, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 2, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 6, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 3, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 602, 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:11.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:11.055 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:11.056 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:11.057 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:11.058 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:11.059 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 320258, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 439, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:11.060 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9416, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 180, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-24T06:51:11.061 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 1456249, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 24298, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-24T06:51:11.062 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 3052, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 931215, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 33, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 3052, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3052, 2024-03-24T06:51:11.063 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 73.917490447, 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.024219361 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3052, 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 3086 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.064 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 3052, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2895, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 157, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 0, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 10, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 0, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 94, 2024-03-24T06:51:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 22, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 6, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 2895, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 20, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-24T06:51:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-24T06:51:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-24T06:51:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 103, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-24T06:51:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 14, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 12, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 12, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 30, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 0, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 9 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 1, 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-24T06:51:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-24T06:51:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 6100, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 3443295, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 6100, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 6100, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 3443295, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 3052, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 931215, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 3052, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 931215, 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-24T06:51:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 3052, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 3052, 2024-03-24T06:51:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 3052, 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 3052, 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x559c0bea4500": { 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 448, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 889007, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 448, 2024-03-24T06:51:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 94, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 889007, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:11.085 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T06:51:12.012 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds.imported=2 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=273 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:12.013 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-24T06:51:12.013 DEBUG:tasks.check_counter:Getting stats from f 2024-03-24T06:51:12.013 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-24T06:51:12.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:12 smithi155 ceph-mon[36262]: pgmap v349: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:51:12.451 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:12 smithi022 ceph-mon[32003]: pgmap v349: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:12.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:12 smithi143 ceph-mon[36520]: pgmap v349: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:14.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:14 smithi155 ceph-mon[36262]: pgmap v350: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2925, 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 2885, 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1055539, 2024-03-24T06:51:14.593 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 3164972, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 17, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 7, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.217341483, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.083852693, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.192861419, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.013379037, 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:14.594 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2877, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.864587634, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000300517 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2319, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.001194937, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000515 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.595 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 2864 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2026, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1689, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 2433390, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 975389, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 36, 2024-03-24T06:51:14.596 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.152263298, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.048623050, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.116210112, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.013880990, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1695, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.103998386, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000061355 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.597 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2649, 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000456122, 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000172 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1820640, 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 164672 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.598 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 426, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 526, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 160393, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 676284, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 16, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.051834031, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.026390145, 2024-03-24T06:51:14.599 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.015500687, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.008546676, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 523, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.069376958, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000132651 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 233, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000035090, 2024-03-24T06:51:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000150 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2000, 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 471264 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.601 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 875, 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.621336094, 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000710098 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:14.602 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000020338, 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000010169 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 116, 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 114, 2024-03-24T06:51:14.603 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 114, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.100959454, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000885609 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 2, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 11, 2024-03-24T06:51:14.604 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 219, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 53, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 166, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 143, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 16, 2024-03-24T06:51:14.605 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 10, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 10, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 12, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 375, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 268, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 2, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 96, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:14.606 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 2, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 35, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 432, 2024-03-24T06:51:14.607 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 324, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 13424234227, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 6, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 432, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:14.608 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 432, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 56, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 233, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:14.609 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 1835, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 1665, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 7, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 8, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 259, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 175, 2024-03-24T06:51:14.610 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 173, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 8, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:14.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-24T06:51:14.612 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 1903, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 1903, 2024-03-24T06:51:14.613 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 3, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 3, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-24T06:51:14.614 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 6540297, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 0, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 234, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 12.409413899, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.053031683 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 0 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.615 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 223, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 231, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 8, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 101, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 113, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 12, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 219, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 235, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 16, 2024-03-24T06:51:14.616 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 10, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 43, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 33, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 60476, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 223612 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 128, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:14.617 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 116, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 16, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 10, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 9, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.017973506, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001997056 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-24T06:51:14.618 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.002528405, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001264202 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.619 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 70, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.032763213, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000468045 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.620 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1, 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000114129, 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000114129 2024-03-24T06:51:14.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.622 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.624 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-24T06:51:14.624 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.624 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.624 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 32, 2024-03-24T06:51:14.626 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.047580201, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001486881 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.627 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.628 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.630 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.632 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.633 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:14.634 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 1, 2024-03-24T06:51:14.635 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 3, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 2, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 1, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 41, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 41, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 657, 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:14.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:14.637 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:14.638 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:14.639 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:14.640 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 296673, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 249, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 792, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 9, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:14.641 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9656, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 186, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-24T06:51:14.642 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 804208, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 5684, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-24T06:51:14.643 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 791, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 2370096, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 33, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 791, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 791, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 128.806365942, 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.162839906 2024-03-24T06:51:14.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 791, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 892 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 791, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 432, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 359, 2024-03-24T06:51:14.645 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 0, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 10, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 0, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 234, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 22, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-24T06:51:14.646 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 6, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 432, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 20, 2024-03-24T06:51:14.647 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:14.648 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-24T06:51:14.649 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-24T06:51:14.650 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 103, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 14, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 11, 2024-03-24T06:51:14.651 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 11, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 115, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 0, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 53 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 1, 2024-03-24T06:51:14.652 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:14.653 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:14.654 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 5373, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 3177423, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 5373, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 5373, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 3177423, 2024-03-24T06:51:14.655 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:14.656 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 791, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2370096, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 791, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2370096, 2024-03-24T06:51:14.657 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-24T06:51:14.658 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 791, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 791, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 791, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 791, 2024-03-24T06:51:14.659 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:14.660 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:14.661 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x55d190d42500": { 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:14.662 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 1903, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 2345993, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 1903, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 234, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2345993, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:14.663 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:14.664 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:14.664 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.664 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:14.664 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T06:51:14.664 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:14 smithi022 ceph-mon[32003]: pgmap v350: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:14.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:14 smithi143 ceph-mon[36520]: pgmap v350: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T06:51:15.638 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds.imported=2 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=8 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=259 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:15.639 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-24T06:51:15.639 DEBUG:tasks.check_counter:Getting stats from i 2024-03-24T06:51:15.640 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-24T06:51:15.897 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:51:16.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:16 smithi155 ceph-mon[36262]: pgmap v351: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:16.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:16 smithi155 ceph-mon[36262]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T06:51:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:16 smithi022 ceph-mon[32003]: pgmap v351: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:16.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:16 smithi022 ceph-mon[32003]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T06:51:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:16 smithi143 ceph-mon[36520]: pgmap v351: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:16.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:16 smithi143 ceph-mon[36520]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T06:51:17.710 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1938, 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1620, 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 683266, 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 2253899, 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 12, 2024-03-24T06:51:17.711 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.134152209, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.046647758, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.084089212, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.017066336, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1622, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.119707313, 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000073802 2024-03-24T06:51:17.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2662, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000527733, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000198 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1620, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1919, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 539309, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 1131556, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 15, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.109932751, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.044863499, 2024-03-24T06:51:17.714 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.080684515, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.003007141, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1916, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.377982496, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000197276 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1200, 2024-03-24T06:51:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000407096, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000339 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2048, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 477664 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1155, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 813, 2024-03-24T06:51:17.716 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 2226514, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 776338, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 41, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 9, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.107807299, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.033024520, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.096909405, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.008270373, 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:17.717 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 809, 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.062489158, 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000077242 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1029, 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000183103, 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000177 2024-03-24T06:51:17.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1831616, 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 162928 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-24T06:51:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 626, 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.461232345, 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000736792 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:17.720 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000009303, 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000004651 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 97, 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 84, 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-24T06:51:17.721 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 84, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.064031475, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000762279 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 13, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 11, 2024-03-24T06:51:17.722 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 182, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 52, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 130, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 108, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 16, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 5, 2024-03-24T06:51:17.723 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 5, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 12, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 332, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 217, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 13, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 99, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-24T06:51:17.724 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 2, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 29, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:17.725 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 312, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 324, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 13424234227, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 6, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 312, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:17.726 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 312, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 34, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 135, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:17.727 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 1399, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 1289, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 8, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 8, 2024-03-24T06:51:17.728 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 264, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 178, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 173, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 8, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-24T06:51:17.729 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:17.730 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-24T06:51:17.731 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 1475, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 1475, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 3, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-24T06:51:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 3, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 6008931, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 0, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 155, 2024-03-24T06:51:17.733 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 3.775747715, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.024359662 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 0 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 186, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 194, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 8, 2024-03-24T06:51:17.734 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 101, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 112, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 11, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 182, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 198, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 16, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 5, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 29, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 24, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 55624, 2024-03-24T06:51:17.735 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 223612 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 100, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 97, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 18, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 3, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2024-03-24T06:51:17.736 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.006753972, 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.002251324 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:17.737 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.738 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 50, 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.027976362, 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000559527 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.739 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.740 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.741 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-24T06:51:17.742 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.743 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.743 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.743 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 31, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.029301141, 2024-03-24T06:51:17.745 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000945198 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.746 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:17.747 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.749 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.750 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.751 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.752 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.753 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:17.754 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 2, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 0, 2024-03-24T06:51:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 2, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 31, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 15, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 608, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-24T06:51:17.756 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:17.757 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:17.758 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:17.759 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 312375, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 243, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 264, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 3, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:17.761 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9656, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 186, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-24T06:51:17.762 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 693899, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 4811, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-24T06:51:17.763 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 575, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 1835225, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 29, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 575, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 575, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 54.293889731, 2024-03-24T06:51:17.764 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.094424156 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 575, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 656 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 575, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 312, 2024-03-24T06:51:17.765 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 263, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 0, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 10, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 0, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 155, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 21, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-24T06:51:17.766 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 6, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 312, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 20, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:17.767 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:17.768 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-24T06:51:17.769 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-24T06:51:17.770 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 104, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 15, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 11, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 11, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 93, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 0, 2024-03-24T06:51:17.771 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 39 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 1, 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-24T06:51:17.772 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-24T06:51:17.773 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 4709, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 3029753, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:17.774 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 4709, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 4709, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 3029753, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.775 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:17.776 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 575, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 1835225, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 575, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1835225, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.777 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 575, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 575, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 575, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 575, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.779 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:17.780 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.781 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x5627c1248500": { 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:17.782 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 1475, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1814627, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 1475, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 155, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1814627, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:17.783 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:17.784 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:17.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.784 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:17.784 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T06:51:18.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:18 smithi022 ceph-mon[32003]: pgmap v353: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-24T06:51:18.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:18 smithi143 ceph-mon[36520]: pgmap v353: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-24T06:51:18.510 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:18 smithi155 ceph-mon[36262]: pgmap v353: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-24T06:51:18.543 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds.imported=2 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=8 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=264 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:18.544 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-24T06:51:18.544 DEBUG:tasks.check_counter:Getting stats from l 2024-03-24T06:51:18.545 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-24T06:51:18.857 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T06:51:20.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:20 smithi022 ceph-mon[32003]: pgmap v354: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:20.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:20 smithi143 ceph-mon[36520]: pgmap v354: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:20.455 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:20 smithi155 ceph-mon[36262]: pgmap v354: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:20.919 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T06:51:20.919 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T06:51:20.919 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1088, 2024-03-24T06:51:20.919 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1065, 2024-03-24T06:51:20.919 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 2075920, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 402391, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 1, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.107600425, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.030171538, 2024-03-24T06:51:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.057613736, 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.026836470, 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1065, 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.034028993, 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000031952 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.921 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1806784, 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 153280 2024-03-24T06:51:20.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 599, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 783, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 147748, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 735072, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8, 2024-03-24T06:51:20.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.057567918, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.020044568, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.013002197, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.019666798, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 779, 2024-03-24T06:51:20.924 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.041075211, 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000052728 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.925 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 549648 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 863, 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 865, 2024-03-24T06:51:20.926 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1114178, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 268394, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 4, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.060946551, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.021114440, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.016273446, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.018599410, 2024-03-24T06:51:20.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 863, 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.024068276, 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000027889 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.928 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-24T06:51:20.929 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3900, 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.088736719, 2024-03-24T06:51:20.930 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000022753 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.931 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 0, 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 0, 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.932 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 0, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 0, 2024-03-24T06:51:20.933 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 464, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 6, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 458, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 456, 2024-03-24T06:51:20.934 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 4, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 0, 2024-03-24T06:51:20.935 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2024-03-24T06:51:20.936 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-24T06:51:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-24T06:51:20.938 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2024-03-24T06:51:20.939 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 0, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 0, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 0, 2024-03-24T06:51:20.940 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-24T06:51:20.941 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-24T06:51:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 0, 2024-03-24T06:51:20.943 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 448, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2024-03-24T06:51:20.944 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 5083311, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 5082737, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.945 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 448 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 324, 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 324, 2024-03-24T06:51:20.946 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 0, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 88, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 88, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 0, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 464, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 464, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 0, 2024-03-24T06:51:20.947 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 0, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 0, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 52108, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 215420 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 0, 2024-03-24T06:51:20.948 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 0, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 0, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-24T06:51:20.950 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.952 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-24T06:51:20.954 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.955 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.956 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-24T06:51:20.957 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.957 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.957 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-24T06:51:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.960 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.961 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-24T06:51:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.964 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.968 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.969 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 2, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 0, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 2, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 0, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2024-03-24T06:51:20.970 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 2519, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-24T06:51:20.971 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T06:51:20.972 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T06:51:20.973 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-24T06:51:20.974 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 204660, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 445, 2024-03-24T06:51:20.975 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9416, 2024-03-24T06:51:20.976 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 180, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 1275425, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 20353, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-24T06:51:20.977 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-24T06:51:20.978 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 2260, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 21, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 3, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 2260, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2260, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 28.441412715, 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.012584695 2024-03-24T06:51:20.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2260, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2261 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 2260, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2260, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 0, 2024-03-24T06:51:20.980 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1394, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 864, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-24T06:51:20.981 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 1, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 0, 2024-03-24T06:51:20.982 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-24T06:51:20.983 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-24T06:51:20.984 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 104, 2024-03-24T06:51:20.985 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 15, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 5, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 5, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 0, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2, 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 0 2024-03-24T06:51:20.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 0, 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-24T06:51:20.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-24T06:51:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 2546, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 3091150, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 2546, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2546, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 3091150, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:20.990 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:21.022 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:21.022 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:21.023 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2260, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 889012, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2260, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 889012, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2260, 2024-03-24T06:51:21.025 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2260, 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2260, 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2260, 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:21.027 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x55a0c08be6e0": { 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-24T06:51:21.029 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-24T06:51:21.030 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.031 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-24T06:51:21.031 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-24T06:51:21.663 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-24T06:51:21.664 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T06:51:21.664 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-24T06:51:21.664 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T06:51:21.675 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T06:51:21.675 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:51:21.675 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T06:51:21.758 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T06:51:21.758 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T06:51:21.814 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:51:22.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 -- 172.21.15.22:0/3959207162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 msgr2=0x7f7f60101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.043 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 --2- 172.21.15.22:0/3959207162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60101b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009930 tx=0x7f7f4c02bc20 comp rx=0 tx=0).stop 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 -- 172.21.15.22:0/3959207162 shutdown_connections 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 --2- 172.21.15.22:0/3959207162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 0x7f7f6010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 --2- 172.21.15.22:0/3959207162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60101b20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 --2- 172.21.15.22:0/3959207162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 0x7f7f60107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 -- 172.21.15.22:0/3959207162 >> 172.21.15.22:0/3959207162 conn(0x7f7f600fd1b0 msgr2=0x7f7f600ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 -- 172.21.15.22:0/3959207162 shutdown_connections 2024-03-24T06:51:22.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.043+0000 7f7f66600640 1 -- 172.21.15.22:0/3959207162 wait complete. 2024-03-24T06:51:22.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.045+0000 7f7f66600640 1 Processor -- start 2024-03-24T06:51:22.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.045+0000 7f7f66600640 1 -- start start 2024-03-24T06:51:22.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 0x7f7f60198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 0x7f7f6019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6019cc40 con 0x7f7f60102060 2024-03-24T06:51:22.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6019cdb0 con 0x7f7f601076b0 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f66600640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6019cf70 con 0x7f7f601016a0 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f5f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f5f400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56586/0 (socket says 172.21.15.22:56586) 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f5f400640 1 -- 172.21.15.22:0/4226751556 learned_addr learned my addr 172.21.15.22:0/4226751556 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f5ea00640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 0x7f7f60198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.046+0000 7f7f5fe00640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 0x7f7f6019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 -- 172.21.15.22:0/4226751556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 msgr2=0x7f7f6019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 0x7f7f6019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 -- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 msgr2=0x7f7f60198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 0x7f7f60198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 -- 172.21.15.22:0/4226751556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f58009840 con 0x7f7f601016a0 2024-03-24T06:51:22.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f5f400640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000e9f0 tx=0x7f7f5000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50009700 con 0x7f7f601016a0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f5000ccc0 con 0x7f7f601016a0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50018820 con 0x7f7f601016a0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.047+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4c009560 con 0x7f7f601016a0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.048+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f601a48f0 con 0x7f7f601016a0 2024-03-24T06:51:22.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.049+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f60068950 con 0x7f7f601016a0 2024-03-24T06:51:22.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.050+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7f50010430 con 0x7f7f601016a0 2024-03-24T06:51:22.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.052+0000 7f7f57e00640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 0x7f7f440a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.052+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f7f50014070 con 0x7f7f601016a0 2024-03-24T06:51:22.052 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.052+0000 7f7f5ea00640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 0x7f7f440a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.053 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.053+0000 7f7f5ea00640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 0x7f7f440a4770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009900 tx=0x7f7f4c005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:22.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.056+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7f50053280 con 0x7f7f601016a0 2024-03-24T06:51:22.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:22 smithi022 ceph-mon[32003]: pgmap v355: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:22.278 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.278+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f60109c30 con 0x7f7f601016a0 2024-03-24T06:51:22.280 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.280+0000 7f7f57e00640 1 -- 172.21.15.22:0/4226751556 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25271 (secure 0 0 0) 0x7f7f50097070 con 0x7f7f601016a0 2024-03-24T06:51:22.281 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:51:22.281 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":104,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:51:15.106526+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:51:22.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.282+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 msgr2=0x7f7f440a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.282+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 0x7f7f440a4770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009900 tx=0x7f7f4c005a20 comp rx=0 tx=0).stop 2024-03-24T06:51:22.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.282+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 msgr2=0x7f7f60197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.282+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000e9f0 tx=0x7f7f5000edf0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 shutdown_connections 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7f440a22b0 0x7f7f440a4770 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f601076b0 0x7f7f6019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7f60102060 0x7f7f60198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 --2- 172.21.15.22:0/4226751556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f601016a0 0x7f7f60197dd0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 >> 172.21.15.22:0/4226751556 conn(0x7f7f600fd1b0 msgr2=0x7f7f600fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 shutdown_connections 2024-03-24T06:51:22.284 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.283+0000 7f7f66600640 1 -- 172.21.15.22:0/4226751556 wait complete. 2024-03-24T06:51:22.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T06:31:39.650101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T06:36:54.347015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T06:36:55.706800+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T06:36:59.747616+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T06:51:22.303 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T06:51:22.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:22 smithi143 ceph-mon[36520]: pgmap v355: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:22.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:22 smithi155 ceph-mon[36262]: pgmap v355: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 -- 172.21.15.22:0/22824706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 msgr2=0x7f0a30107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 --2- 172.21.15.22:0/22824706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a30107a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009cd0 tx=0x7f0a2002c0f0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 -- 172.21.15.22:0/22824706 shutdown_connections 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 --2- 172.21.15.22:0/22824706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30102050 0x7f0a3010e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 --2- 172.21.15.22:0/22824706 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30101690 0x7f0a30101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 --2- 172.21.15.22:0/22824706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a30107a80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.536+0000 7f0a36e00640 1 -- 172.21.15.22:0/22824706 >> 172.21.15.22:0/22824706 conn(0x7f0a300fd1a0 msgr2=0x7f0a300ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:22.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.537+0000 7f0a36e00640 1 -- 172.21.15.22:0/22824706 shutdown_connections 2024-03-24T06:51:22.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.537+0000 7f0a36e00640 1 -- 172.21.15.22:0/22824706 wait complete. 2024-03-24T06:51:22.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.538+0000 7f0a36e00640 1 Processor -- start 2024-03-24T06:51:22.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.538+0000 7f0a36e00640 1 -- start start 2024-03-24T06:51:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 0x7f0a3019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a3019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3019ee90 con 0x7f0a301076a0 2024-03-24T06:51:22.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3019f000 con 0x7f0a30101690 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a36e00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3019f1c0 con 0x7f0a30102050 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:40580/0 (socket says 172.21.15.22:40580) 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a2fe00640 1 -- 172.21.15.22:0/3811963203 learned_addr learned my addr 172.21.15.22:0/3811963203 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a27400640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 0x7f0a3019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.539+0000 7f0a34e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a3019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 -- 172.21.15.22:0/3811963203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 msgr2=0x7f0a3019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 0x7f0a3019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 -- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 msgr2=0x7f0a3019e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a3019e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 -- 172.21.15.22:0/3811963203 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a28009490 con 0x7f0a30101690 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a27400640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 0x7f0a3019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2fe00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0a20004af0 tx=0x7f0a200325c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20034e20 con 0x7f0a30101690 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a20009900 con 0x7f0a30101690 2024-03-24T06:51:22.543 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a20003e50 con 0x7f0a30101690 2024-03-24T06:51:22.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200047c0 con 0x7f0a30101690 2024-03-24T06:51:22.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.540+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a301a6b40 con 0x7f0a30101690 2024-03-24T06:51:22.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.541+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a30068500 con 0x7f0a30101690 2024-03-24T06:51:22.544 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.543+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f0a2000a040 con 0x7f0a30101690 2024-03-24T06:51:22.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.545+0000 7f0a2d600640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 0x7f0a140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:22.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.545+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f0a200f0b00 con 0x7f0a30101690 2024-03-24T06:51:22.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.545+0000 7f0a27400640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 0x7f0a140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:22.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.546+0000 7f0a27400640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 0x7f0a140a46f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a3010e890 tx=0x7f0a18005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:22.549 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.549+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a200f2e60 con 0x7f0a30101690 2024-03-24T06:51:22.812 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.812+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a3019f380 con 0x7f0a30101690 2024-03-24T06:51:22.813 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.813+0000 7f0a2d600640 1 -- 172.21.15.22:0/3811963203 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7f0a200390a0 con 0x7f0a30101690 2024-03-24T06:51:22.813 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T06:51:22.817 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 msgr2=0x7f0a140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 0x7f0a140a46f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0a3010e890 tx=0x7f0a18005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 msgr2=0x7f0a30199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0a20004af0 tx=0x7f0a200325c0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 shutdown_connections 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f0a140a2230 0x7f0a140a46f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a301076a0 0x7f0a3019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.818 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a30102050 0x7f0a3019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 --2- 172.21.15.22:0/3811963203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a30101690 0x7f0a30199fd0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:22.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 >> 172.21.15.22:0/3811963203 conn(0x7f0a300fd1a0 msgr2=0x7f0a3010cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:22.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.816+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 shutdown_connections 2024-03-24T06:51:22.819 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:22.817+0000 7f0a36e00640 1 -- 172.21.15.22:0/3811963203 wait complete. 2024-03-24T06:51:22.838 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T06:51:23.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 -- 172.21.15.22:0/3134992586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c4102060 msgr2=0x7f05c410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 --2- 172.21.15.22:0/3134992586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c4102060 0x7f05c410e570 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f05bc009cd0 tx=0x7f05bc02c0e0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 -- 172.21.15.22:0/3134992586 shutdown_connections 2024-03-24T06:51:23.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 --2- 172.21.15.22:0/3134992586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c4102060 0x7f05c410e570 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 --2- 172.21.15.22:0/3134992586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41016a0 0x7f05c4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 --2- 172.21.15.22:0/3134992586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c41076b0 0x7f05c4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 -- 172.21.15.22:0/3134992586 >> 172.21.15.22:0/3134992586 conn(0x7f05c4078b10 msgr2=0x7f05c4078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:23.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 -- 172.21.15.22:0/3134992586 shutdown_connections 2024-03-24T06:51:23.084 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.082+0000 7f05cbc00640 1 -- 172.21.15.22:0/3134992586 wait complete. 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.083+0000 7f05cbc00640 1 Processor -- start 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 -- start start 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 0x7f05c419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41076b0 0x7f05c419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f05c419edc0 con 0x7f05c4102060 2024-03-24T06:51:23.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f05c419ef30 con 0x7f05c41076b0 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.084+0000 7f05cbc00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05c419f0f0 con 0x7f05c41016a0 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c3e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 0x7f05c419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56620/0 (socket says 172.21.15.22:56620) 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 -- 172.21.15.22:0/1100284227 learned_addr learned my addr 172.21.15.22:0/1100284227 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9c00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41076b0 0x7f05c419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 -- 172.21.15.22:0/1100284227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41076b0 msgr2=0x7f05c419e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.088 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41076b0 0x7f05c419e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 -- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 msgr2=0x7f05c419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 0x7f05c419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c9200640 1 -- 172.21.15.22:0/1100284227 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc009900 con 0x7f05c41016a0 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.085+0000 7f05c3e00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 0x7f05c419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05c9200640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e9f0 tx=0x7f05b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8009700 con 0x7f05c41016a0 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05b800cd80 con 0x7f05c41016a0 2024-03-24T06:51:23.089 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c419f3d0 con 0x7f05c41016a0 2024-03-24T06:51:23.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8018920 con 0x7f05c41016a0 2024-03-24T06:51:23.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.086+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c41a6c50 con 0x7f05c41016a0 2024-03-24T06:51:23.090 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.087+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c4068950 con 0x7f05c41016a0 2024-03-24T06:51:23.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.089+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f05b8018a80 con 0x7f05c41016a0 2024-03-24T06:51:23.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.090+0000 7f05c1600640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 0x7f05ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.091+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f05b8014070 con 0x7f05c41016a0 2024-03-24T06:51:23.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.091+0000 7f05c3e00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 0x7f05ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.092 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.091+0000 7f05c3e00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 0x7f05ac0a4770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f05c410d810 tx=0x7f05b0005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:23.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.097+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f05b80974a0 con 0x7f05c41016a0 2024-03-24T06:51:23.311 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.311+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f05c41118a0 con 0x7f05c41016a0 2024-03-24T06:51:23.312 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.312+0000 7f05c1600640 1 -- 172.21.15.22:0/1100284227 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7f05b8026020 con 0x7f05c41016a0 2024-03-24T06:51:23.312 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:51:23.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 msgr2=0x7f05ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 0x7f05ac0a4770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f05c410d810 tx=0x7f05b0005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:23.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 msgr2=0x7f05c4199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.315 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f05b800e9f0 tx=0x7f05b800eeb0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 shutdown_connections 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f05ac0a22b0 0x7f05ac0a4770 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05c41076b0 0x7f05c419e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05c4102060 0x7f05c419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 --2- 172.21.15.22:0/1100284227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05c41016a0 0x7f05c4199f00 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.314+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 >> 172.21.15.22:0/1100284227 conn(0x7f05c4078b10 msgr2=0x7f05c41031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.315+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 shutdown_connections 2024-03-24T06:51:23.316 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.315+0000 7f05cbc00640 1 -- 172.21.15.22:0/1100284227 wait complete. 2024-03-24T06:51:23.334 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T06:51:23.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:23 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/4226751556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:23.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:23 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3811963203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:51:23.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:23 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/4226751556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:23.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:23 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3811963203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:51:23.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:23 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/4226751556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:23.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:23 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3811963203' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 -- 172.21.15.22:0/1667197375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 msgr2=0x7fd8b0110780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/1667197375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b0110780 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009900 tx=0x7fd8a802bc30 comp rx=0 tx=0).stop 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 -- 172.21.15.22:0/1667197375 shutdown_connections 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/1667197375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b0110780 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/1667197375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01038b0 0x7fd8b0103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/1667197375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01098c0 0x7fd8b0109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 -- 172.21.15.22:0/1667197375 >> 172.21.15.22:0/1667197375 conn(0x7fd8b00ff3c0 msgr2=0x7fd8b01017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:23.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 -- 172.21.15.22:0/1667197375 shutdown_connections 2024-03-24T06:51:23.576 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.574+0000 7fd8b4c00640 1 -- 172.21.15.22:0/1667197375 wait complete. 2024-03-24T06:51:23.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.575+0000 7fd8b4c00640 1 Processor -- start 2024-03-24T06:51:23.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.575+0000 7fd8b4c00640 1 -- start start 2024-03-24T06:51:23.578 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01098c0 0x7fd8b019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b019edd0 con 0x7fd8b01098c0 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b019ef40 con 0x7fd8b0104270 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8b4c00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b019f100 con 0x7fd8b01038b0 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.579 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56638/0 (socket says 172.21.15.22:56638) 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ade00640 1 -- 172.21.15.22:0/260090605 learned_addr learned my addr 172.21.15.22:0/260090605 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ad400640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ae800640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01098c0 0x7fd8b019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ade00640 1 -- 172.21.15.22:0/260090605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 msgr2=0x7fd8b019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.576+0000 7fd8ade00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8ade00640 1 -- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01098c0 msgr2=0x7fd8b019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8ade00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01098c0 0x7fd8b019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8ade00640 1 -- 172.21.15.22:0/260090605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a8009560 con 0x7fd8b01038b0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8ad400640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8ade00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000eb90 tx=0x7fd8a000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00099b0 con 0x7fd8b01038b0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a000ee90 con 0x7fd8b01038b0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b019f3e0 con 0x7fd8b01038b0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00107a0 con 0x7fd8b01038b0 2024-03-24T06:51:23.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.577+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b01a6c60 con 0x7fd8b01038b0 2024-03-24T06:51:23.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.579+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b0068950 con 0x7fd8b01038b0 2024-03-24T06:51:23.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.580+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fd8a0005020 con 0x7fd8b01038b0 2024-03-24T06:51:23.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.582+0000 7fd8a6a00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 0x7fd8940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:23.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.582+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7fd8a0014070 con 0x7fd8b01038b0 2024-03-24T06:51:23.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.582+0000 7fd8ad400640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 0x7fd8940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:23.583 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.583+0000 7fd8ad400640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 0x7fd8940a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0101e00 tx=0x7fd898005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:23.586 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.585+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd8a00974a0 con 0x7fd8b01038b0 2024-03-24T06:51:23.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.792+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd8b0113a50 con 0x7fd8b01038b0 2024-03-24T06:51:23.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.793+0000 7fd8a6a00640 1 -- 172.21.15.22:0/260090605 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7fd8a009ab60 con 0x7fd8b01038b0 2024-03-24T06:51:23.793 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2024-03-24T06:51:23.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 msgr2=0x7fd8940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 0x7fd8940a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0101e00 tx=0x7fd898005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:23.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 msgr2=0x7fd8b0199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:23.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000eb90 tx=0x7fd8a000c6a0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.797 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 shutdown_connections 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fd8940a2230 0x7fd8940a46f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd8b01098c0 0x7fd8b019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8b0104270 0x7fd8b019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 --2- 172.21.15.22:0/260090605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8b01038b0 0x7fd8b0199f10 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.796+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 >> 172.21.15.22:0/260090605 conn(0x7fd8b00ff3c0 msgr2=0x7fd8b00ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.797+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 shutdown_connections 2024-03-24T06:51:23.798 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:23.797+0000 7fd8b4c00640 1 -- 172.21.15.22:0/260090605 wait complete. 2024-03-24T06:51:23.819 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T06:51:24.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 -- 172.21.15.22:0/1338479887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c10bb60 msgr2=0x7f176c10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 --2- 172.21.15.22:0/1338479887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c10bb60 0x7f176c10e000 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1764009be0 tx=0x7f176402bf80 comp rx=0 tx=0).stop 2024-03-24T06:51:24.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 -- 172.21.15.22:0/1338479887 shutdown_connections 2024-03-24T06:51:24.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 --2- 172.21.15.22:0/1338479887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c10bb60 0x7f176c10e000 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 --2- 172.21.15.22:0/1338479887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c103060 0x7f176c10b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.041 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 --2- 172.21.15.22:0/1338479887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c102740 0x7f176c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 -- 172.21.15.22:0/1338479887 >> 172.21.15.22:0/1338479887 conn(0x7f176c0fbf60 msgr2=0x7f176c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:24.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 -- 172.21.15.22:0/1338479887 shutdown_connections 2024-03-24T06:51:24.042 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.040+0000 7f176be00640 1 -- 172.21.15.22:0/1338479887 wait complete. 2024-03-24T06:51:24.044 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 Processor -- start 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 -- start start 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 0x7f176c199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 0x7f176c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f176c19ec50 con 0x7f176c103060 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f176c19edc0 con 0x7f176c102740 2024-03-24T06:51:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.041+0000 7f176be00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f176c19ef80 con 0x7f176c10bb60 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:56660/0 (socket says 172.21.15.22:56660) 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 -- 172.21.15.22:0/2146134541 learned_addr learned my addr 172.21.15.22:0/2146134541 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176a000640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 0x7f176c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176aa00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 0x7f176c199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 -- 172.21.15.22:0/2146134541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 msgr2=0x7f176c199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 0x7f176c199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 -- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 msgr2=0x7f176c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 0x7f176c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 -- 172.21.15.22:0/2146134541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175c009560 con 0x7f176c10bb60 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176aa00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 0x7f176c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176a000640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 0x7f176c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.042+0000 7f176b400640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1764036040 tx=0x7f1764004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.043+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764004620 con 0x7f176c10bb60 2024-03-24T06:51:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.043+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1764004780 con 0x7f176c10bb60 2024-03-24T06:51:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.043+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176403e630 con 0x7f176c10bb60 2024-03-24T06:51:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.043+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1764009840 con 0x7f176c10bb60 2024-03-24T06:51:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.043+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c1a6900 con 0x7f176c10bb60 2024-03-24T06:51:24.048 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.044+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176c069150 con 0x7f176c10bb60 2024-03-24T06:51:24.049 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.046+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f1764034b30 con 0x7f176c10bb60 2024-03-24T06:51:24.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.049+0000 7f1763400640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 0x7f17500a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.049+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f17640f0ff0 con 0x7f176c10bb60 2024-03-24T06:51:24.050 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.049+0000 7f176aa00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 0x7f17500a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.050+0000 7f176aa00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 0x7f17500a4770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f175c005fd0 tx=0x7f175c009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:24.051 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.050+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f17640b99a0 con 0x7f176c10bb60 2024-03-24T06:51:24.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.252+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f176c1116f0 con 0x7f176c10bb60 2024-03-24T06:51:24.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.253+0000 7f1763400640 1 -- 172.21.15.22:0/2146134541 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) ==== 103+0+11 (secure 0 0 0) 0x7f17640bd060 con 0x7f176c10bb60 2024-03-24T06:51:24.254 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 msgr2=0x7f17500a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 0x7f17500a4770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f175c005fd0 tx=0x7f175c009260 comp rx=0 tx=0).stop 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 msgr2=0x7f176c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1764036040 tx=0x7f1764004270 comp rx=0 tx=0).stop 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 shutdown_connections 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f17500a22b0 0x7f17500a4770 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f176c10bb60 0x7f176c19e650 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f176c103060 0x7f176c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 --2- 172.21.15.22:0/2146134541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f176c102740 0x7f176c199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 >> 172.21.15.22:0/2146134541 conn(0x7f176c0fbf60 msgr2=0x7f176c10dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:24.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 shutdown_connections 2024-03-24T06:51:24.258 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.256+0000 7f176be00640 1 -- 172.21.15.22:0/2146134541 wait complete. 2024-03-24T06:51:24.278 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T06:51:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:24 smithi022 ceph-mon[32003]: pgmap v356: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:24 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1100284227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:24 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/260090605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:24 smithi143 ceph-mon[36520]: pgmap v356: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:24.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:24 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1100284227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:24 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/260090605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.492+0000 7f72cbe00640 1 -- 172.21.15.22:0/3026235968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c4104270 msgr2=0x7f72c4110780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.492+0000 7f72cbe00640 1 --2- 172.21.15.22:0/3026235968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c4104270 0x7f72c4110780 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009be0 tx=0x7f72bc02bef0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.493 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.492+0000 7f72cbe00640 1 -- 172.21.15.22:0/3026235968 shutdown_connections 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.492+0000 7f72cbe00640 1 --2- 172.21.15.22:0/3026235968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c4104270 0x7f72c4110780 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.493+0000 7f72cbe00640 1 --2- 172.21.15.22:0/3026235968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.493+0000 7f72cbe00640 1 --2- 172.21.15.22:0/3026235968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c41098c0 0x7f72c4109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.493+0000 7f72cbe00640 1 -- 172.21.15.22:0/3026235968 >> 172.21.15.22:0/3026235968 conn(0x7f72c40ff360 msgr2=0x7f72c4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.493+0000 7f72cbe00640 1 -- 172.21.15.22:0/3026235968 shutdown_connections 2024-03-24T06:51:24.494 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.493+0000 7f72cbe00640 1 -- 172.21.15.22:0/3026235968 wait complete. 2024-03-24T06:51:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.494+0000 7f72cbe00640 1 Processor -- start 2024-03-24T06:51:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.494+0000 7f72cbe00640 1 -- start start 2024-03-24T06:51:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 0x7f72c419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c41098c0 0x7f72c419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f72c419ed90 con 0x7f72c41038b0 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f72c419ef00 con 0x7f72c41098c0 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72cbe00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f72c419f0c0 con 0x7f72c4104270 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72c9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72c9400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37512/0 (socket says 172.21.15.22:37512) 2024-03-24T06:51:24.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72c9400640 1 -- 172.21.15.22:0/129630184 learned_addr learned my addr 172.21.15.22:0/129630184 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72c8a00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 0x7f72c419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.495+0000 7f72c9e00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c41098c0 0x7f72c419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 -- 172.21.15.22:0/129630184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 msgr2=0x7f72c419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 0x7f72c419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 -- 172.21.15.22:0/129630184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c41098c0 msgr2=0x7f72c419e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c41098c0 0x7f72c419e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 -- 172.21.15.22:0/129630184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b8009490 con 0x7f72c41038b0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c8a00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 0x7f72c419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c9400640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72b800ea60 tx=0x7f72b800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.496+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b800cd20 con 0x7f72c41038b0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.497+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72bc009840 con 0x7f72c41038b0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.497+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b800ce80 con 0x7f72c41038b0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.497+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8010690 con 0x7f72c41038b0 2024-03-24T06:51:24.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.497+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c41a6a40 con 0x7f72c41038b0 2024-03-24T06:51:24.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.498+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c4068950 con 0x7f72c41038b0 2024-03-24T06:51:24.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.502+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f72b8003ee0 con 0x7f72c41038b0 2024-03-24T06:51:24.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.504+0000 7f72c2000640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 0x7f72ac0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:24.504 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.504+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f72b8014070 con 0x7f72c41038b0 2024-03-24T06:51:24.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.504+0000 7f72c8a00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 0x7f72ac0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:24.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.505+0000 7f72c8a00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 0x7f72ac0a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f72c41019e0 tx=0x7f72b0009260 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:24.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.509+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f72b8097110 con 0x7f72c41038b0 2024-03-24T06:51:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:24 smithi155 ceph-mon[36262]: pgmap v356: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:24 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1100284227' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:24 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/260090605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T06:51:24.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.728+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f72c419b220 con 0x7f72c41038b0 2024-03-24T06:51:24.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.729+0000 7f72c2000640 1 -- 172.21.15.22:0/129630184 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) ==== 81+0+937 (secure 0 0 0) 0x7f72b809a7d0 con 0x7f72c41038b0 2024-03-24T06:51:24.729 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/673082346 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/3288864197 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/440133497 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/503368228 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/503368228 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/896325545 2024-03-25T06:28:54.941425+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/4236060832 2024-03-25T06:40:00.112683+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2695260292 2024-03-25T06:28:54.941425+0000 2024-03-24T06:51:24.730 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/2009113619 2024-03-25T06:28:54.941425+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/2447624484 2024-03-25T06:28:20.396453+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6800/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:6801/37818756 2024-03-25T06:28:20.396453+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1811358368 2024-03-25T06:28:20.396453+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/3441957019 2024-03-25T06:28:20.396453+0000 2024-03-24T06:51:24.731 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.15.22:0/1555230471 2024-03-25T06:28:54.941425+0000 2024-03-24T06:51:24.732 INFO:teuthology.orchestra.run.smithi022.stderr:listed 16 entries 2024-03-24T06:51:24.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.731+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 msgr2=0x7f72ac0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.731+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 0x7f72ac0a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f72c41019e0 tx=0x7f72b0009260 comp rx=0 tx=0).stop 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.731+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 msgr2=0x7f72c4199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.731+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72b800ea60 tx=0x7f72b800ee60 comp rx=0 tx=0).stop 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 shutdown_connections 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f72ac0a2230 0x7f72ac0a46f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c41098c0 0x7f72c419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72c4104270 0x7f72c419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.733 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 --2- 172.21.15.22:0/129630184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f72c41038b0 0x7f72c4199ed0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 >> 172.21.15.22:0/129630184 conn(0x7f72c40ff360 msgr2=0x7f72c40ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 shutdown_connections 2024-03-24T06:51:24.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:24.732+0000 7f72cbe00640 1 -- 172.21.15.22:0/129630184 wait complete. 2024-03-24T06:51:24.753 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T06:51:24.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:51:24.753 DEBUG:teuthology.orchestra.run.smithi022:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T06:51:25.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:25 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2146134541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:51:25.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:25 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/129630184' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:51:25.334 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T06:51:25.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:51:25.334 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T06:51:25.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:25 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2146134541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:51:25.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:25 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/129630184' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:51:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:25 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2146134541' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T06:51:25.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:25 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/129630184' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T06:51:25.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:51:25.430 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T06:51:25.542 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T06:51:25.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:51:25.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T06:51:25.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T06:51:25.615 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link delete ceph-brx 2024-03-24T06:51:25.736 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T06:51:25.747 INFO:tasks.cephadm:Teardown begin 2024-03-24T06:51:25.747 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T06:51:25.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:51:25.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 -- 172.21.15.22:0/2082845524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100520 msgr2=0x7f8388100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:25.950 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 --2- 172.21.15.22:0/2082845524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100520 0x7f8388100900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8378009900 tx=0x7f837802bc40 comp rx=0 tx=0).stop 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 -- 172.21.15.22:0/2082845524 shutdown_connections 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 --2- 172.21.15.22:0/2082845524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838810e0d0 0x7f83881104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 --2- 172.21.15.22:0/2082845524 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8388100e40 0x7f838810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 --2- 172.21.15.22:0/2082845524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100520 0x7f8388100900 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 -- 172.21.15.22:0/2082845524 >> 172.21.15.22:0/2082845524 conn(0x7f8388077770 msgr2=0x7f8388077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 -- 172.21.15.22:0/2082845524 shutdown_connections 2024-03-24T06:51:25.951 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.950+0000 7f838e200640 1 -- 172.21.15.22:0/2082845524 wait complete. 2024-03-24T06:51:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.951+0000 7f838e200640 1 Processor -- start 2024-03-24T06:51:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.951+0000 7f838e200640 1 -- start start 2024-03-24T06:51:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100e40 0x7f838819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:25.953 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f838810e0d0 0x7f838819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f838819ee30 con 0x7f838810e0d0 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f838819efa0 con 0x7f8388100520 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f838e200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f838819f160 con 0x7f8388100e40 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8386a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100e40 0x7f838819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:55422/0 (socket says 172.21.15.22:55422) 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f838810e0d0 0x7f838819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:25.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 -- 172.21.15.22:0/3951786721 learned_addr learned my addr 172.21.15.22:0/3951786721 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 -- 172.21.15.22:0/3951786721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100e40 msgr2=0x7f838819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100e40 0x7f838819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 -- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f838810e0d0 msgr2=0x7f838819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.952+0000 7f8387400640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f838810e0d0 0x7f838819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f8387400640 1 -- 172.21.15.22:0/3951786721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8374009490 con 0x7f8388100520 2024-03-24T06:51:25.955 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f8387400640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f837802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:25.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83780042d0 con 0x7f8388100520 2024-03-24T06:51:25.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8380009840 con 0x7f8388100520 2024-03-24T06:51:25.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8378004430 con 0x7f8388100520 2024-03-24T06:51:25.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378003c50 con 0x7f8388100520 2024-03-24T06:51:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.953+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8378009560 con 0x7f8388100520 2024-03-24T06:51:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.955+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388068950 con 0x7f8388100520 2024-03-24T06:51:25.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.956+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f83780054f0 con 0x7f8388100520 2024-03-24T06:51:25.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.958+0000 7f837fe00640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 0x7f836c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:25.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.958+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f83780f0960 con 0x7f8388100520 2024-03-24T06:51:25.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.958+0000 7f8386a00640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 0x7f836c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:25.959 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.959+0000 7f8386a00640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 0x7f836c0a4770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f83880fffd0 tx=0x7f8374005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:25.962 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:25.962+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f83780b9380 con 0x7f8388100520 2024-03-24T06:51:26.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.282+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8388109260 con 0x7f8388100520 2024-03-24T06:51:26.283 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.283+0000 7f837fe00640 1 -- 172.21.15.22:0/3951786721 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f83780bca40 con 0x7f8388100520 2024-03-24T06:51:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:51:26.284 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:51:26.284 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:51:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 msgr2=0x7f836c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 0x7f836c0a4770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f83880fffd0 tx=0x7f8374005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 msgr2=0x7f8388199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f837802c640 comp rx=0 tx=0).stop 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 shutdown_connections 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f836c0a22b0 0x7f836c0a4770 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f838810e0d0 0x7f838819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8388100e40 0x7f838819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.287 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 --2- 172.21.15.22:0/3951786721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8388100520 0x7f8388199f70 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 >> 172.21.15.22:0/3951786721 conn(0x7f8388077770 msgr2=0x7f838810f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:26.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 shutdown_connections 2024-03-24T06:51:26.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.286+0000 7f838e200640 1 -- 172.21.15.22:0/3951786721 wait complete. 2024-03-24T06:51:26.303 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:26 smithi022 ceph-mon[32003]: pgmap v357: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:26.307 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T06:51:26.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:26 smithi143 ceph-mon[36520]: pgmap v357: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:26.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.519+0000 7f9c60a00640 1 -- 172.21.15.22:0/499808193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c107600 msgr2=0x7f9c5c1079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.519+0000 7f9c60a00640 1 --2- 172.21.15.22:0/499808193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c107600 0x7f9c5c1079e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009930 tx=0x7f9c4c02bc30 comp rx=0 tx=0).stop 2024-03-24T06:51:26.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 -- 172.21.15.22:0/499808193 shutdown_connections 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 --2- 172.21.15.22:0/499808193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c069230 0x7f9c5c10e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 --2- 172.21.15.22:0/499808193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 --2- 172.21.15.22:0/499808193 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c107600 0x7f9c5c1079e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 -- 172.21.15.22:0/499808193 >> 172.21.15.22:0/499808193 conn(0x7f9c5c076020 msgr2=0x7f9c5c076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 -- 172.21.15.22:0/499808193 shutdown_connections 2024-03-24T06:51:26.521 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.520+0000 7f9c60a00640 1 -- 172.21.15.22:0/499808193 wait complete. 2024-03-24T06:51:26.523 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.521+0000 7f9c60a00640 1 Processor -- start 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.521+0000 7f9c60a00640 1 -- start start 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.521+0000 7f9c60a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c60a00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 0x7f9c5c100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c60a00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c107600 0x7f9c5c101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c60a00640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5c1a5f40 con 0x7f9c5c069230 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c60a00640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5c1a60b0 con 0x7f9c5c107600 2024-03-24T06:51:26.524 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c60a00640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5c1a6270 con 0x7f9c5c068870 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c53e00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 0x7f9c5c100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38594/0 (socket says 172.21.15.22:38594) 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 -- 172.21.15.22:0/2424525877 learned_addr learned my addr 172.21.15.22:0/2424525877 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5b200640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c107600 0x7f9c5c101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 -- 172.21.15.22:0/2424525877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c107600 msgr2=0x7f9c5c101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c107600 0x7f9c5c101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.525 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 -- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 msgr2=0x7f9c5c100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 0x7f9c5c100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c5a800640 1 -- 172.21.15.22:0/2424525877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c54009840 con 0x7f9c5c068870 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.522+0000 7f9c53e00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 0x7f9c5c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c5a800640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c000c00 tx=0x7f9c4c02c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0042d0 con 0x7f9c5c068870 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c009560 con 0x7f9c5c068870 2024-03-24T06:51:26.526 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c4c004430 con 0x7f9c5c068870 2024-03-24T06:51:26.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003b20 con 0x7f9c5c068870 2024-03-24T06:51:26.527 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.523+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c5c1a68c0 con 0x7f9c5c068870 2024-03-24T06:51:26.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:26 smithi155 ceph-mon[36262]: pgmap v357: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T06:51:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.524+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c5c068420 con 0x7f9c5c068870 2024-03-24T06:51:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.526+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f9c4c003c80 con 0x7f9c5c068870 2024-03-24T06:51:26.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.527+0000 7f9c53400640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 0x7f9c400a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:26.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.528+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f9c4c0f0af0 con 0x7f9c5c068870 2024-03-24T06:51:26.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.528+0000 7f9c53e00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 0x7f9c400a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:26.529 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.529+0000 7f9c53e00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 0x7f9c400a4820 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c1021d0 tx=0x7f9c44005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:26.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.531+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c4c0b9480 con 0x7f9c5c068870 2024-03-24T06:51:26.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.786+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c5c109a40 con 0x7f9c5c068870 2024-03-24T06:51:26.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.787+0000 7f9c53400640 1 -- 172.21.15.22:0/2424525877 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11939 (secure 0 0 0) 0x7f9c4c043020 con 0x7f9c5c068870 2024-03-24T06:51:26.788 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:51:26.788 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/4113284449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":4113284449},{"type":"v1","addr":"172.21.15.22:6837","nonce":4113284449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6839/2318828115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6838","nonce":2318828115},{"type":"v1","addr":"172.21.15.22:6839","nonce":2318828115}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T06:36:56.216020+0000","modified":"2024-03-24T06:46:05.176270+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24601,"mds_1":24584,"mds_2":24565,"mds_3":24544,"mds_4":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/3570497194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":3570497194},{"type":"v1","addr":"172.21.15.143:6833","nonce":3570497194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6833/715907814","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":715907814},{"type":"v1","addr":"172.21.15.155:6833","nonce":715907814}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.22:6835/298241489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6834","nonce":298241489},{"type":"v1","addr":"172.21.15.22:6835","nonce":298241489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6835/1050031463","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1050031463},{"type":"v1","addr":"172.21.15.143:6835","nonce":1050031463}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/988781970","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":988781970},{"type":"v1","addr":"172.21.15.143:6837","nonce":988781970}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.155:6835/1202752642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1202752642},{"type":"v1","addr":"172.21.15.155:6835","nonce":1202752642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":16,"addr":"172.21.15.155:6837/1157072254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1157072254},{"type":"v1","addr":"172.21.15.155:6837","nonce":1157072254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24565":{"gid":24565,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.155:6839/2828760693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":2828760693},{"type":"v1","addr":"172.21.15.155:6839","nonce":2828760693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/2325215622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":2325215622},{"type":"v1","addr":"172.21.15.143:6839","nonce":2325215622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6841/857116957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6840","nonce":857116957},{"type":"v1","addr":"172.21.15.22:6841","nonce":857116957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24601,"qdb_cluster":[24565,24544,24532,24584,24601]},"id":1}]} 2024-03-24T06:51:26.789 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 37 2024-03-24T06:51:26.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 msgr2=0x7f9c400a4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 0x7f9c400a4820 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c1021d0 tx=0x7f9c44005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 msgr2=0x7f9c5c100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c000c00 tx=0x7f9c4c02c5f0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 shutdown_connections 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f9c400a2360 0x7f9c400a4820 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c5c107600 0x7f9c5c101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c5c069230 0x7f9c5c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 --2- 172.21.15.22:0/2424525877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c5c068870 0x7f9c5c100790 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:26.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 >> 172.21.15.22:0/2424525877 conn(0x7f9c5c076020 msgr2=0x7f9c5c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:26.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 shutdown_connections 2024-03-24T06:51:26.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:26.790+0000 7f9c60a00640 1 -- 172.21.15.22:0/2424525877 wait complete. 2024-03-24T06:51:26.810 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 -- 172.21.15.22:0/557999995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814107600 msgr2=0x7f88141079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 --2- 172.21.15.22:0/557999995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814107600 0x7f88141079e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8808009930 tx=0x7f880802bc20 comp rx=0 tx=0).stop 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 -- 172.21.15.22:0/557999995 shutdown_connections 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 --2- 172.21.15.22:0/557999995 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814069230 0x7f881410e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 --2- 172.21.15.22:0/557999995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814068870 0x7f8814068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 --2- 172.21.15.22:0/557999995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814107600 0x7f88141079e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 -- 172.21.15.22:0/557999995 >> 172.21.15.22:0/557999995 conn(0x7f8814076020 msgr2=0x7f8814076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 -- 172.21.15.22:0/557999995 shutdown_connections 2024-03-24T06:51:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.028+0000 7f881a400640 1 -- 172.21.15.22:0/557999995 wait complete. 2024-03-24T06:51:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.029+0000 7f881a400640 1 Processor -- start 2024-03-24T06:51:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 -- start start 2024-03-24T06:51:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814068870 0x7f8814100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814069230 0x7f8814100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f88141a5e70 con 0x7f8814068870 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f88141a5fe0 con 0x7f8814069230 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f881a400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f88141a61a0 con 0x7f8814107600 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f8819000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814068870 0x7f8814100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.030+0000 7f8819a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8813e00640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814069230 0x7f8814100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38614/0 (socket says 172.21.15.22:38614) 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 -- 172.21.15.22:0/394992564 learned_addr learned my addr 172.21.15.22:0/394992564 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 -- 172.21.15.22:0/394992564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814069230 msgr2=0x7f8814100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814069230 0x7f8814100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 -- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814068870 msgr2=0x7f8814100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814068870 0x7f8814100790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 -- 172.21.15.22:0/394992564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8808009560 con 0x7f8814107600 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.031+0000 7f8819a00640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f880c00eaa0 tx=0x7f880c00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.032+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c00cdb0 con 0x7f8814107600 2024-03-24T06:51:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.032+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88141a6480 con 0x7f8814107600 2024-03-24T06:51:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.032+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c005020 con 0x7f8814107600 2024-03-24T06:51:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.032+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0116b0 con 0x7f8814107600 2024-03-24T06:51:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.032+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88141a6940 con 0x7f8814107600 2024-03-24T06:51:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.033+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8814068420 con 0x7f8814107600 2024-03-24T06:51:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.035+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f880c005520 con 0x7f8814107600 2024-03-24T06:51:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.036+0000 7f8811600640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 0x7f87fc0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.036+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f880c0ce570 con 0x7f8814107600 2024-03-24T06:51:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.036+0000 7f8819000640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 0x7f87fc0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.037+0000 7f8819000640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 0x7f87fc0a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8808005ab0 tx=0x7f8808005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:27.040 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.040+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f880c096f00 con 0x7f8814107600 2024-03-24T06:51:27.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.249+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8814109a20 con 0x7f8814107600 2024-03-24T06:51:27.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.252+0000 7f8811600640 1 -- 172.21.15.22:0/394992564 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25271 (secure 0 0 0) 0x7f880c01f090 con 0x7f8814107600 2024-03-24T06:51:27.253 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T06:51:27.253 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":104,"fsid":"7bca48fe-e9a7-11ee-95cd-87774f69a715","created":"2024-03-24T06:27:47.343581+0000","modified":"2024-03-24T06:51:15.106526+0000","last_up_change":"2024-03-24T06:35:57.543453+0000","last_in_change":"2024-03-24T06:35:37.431406+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T06:31:39.650101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T06:36:54.347015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T06:36:55.706800+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T06:36:59.747616+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b50e1d7a-25e6-45e4-8f2e-51198cd3c017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6803","nonce":671685493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6805","nonce":671685493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6809","nonce":671685493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":671685493},{"type":"v1","addr":"172.21.15.22:6807","nonce":671685493}]},"public_addr":"172.21.15.22:6803/671685493","cluster_addr":"172.21.15.22:6805/671685493","heartbeat_back_addr":"172.21.15.22:6809/671685493","heartbeat_front_addr":"172.21.15.22:6807/671685493","state":["exists","up"]},{"osd":1,"uuid":"572a8c88-6fa5-4e41-9c09-562f3cdd36f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6801","nonce":2665804124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6803","nonce":2665804124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6807","nonce":2665804124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2665804124},{"type":"v1","addr":"172.21.15.143:6805","nonce":2665804124}]},"public_addr":"172.21.15.143:6801/2665804124","cluster_addr":"172.21.15.143:6803/2665804124","heartbeat_back_addr":"172.21.15.143:6807/2665804124","heartbeat_front_addr":"172.21.15.143:6805/2665804124","state":["exists","up"]},{"osd":2,"uuid":"43917ed1-92cb-42b5-98e4-a92deee31e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6801","nonce":2561722475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6803","nonce":2561722475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6807","nonce":2561722475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2561722475},{"type":"v1","addr":"172.21.15.155:6805","nonce":2561722475}]},"public_addr":"172.21.15.155:6801/2561722475","cluster_addr":"172.21.15.155:6803/2561722475","heartbeat_back_addr":"172.21.15.155:6807/2561722475","heartbeat_front_addr":"172.21.15.155:6805/2561722475","state":["exists","up"]},{"osd":3,"uuid":"85fc0b5c-cb79-4dc9-84c8-95968df70ee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6811","nonce":3163117945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6813","nonce":3163117945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6817","nonce":3163117945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":3163117945},{"type":"v1","addr":"172.21.15.22:6815","nonce":3163117945}]},"public_addr":"172.21.15.22:6811/3163117945","cluster_addr":"172.21.15.22:6813/3163117945","heartbeat_back_addr":"172.21.15.22:6817/3163117945","heartbeat_front_addr":"172.21.15.22:6815/3163117945","state":["exists","up"]},{"osd":4,"uuid":"c774c865-b9c3-4054-a328-e0c3a3c84d56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6809","nonce":4206631935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6811","nonce":4206631935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6815","nonce":4206631935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":4206631935},{"type":"v1","addr":"172.21.15.143:6813","nonce":4206631935}]},"public_addr":"172.21.15.143:6809/4206631935","cluster_addr":"172.21.15.143:6811/4206631935","heartbeat_back_addr":"172.21.15.143:6815/4206631935","heartbeat_front_addr":"172.21.15.143:6813/4206631935","state":["exists","up"]},{"osd":5,"uuid":"c149b822-fc96-49aa-bb16-e1318e3572e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6809","nonce":455132245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6811","nonce":455132245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6815","nonce":455132245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":455132245},{"type":"v1","addr":"172.21.15.155:6813","nonce":455132245}]},"public_addr":"172.21.15.155:6809/455132245","cluster_addr":"172.21.15.155:6811/455132245","heartbeat_back_addr":"172.21.15.155:6815/455132245","heartbeat_front_addr":"172.21.15.155:6813/455132245","state":["exists","up"]},{"osd":6,"uuid":"7b5e6e76-f3b1-4d5b-855c-3e22b8bcb9b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6819","nonce":3272271064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6821","nonce":3272271064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6825","nonce":3272271064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":3272271064},{"type":"v1","addr":"172.21.15.22:6823","nonce":3272271064}]},"public_addr":"172.21.15.22:6819/3272271064","cluster_addr":"172.21.15.22:6821/3272271064","heartbeat_back_addr":"172.21.15.22:6825/3272271064","heartbeat_front_addr":"172.21.15.22:6823/3272271064","state":["exists","up"]},{"osd":7,"uuid":"1f669e98-5416-456f-8f7c-e3d8fd6608c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6817","nonce":1253244511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6819","nonce":1253244511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6823","nonce":1253244511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":1253244511},{"type":"v1","addr":"172.21.15.143:6821","nonce":1253244511}]},"public_addr":"172.21.15.143:6817/1253244511","cluster_addr":"172.21.15.143:6819/1253244511","heartbeat_back_addr":"172.21.15.143:6823/1253244511","heartbeat_front_addr":"172.21.15.143:6821/1253244511","state":["exists","up"]},{"osd":8,"uuid":"75386ae9-abf1-4061-b5a6-ae879ab75235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6817","nonce":3178166768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6819","nonce":3178166768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6823","nonce":3178166768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3178166768},{"type":"v1","addr":"172.21.15.155:6821","nonce":3178166768}]},"public_addr":"172.21.15.155:6817/3178166768","cluster_addr":"172.21.15.155:6819/3178166768","heartbeat_back_addr":"172.21.15.155:6823/3178166768","heartbeat_front_addr":"172.21.15.155:6821/3178166768","state":["exists","up"]},{"osd":9,"uuid":"0937ff6c-3ac5-4da0-b56c-f8286d053eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6827","nonce":2177893113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6829","nonce":2177893113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6833","nonce":2177893113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":2177893113},{"type":"v1","addr":"172.21.15.22:6831","nonce":2177893113}]},"public_addr":"172.21.15.22:6827/2177893113","cluster_addr":"172.21.15.22:6829/2177893113","heartbeat_back_addr":"172.21.15.22:6833/2177893113","heartbeat_front_addr":"172.21.15.22:6831/2177893113","state":["exists","up"]},{"osd":10,"uuid":"095f0ee4-d45c-48ea-961d-f60c8c0869bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6825","nonce":2026852023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6827","nonce":2026852023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6831","nonce":2026852023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2026852023},{"type":"v1","addr":"172.21.15.143:6829","nonce":2026852023}]},"public_addr":"172.21.15.143:6825/2026852023","cluster_addr":"172.21.15.143:6827/2026852023","heartbeat_back_addr":"172.21.15.143:6831/2026852023","heartbeat_front_addr":"172.21.15.143:6829/2026852023","state":["exists","up"]},{"osd":11,"uuid":"4a545a5d-f6ad-4d53-9eb9-80718c3a7fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6825","nonce":2681113949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6827","nonce":2681113949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6831","nonce":2681113949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":2681113949},{"type":"v1","addr":"172.21.15.155:6829","nonce":2681113949}]},"public_addr":"172.21.15.155:6825/2681113949","cluster_addr":"172.21.15.155:6827/2681113949","heartbeat_back_addr":"172.21.15.155:6831/2681113949","heartbeat_front_addr":"172.21.15.155:6829/2681113949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:30:48.715339+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:13.375908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:31:37.348086+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:09.188727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:32:34.835969+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:00.240672+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:33:35.564134+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:01.948474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:34:27.900095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:02.956016+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:29.972084+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T06:35:55.731280+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/673082346":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/3288864197":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/440133497":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6800/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:6801/503368228":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/896325545":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/4236060832":"2024-03-25T06:40:00.112683+0000","172.21.15.22:0/2695260292":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6801/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:6800/2009113619":"2024-03-25T06:28:54.941425+0000","172.21.15.22:0/2447624484":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6800/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:6801/37818756":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1811358368":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/3441957019":"2024-03-25T06:28:20.396453+0000","172.21.15.22:0/1555230471":"2024-03-25T06:28:54.941425+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T06:51:27.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 msgr2=0x7f87fc0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 0x7f87fc0a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8808005ab0 tx=0x7f8808005fd0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 msgr2=0x7f8814101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f880c00eaa0 tx=0x7f880c00ef60 comp rx=0 tx=0).stop 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 shutdown_connections 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f87fc0a2260 0x7f87fc0a4720 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8814107600 0x7f8814101210 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814069230 0x7f8814100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.256 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 --2- 172.21.15.22:0/394992564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8814068870 0x7f8814100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 >> 172.21.15.22:0/394992564 conn(0x7f8814076020 msgr2=0x7f88140ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:27.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 shutdown_connections 2024-03-24T06:51:27.257 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.255+0000 7f881a400640 1 -- 172.21.15.22:0/394992564 wait complete. 2024-03-24T06:51:27.273 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T06:51:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/3951786721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/2424525877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.473+0000 7f7d6d400640 1 -- 172.21.15.22:0/2010650185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6810ded0 msgr2=0x7f7d681102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.473+0000 7f7d6d400640 1 --2- 172.21.15.22:0/2010650185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6810ded0 0x7f7d681102c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009ca0 tx=0x7f7d6002c100 comp rx=0 tx=0).stop 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 -- 172.21.15.22:0/2010650185 shutdown_connections 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 --2- 172.21.15.22:0/2010650185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d6810ded0 0x7f7d681102c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 --2- 172.21.15.22:0/2010650185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d68100cd0 0x7f7d6810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 --2- 172.21.15.22:0/2010650185 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 0x7f7d68100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 -- 172.21.15.22:0/2010650185 >> 172.21.15.22:0/2010650185 conn(0x7f7d680fbf60 msgr2=0x7f7d680fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:27.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 -- 172.21.15.22:0/2010650185 shutdown_connections 2024-03-24T06:51:27.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.474+0000 7f7d6d400640 1 -- 172.21.15.22:0/2010650185 wait complete. 2024-03-24T06:51:27.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 Processor -- start 2024-03-24T06:51:27.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 -- start start 2024-03-24T06:51:27.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 0x7f7d68199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.476 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 0x7f7d6819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6819ec70 con 0x7f7d6810ded0 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6819ede0 con 0x7f7d68100cd0 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d6d400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6819efa0 con 0x7f7d68100320 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.475+0000 7f7d67400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 0x7f7d68199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.22:55464/0 (socket says 172.21.15.22:55464) 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 -- 172.21.15.22:0/1174178823 learned_addr learned my addr 172.21.15.22:0/1174178823 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d6ca00640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 0x7f7d6819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 -- 172.21.15.22:0/1174178823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 msgr2=0x7f7d68199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 0x7f7d68199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 -- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 msgr2=0x7f7d6819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 0x7f7d6819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 -- 172.21.15.22:0/1174178823 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d60009900 con 0x7f7d68100cd0 2024-03-24T06:51:27.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d6ca00640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 0x7f7d6819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d67400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400d950 tx=0x7f7d5400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.476+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54009a30 con 0x7f7d68100cd0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.477+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d54009b90 con 0x7f7d68100cd0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.477+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54004a00 con 0x7f7d68100cd0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.477+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6819f280 con 0x7f7d68100cd0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.477+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d681a6b00 con 0x7f7d68100cd0 2024-03-24T06:51:27.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.478+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d68068420 con 0x7f7d68100cd0 2024-03-24T06:51:27.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.480+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f7d54009d00 con 0x7f7d68100cd0 2024-03-24T06:51:27.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.482+0000 7f7d64c00640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 0x7f7d4c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:27.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.482+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9467+0+0 (secure 0 0 0) 0x7f7d540cddc0 con 0x7f7d68100cd0 2024-03-24T06:51:27.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.482+0000 7f7d67e00640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 0x7f7d4c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:27.483 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.483+0000 7f7d67e00640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 0x7f7d4c0a4770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009bd0 tx=0x7f7d58005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:27.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.486+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7d54096640 con 0x7f7d68100cd0 2024-03-24T06:51:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/3951786721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/2424525877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/3951786721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/2424525877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T06:51:27.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.779+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7d6819b100 con 0x7f7d68100cd0 2024-03-24T06:51:27.791 INFO:teuthology.orchestra.run.smithi022.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T06:51:27.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.789+0000 7f7d64c00640 1 -- 172.21.15.22:0/1174178823 <== mon.2 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f7d54099d00 con 0x7f7d68100cd0 2024-03-24T06:51:27.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 msgr2=0x7f7d4c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 0x7f7d4c0a4770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009bd0 tx=0x7f7d58005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 msgr2=0x7f7d6819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400d950 tx=0x7f7d5400de10 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 shutdown_connections 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f7d4c0a22b0 0x7f7d4c0a4770 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7d6810ded0 0x7f7d6819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d68100cd0 0x7f7d6819a340 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 --2- 172.21.15.22:0/1174178823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d68100320 0x7f7d68199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:27.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 >> 172.21.15.22:0/1174178823 conn(0x7f7d680fbf60 msgr2=0x7f7d6810f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:27.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 shutdown_connections 2024-03-24T06:51:27.795 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:27.793+0000 7f7d6d400640 1 -- 172.21.15.22:0/1174178823 wait complete. 2024-03-24T06:51:27.814 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T06:51:27.910 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a[31999]: 2024-03-24T06:51:27.780+0000 7f2a0ac00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T06:51:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.011+0000 7fb061400640 1 -- 172.21.15.22:0/3240126207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 msgr2=0x7fb05c105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.011+0000 7fb061400640 1 --2- 172.21.15.22:0/3240126207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c105570 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb048009900 tx=0x7fb04802bcd0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 -- 172.21.15.22:0/3240126207 shutdown_connections 2024-03-24T06:51:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 --2- 172.21.15.22:0/3240126207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 --2- 172.21.15.22:0/3240126207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c105570 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 --2- 172.21.15.22:0/3240126207 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 -- 172.21.15.22:0/3240126207 >> 172.21.15.22:0/3240126207 conn(0x7fb05c077770 msgr2=0x7fb05c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 -- 172.21.15.22:0/3240126207 shutdown_connections 2024-03-24T06:51:28.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.012+0000 7fb061400640 1 -- 172.21.15.22:0/3240126207 wait complete. 2024-03-24T06:51:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 Processor -- start 2024-03-24T06:51:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 -- start start 2024-03-24T06:51:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:28.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c1a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1a3200 con 0x7fb05c100790 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1a3370 con 0x7fb05c105ab0 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb061400640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1a3530 con 0x7fb05c1010b0 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.013+0000 7fb05a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34344/0 (socket says 172.21.15.22:34344) 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 -- 172.21.15.22:0/1080443635 learned_addr learned my addr 172.21.15.22:0/1080443635 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:28.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb059e00640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05b200640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c1a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 -- 172.21.15.22:0/1080443635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 msgr2=0x7fb05c19e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c19e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 -- 172.21.15.22:0/1080443635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 msgr2=0x7fb05c1a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c1a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 -- 172.21.15.22:0/1080443635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb048009560 con 0x7fb05c100790 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb059e00640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05b200640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c1a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T06:51:28.017 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.014+0000 7fb05a800640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00e9f0 tx=0x7fb04c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.015+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c009700 con 0x7fb05c100790 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.015+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb04c00cd80 con 0x7fb05c100790 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.015+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c1a3810 con 0x7fb05c100790 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.015+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c018920 con 0x7fb05c100790 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.015+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c1ab090 con 0x7fb05c100790 2024-03-24T06:51:28.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.016+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c067c70 con 0x7fb05c100790 2024-03-24T06:51:28.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.020+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7fb04c018a80 con 0x7fb05c100790 2024-03-24T06:51:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.021+0000 7fb053400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 0x7fb0400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.022+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9809+0+0 (secure 0 0 0) 0x7fb04c014070 con 0x7fb05c100790 2024-03-24T06:51:28.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.022+0000 7fb059e00640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 0x7fb0400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.023+0000 7fb059e00640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 0x7fb0400a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb048005ab0 tx=0x7fb048005a20 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:28.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.023+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb04c097610 con 0x7fb05c100790 2024-03-24T06:51:28.157 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.157 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mds[65672]: -- 172.21.15.143:0/1020469562 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c9a68521a0 con 0x55c9a6828800 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: starting mds.k at 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-mds[67898]: -- 172.21.15.155:0/3360435293 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556145fc41a0 con 0x556145f9ac00 2024-03-24T06:51:28.158 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: starting mds.i at 2024-03-24T06:51:28.159 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.159 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.159 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mds[66617]: -- 172.21.15.143:0/4181458872 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561fe927a1a0 con 0x561fe9251000 2024-03-24T06:51:28.159 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: starting mds.b at 2024-03-24T06:51:28.159 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: starting mds.c at 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e[65990]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e[65990]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mds[65994]: -- 172.21.15.143:0/3094558698 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d15c3a1a0 con 0x557d15c11000 2024-03-24T06:51:28.160 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e[65990]: starting mds.e at 2024-03-24T06:51:28.161 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.161 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.161 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-mds[67280]: -- 172.21.15.155:0/836113257 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ff81a21a0 con 0x560ff8179000 2024-03-24T06:51:28.161 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: starting mds.f at 2024-03-24T06:51:28.161 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mds[66298]: -- 172.21.15.143:0/1059847675 <== mon.2 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56300b819080 con 0x56300b7e3000 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-mds[66298]: -- 172.21.15.143:0/1059847675 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56300b80c1a0 con 0x56300b7e3000 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:51:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: starting mds.h at 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.162 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-mds[66969]: -- 172.21.15.155:0/2793880650 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d33a76e1a0 con 0x55d33a744800 2024-03-24T06:51:28.163 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:51:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: starting mds.l at 2024-03-24T06:51:28.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.270+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb05c117e80 con 0x7fb05c100790 2024-03-24T06:51:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: pgmap v358: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 942 B/s wr, 13 op/s 2024-03-24T06:51:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/394992564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:28.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1174178823' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T06:51:28.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:28 smithi022 ceph-mon[32003]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-mds[72201]: -- 172.21.15.22:0/1440575194 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d5269aa1a0 con 0x55d526980c00 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: starting mds.j at 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: ignoring --setuser ceph since I am not root 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: ignoring --setgroup ceph since I am not root 2024-03-24T06:51:28.401 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-mds[73570]: -- 172.21.15.22:0/3180227175 <== mon.2 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a91a82a1a0 con 0x55a91a800c00 2024-03-24T06:51:28.402 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:51:27 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: starting mds.a at 2024-03-24T06:51:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: pgmap v358: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 942 B/s wr, 13 op/s 2024-03-24T06:51:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/394992564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:28.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1174178823' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T06:51:28.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:28 smithi143 ceph-mon[36520]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T06:51:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: pgmap v358: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 942 B/s wr, 13 op/s 2024-03-24T06:51:28.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/394992564' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1174178823' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T06:51:28.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:28 smithi155 ceph-mon[36262]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T06:51:28.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.786+0000 7fb053400640 1 -- 172.21.15.22:0/1080443635 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7fb04c09acd0 con 0x7fb05c100790 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 msgr2=0x7fb0400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 0x7fb0400a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb048005ab0 tx=0x7fb048005a20 comp rx=0 tx=0).stop 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 msgr2=0x7fb05c19e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00e9f0 tx=0x7fb04c00eeb0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 shutdown_connections 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7fb0400a2230 0x7fb0400a46f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05c105ab0 0x7fb05c1a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb05c1010b0 0x7fb05c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 --2- 172.21.15.22:0/1080443635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb05c100790 0x7fb05c19e340 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:28.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 >> 172.21.15.22:0/1080443635 conn(0x7fb05c077770 msgr2=0x7fb05c103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:28.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 shutdown_connections 2024-03-24T06:51:28.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:28.789+0000 7fb061400640 1 -- 172.21.15.22:0/1080443635 wait complete. 2024-03-24T06:51:28.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T06:51:29.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 -- 172.21.15.22:0/65417036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35081096c0 msgr2=0x7f3508109aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:29.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 --2- 172.21.15.22:0/65417036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35081096c0 0x7f3508109aa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009930 tx=0x7f34f802bc30 comp rx=0 tx=0).stop 2024-03-24T06:51:29.015 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 -- 172.21.15.22:0/65417036 shutdown_connections 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 --2- 172.21.15.22:0/65417036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3508104070 0x7f3508110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 --2- 172.21.15.22:0/65417036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f3508103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 --2- 172.21.15.22:0/65417036 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f35081096c0 0x7f3508109aa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 -- 172.21.15.22:0/65417036 >> 172.21.15.22:0/65417036 conn(0x7f35080ff180 msgr2=0x7f35081015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 -- 172.21.15.22:0/65417036 shutdown_connections 2024-03-24T06:51:29.016 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.015+0000 7f3507200640 1 -- 172.21.15.22:0/65417036 wait complete. 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 Processor -- start 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 -- start start 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3508104070 0x7f350819ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35081096c0 0x7f3508199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f350819a300 con 0x7f3508104070 2024-03-24T06:51:29.018 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f350819a470 con 0x7f35081096c0 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.016+0000 7f3507200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f350819a630 con 0x7f35081036b0 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3508104070 0x7f350819ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:38714/0 (socket says 172.21.15.22:38714) 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 -- 172.21.15.22:0/1687487236 learned_addr learned my addr 172.21.15.22:0/1687487236 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3506800640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35081096c0 0x7f3508199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 -- 172.21.15.22:0/1687487236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35081096c0 msgr2=0x7f3508199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35081096c0 0x7f3508199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:29.019 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 -- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3508104070 msgr2=0x7f350819ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3508104070 0x7f350819ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 -- 172.21.15.22:0/1687487236 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f8009560 con 0x7f35081036b0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.017+0000 7f3505e00640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009230 tx=0x7f34f802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.018+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8004350 con 0x7f35081036b0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.018+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f80044b0 con 0x7f35081036b0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.018+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350819a8b0 con 0x7f35081036b0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.018+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8003d70 con 0x7f35081036b0 2024-03-24T06:51:29.020 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.018+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35081a69e0 con 0x7f35081036b0 2024-03-24T06:51:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.019+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3508104df0 con 0x7f35081036b0 2024-03-24T06:51:29.021 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.021+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f34f8004df0 con 0x7f35081036b0 2024-03-24T06:51:29.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.022+0000 7f34fea00640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 0x7f34ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.022+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9809+0+0 (secure 0 0 0) 0x7f34f80f0cd0 con 0x7f35081036b0 2024-03-24T06:51:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.022+0000 7f3505400640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 0x7f34ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:29.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.023+0000 7f3505400640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 0x7f34ec0a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f350819b300 tx=0x7f34f4005f90 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.024+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f34f80b9500 con 0x7f35081036b0 2024-03-24T06:51:29.225 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:29.225+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f350819aaa0 con 0x7f35081036b0 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: Cluster is now healthy 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:51:29.429 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:51:29.430 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6832/3468274572,v1:172.21.15.143:6835/3468274572] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6833/462915913,v1:172.21.15.155:6835/462915913] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6834/3557939340,v1:172.21.15.143:6837/3557939340] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6836/3035235547,v1:172.21.15.155:6837/3035235547] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6838/4217759618,v1:172.21.15.143:6839/4217759618] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6838/1161047411,v1:172.21.15.155:6839/1161047411] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.155:6832/395323673,v1:172.21.15.155:6834/395323673] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.143:6833/334097643,v1:172.21.15.143:6836/334097643] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6834/76771825,v1:172.21.15.22:6840/76771825] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: mds.? [v2:172.21.15.22:6835/4144725196,v1:172.21.15.22:6841/4144725196] up:boot 2024-03-24T06:51:29.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:29 smithi143 ceph-mon[36520]: fsmap 12 up:standby 2024-03-24T06:51:29.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: Cluster is now healthy 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:51:29.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6832/3468274572,v1:172.21.15.143:6835/3468274572] up:boot 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6833/462915913,v1:172.21.15.155:6835/462915913] up:boot 2024-03-24T06:51:29.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6834/3557939340,v1:172.21.15.143:6837/3557939340] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6836/3035235547,v1:172.21.15.155:6837/3035235547] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6838/4217759618,v1:172.21.15.143:6839/4217759618] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6838/1161047411,v1:172.21.15.155:6839/1161047411] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.155:6832/395323673,v1:172.21.15.155:6834/395323673] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.143:6833/334097643,v1:172.21.15.143:6836/334097643] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6834/76771825,v1:172.21.15.22:6840/76771825] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: mds.? [v2:172.21.15.22:6835/4144725196,v1:172.21.15.22:6841/4144725196] up:boot 2024-03-24T06:51:29.530 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:29 smithi155 ceph-mon[36262]: fsmap 12 up:standby 2024-03-24T06:51:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T06:51:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T06:51:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T06:51:29.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: Cluster is now healthy 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1080443635' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T06:51:29.652 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: from='mgr.25492 172.21.15.22:0/3228018484' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6832/3468274572,v1:172.21.15.143:6835/3468274572] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6833/462915913,v1:172.21.15.155:6835/462915913] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6834/3557939340,v1:172.21.15.143:6837/3557939340] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6836/3035235547,v1:172.21.15.155:6837/3035235547] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6838/4217759618,v1:172.21.15.143:6839/4217759618] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6838/1161047411,v1:172.21.15.155:6839/1161047411] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.155:6832/395323673,v1:172.21.15.155:6834/395323673] up:boot 2024-03-24T06:51:29.653 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.143:6833/334097643,v1:172.21.15.143:6836/334097643] up:boot 2024-03-24T06:51:29.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6834/76771825,v1:172.21.15.22:6840/76771825] up:boot 2024-03-24T06:51:29.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: mds.? [v2:172.21.15.22:6835/4144725196,v1:172.21.15.22:6841/4144725196] up:boot 2024-03-24T06:51:29.654 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:29 smithi022 ceph-mon[32003]: fsmap 12 up:standby 2024-03-24T06:51:30.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.166+0000 7f34fea00640 1 -- 172.21.15.22:0/1687487236 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) ==== 181+0+0 (secure 0 0 0) 0x7f34f80bcbc0 con 0x7f35081036b0 2024-03-24T06:51:30.168 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_metadata' removed 2024-03-24T06:51:30.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 msgr2=0x7f34ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:30.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 0x7f34ec0a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f350819b300 tx=0x7f34f4005f90 comp rx=0 tx=0).stop 2024-03-24T06:51:30.172 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 msgr2=0x7f350819fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f34f8009230 tx=0x7f34f802c6c0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 shutdown_connections 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f34ec0a2230 0x7f34ec0a46f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35081096c0 0x7f3508199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3508104070 0x7f350819ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.173 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 --2- 172.21.15.22:0/1687487236 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35081036b0 0x7f350819fa60 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.172+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 >> 172.21.15.22:0/1687487236 conn(0x7f35080ff180 msgr2=0x7f35080ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:30.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.173+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 shutdown_connections 2024-03-24T06:51:30.174 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.174+0000 7f3507200640 1 -- 172.21.15.22:0/1687487236 wait complete. 2024-03-24T06:51:30.200 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 -- 172.21.15.22:0/3355662416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0108870 msgr2=0x7efde010c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 --2- 172.21.15.22:0/3355662416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0108870 0x7efde010c830 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009930 tx=0x7efdc802bc30 comp rx=0 tx=0).stop 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 -- 172.21.15.22:0/3355662416 shutdown_connections 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 --2- 172.21.15.22:0/3355662416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde0114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 --2- 172.21.15.22:0/3355662416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0108870 0x7efde010c830 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 --2- 172.21.15.22:0/3355662416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde0107ec0 0x7efde01082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.398+0000 7efddf200640 1 -- 172.21.15.22:0/3355662416 >> 172.21.15.22:0/3355662416 conn(0x7efde0077530 msgr2=0x7efde0106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:30.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.399+0000 7efddf200640 1 -- 172.21.15.22:0/3355662416 shutdown_connections 2024-03-24T06:51:30.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.399+0000 7efddf200640 1 -- 172.21.15.22:0/3355662416 wait complete. 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 Processor -- start 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 -- start start 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde01a2b20 0x7efde01a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:30.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7efde01a3510 con 0x7efde0107ec0 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efde01a3680 con 0x7efde01a2b20 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddf200640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efde01a37f0 con 0x7efde010cf10 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddde00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddde00640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34370/0 (socket says 172.21.15.22:34370) 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efddde00640 1 -- 172.21.15.22:0/1519956888 learned_addr learned my addr 172.21.15.22:0/1519956888 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.400+0000 7efdde800640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde01a2b20 0x7efde01a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efdd7e00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 -- 172.21.15.22:0/1519956888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 msgr2=0x7efde019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:30.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 -- 172.21.15.22:0/1519956888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde01a2b20 msgr2=0x7efde01a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde01a2b20 0x7efde01a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 -- 172.21.15.22:0/1519956888 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc8009560 con 0x7efde0107ec0 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efdd7e00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efddde00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdd000e9f0 tx=0x7efdd000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.401+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0009700 con 0x7efde0107ec0 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.402+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd000cd80 con 0x7efde0107ec0 2024-03-24T06:51:30.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.402+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde01aadd0 con 0x7efde0107ec0 2024-03-24T06:51:30.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.402+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0018920 con 0x7efde0107ec0 2024-03-24T06:51:30.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.402+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde01ab1f0 con 0x7efde0107ec0 2024-03-24T06:51:30.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.403+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde0068870 con 0x7efde0107ec0 2024-03-24T06:51:30.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.405+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7efdd0018a80 con 0x7efde0107ec0 2024-03-24T06:51:30.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.406+0000 7efdd6a00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:30.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.406+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9284+0+0 (secure 0 0 0) 0x7efdd0014070 con 0x7efde0107ec0 2024-03-24T06:51:30.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.406+0000 7efdd7e00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:30.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.407+0000 7efdd7e00640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 0x7efdc40a46f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009230 tx=0x7efdc8005fd0 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:30.411 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.411+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efdd0053280 con 0x7efde0107ec0 2024-03-24T06:51:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:30 smithi143 ceph-mon[36520]: pgmap v360: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-24T06:51:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:30 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1687487236' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:30.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:30 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:30 smithi155 ceph-mon[36262]: pgmap v360: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-24T06:51:30.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:30 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1687487236' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:30.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:30 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:30.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:30.614+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efde019f5b0 con 0x7efde0107ec0 2024-03-24T06:51:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:30 smithi022 ceph-mon[32003]: pgmap v360: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-03-24T06:51:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:30 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1687487236' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:30.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:30 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:31.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.188+0000 7efdd6a00640 1 -- 172.21.15.22:0/1519956888 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) ==== 169+0+0 (secure 0 0 0) 0x7efdd0097140 con 0x7efde0107ec0 2024-03-24T06:51:31.189 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_data' removed 2024-03-24T06:51:31.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 msgr2=0x7efdc40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 0x7efdc40a46f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009230 tx=0x7efdc8005fd0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 msgr2=0x7efde019e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdd000e9f0 tx=0x7efdd000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 shutdown_connections 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efde01a2b20 0x7efde01a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efde010cf10 0x7efde019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 --2- 172.21.15.22:0/1519956888 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efde0107ec0 0x7efde019e2b0 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdd000e9f0 tx=0x7efdd000eeb0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.194+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 >> 172.21.15.22:0/1519956888 conn(0x7efde0077530 msgr2=0x7efde0079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:31.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.196+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 shutdown_connections 2024-03-24T06:51:31.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.197+0000 7efddf200640 1 -- 172.21.15.22:0/1519956888 wait complete. 2024-03-24T06:51:31.224 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-24T06:51:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 -- 172.21.15.22:0/2399049547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 msgr2=0x7f7868108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 --2- 172.21.15.22:0/2399049547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f7868108150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7854009900 tx=0x7f785402bcd0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 -- 172.21.15.22:0/2399049547 shutdown_connections 2024-03-24T06:51:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 --2- 172.21.15.22:0/2399049547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 0x7f7868114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.426 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 --2- 172.21.15.22:0/2399049547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f7868108150 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 --2- 172.21.15.22:0/2399049547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 0x7f786810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.425+0000 7f786e000640 1 -- 172.21.15.22:0/2399049547 >> 172.21.15.22:0/2399049547 conn(0x7f78680774e0 msgr2=0x7f7868106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.426+0000 7f786e000640 1 -- 172.21.15.22:0/2399049547 shutdown_connections 2024-03-24T06:51:31.427 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.426+0000 7f786e000640 1 -- 172.21.15.22:0/2399049547 wait complete. 2024-03-24T06:51:31.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 Processor -- start 2024-03-24T06:51:31.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 -- start start 2024-03-24T06:51:31.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 0x7f786819e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 0x7f78681a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 -- 0x7f78681a3140 con 0x7f7868107cf0 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78681a32b0 con 0x7f7868108690 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.427+0000 7f786e000640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78681a3470 con 0x7f786810ddb0 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34388/0 (socket says 172.21.15.22:34388) 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 -- 172.21.15.22:0/1933513148 learned_addr learned my addr 172.21.15.22:0/1933513148 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T06:51:31.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7866a00640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 0x7f786819e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867e00640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 0x7f78681a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 -- 172.21.15.22:0/1933513148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 msgr2=0x7f78681a2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 0x7f78681a2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 -- 172.21.15.22:0/1933513148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 msgr2=0x7f786819e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 0x7f786819e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:31.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.428+0000 7f7867400640 1 -- 172.21.15.22:0/1933513148 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7854009560 con 0x7f7868107cf0 2024-03-24T06:51:31.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:31 smithi143 ceph-mon[36520]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:31.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:31 smithi143 ceph-mon[36520]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T06:51:31.431 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:31 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f7867400640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f785800e9f0 tx=0x7f785800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858009720 con 0x7f7868107cf0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785800cd80 con 0x7f7868107cf0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78681a3750 con 0x7f7868107cf0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858010690 con 0x7f7868107cf0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.429+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78681aafd0 con 0x7f7868107cf0 2024-03-24T06:51:31.432 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.430+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7868068950 con 0x7f7868107cf0 2024-03-24T06:51:31.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.432+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 27) ==== 136887+0+0 (secure 0 0 0) 0x7f78580046c0 con 0x7f7868107cf0 2024-03-24T06:51:31.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.433+0000 7f785fe00640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 0x7f784c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T06:51:31.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.433+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8754+0+0 (secure 0 0 0) 0x7f7858014070 con 0x7f7868107cf0 2024-03-24T06:51:31.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.434+0000 7f7866a00640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 0x7f784c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T06:51:31.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.435+0000 7f7866a00640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 0x7f784c0a4770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7854006010 tx=0x7f7854005e40 comp rx=0 tx=0).ready entity=mgr.25492 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T06:51:31.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.438+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7858096f90 con 0x7f7868107cf0 2024-03-24T06:51:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:31 smithi155 ceph-mon[36262]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:31.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:31 smithi155 ceph-mon[36262]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T06:51:31.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:31 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:31 smithi022 ceph-mon[32003]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:31 smithi022 ceph-mon[32003]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T06:51:31.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:31 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:31.656 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:31.656+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f78681101a0 con 0x7f7868107cf0 2024-03-24T06:51:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.191+0000 7f785fe00640 1 -- 172.21.15.22:0/1933513148 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v108) ==== 178+0+0 (secure 0 0 0) 0x7f785809a650 con 0x7f7868107cf0 2024-03-24T06:51:32.192 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_data_ec' removed 2024-03-24T06:51:32.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.195+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 msgr2=0x7f784c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:32.196 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.195+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 0x7f784c0a4770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7854006010 tx=0x7f7854005e40 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.195+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 msgr2=0x7f786819e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.195+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f785800e9f0 tx=0x7f785800edf0 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 shutdown_connections 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:6800/1309982651,v1:172.21.15.22:6801/1309982651] conn(0x7f784c0a22b0 0x7f784c0a4770 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f786810ddb0 0x7f78681a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7868108690 0x7f786819e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 --2- 172.21.15.22:0/1933513148 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7868107cf0 0x7f786819e2d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T06:51:32.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 >> 172.21.15.22:0/1933513148 conn(0x7f78680774e0 msgr2=0x7f7868078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T06:51:32.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.196+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 shutdown_connections 2024-03-24T06:51:32.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T06:51:32.197+0000 7f786e000640 1 -- 172.21.15.22:0/1933513148 wait complete. 2024-03-24T06:51:32.231 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:51:32.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:51:32.404 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:51:32.455 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:32 smithi155 ceph-mon[36262]: pgmap v362: 97 pgs: 97 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:51:32.455 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:32 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:32.455 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:32 smithi155 ceph-mon[36262]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T06:51:32.455 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:32 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:32.456 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T06:51:32.456 DEBUG:teuthology.orchestra.run.smithi022:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T06:51:32.534 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T06:51:32.564 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T06:51:32.592 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T06:51:32.593 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T06:51:32.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a 2024-03-24T06:51:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:32 smithi022 ceph-mon[32003]: pgmap v362: 97 pgs: 97 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:51:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:32 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:32.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:32 smithi022 ceph-mon[32003]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T06:51:32.651 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:32 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:32 smithi143 ceph-mon[36520]: pgmap v362: 97 pgs: 97 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T06:51:32.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:32 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1519956888' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:32.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:32 smithi143 ceph-mon[36520]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T06:51:32.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:32 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T06:51:33.149 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:32 smithi022 systemd[1]: Stopping Ceph mon.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:33 smithi155 ceph-mon[36262]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:33.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:33 smithi155 ceph-mon[36262]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T06:51:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 ceph-mon[32003]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 ceph-mon[32003]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T06:51:33.649 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a[31999]: 2024-03-24T06:51:33.386+0000 7f2a11a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:51:33.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a[31999]: 2024-03-24T06:51:33.386+0000 7f2a11a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T06:51:33.650 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 podman[103729]: 2024-03-24 06:51:33.460020883 +0000 UTC m=+0.254466253 container died c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T06:51:33.678 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:33 smithi143 ceph-mon[36520]: from='client.? 172.21.15.22:0/1933513148' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T06:51:33.679 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:33 smithi143 ceph-mon[36520]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T06:51:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 podman[103729]: 2024-03-24 06:51:33.94294732 +0000 UTC m=+0.737392690 container cleanup c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:51:34.399 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:33 smithi022 bash[103729]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a 2024-03-24T06:51:34.400 INFO:journalctl@ceph.mon.a.smithi022.stdout:Mar 24 06:51:34 smithi022 podman[103739]: 2024-03-24 06:51:34.193466587 +0000 UTC m=+0.712135344 container remove c6e194ac910d5896fac2224956b71c29f97861aa14370048c7648398fc0e4540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T06:51:34.699 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T06:51:35.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:35.015 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T06:51:35.015 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T06:51:35.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.b 2024-03-24T06:51:35.428 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:35 smithi143 systemd[1]: Stopping Ceph mon.b for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:35.928 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b[36516]: 2024-03-24T06:51:35.649+0000 7fcf6837c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:51:35.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b[36516]: 2024-03-24T06:51:35.649+0000 7fcf6837c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T06:51:35.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:35 smithi143 podman[72357]: 2024-03-24 06:51:35.797543246 +0000 UTC m=+0.334400859 container died afc985d51a41c8a62779659b1615ee7ea1859dcaa2739318781069475ca51004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T06:51:41.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:41 smithi143 podman[72357]: 2024-03-24 06:51:41.665016768 +0000 UTC m=+6.201874372 container cleanup afc985d51a41c8a62779659b1615ee7ea1859dcaa2739318781069475ca51004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T06:51:41.929 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:41 smithi143 bash[72357]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b 2024-03-24T06:51:45.051 INFO:journalctl@ceph.mon.b.smithi143.stdout:Mar 24 06:51:44 smithi143 podman[72367]: 2024-03-24 06:51:44.767530376 +0000 UTC m=+8.960420119 container remove afc985d51a41c8a62779659b1615ee7ea1859dcaa2739318781069475ca51004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T06:51:45.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T06:51:45.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:45.178 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T06:51:45.178 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T06:51:45.178 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.c 2024-03-24T06:51:45.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:45 smithi155 systemd[1]: Stopping Ceph mon.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:47.777 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:47 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c[36258]: 2024-03-24T06:51:47.467+0000 7fc09c4aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:51:47.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:47 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c[36258]: 2024-03-24T06:51:47.467+0000 7fc09c4aa700 -1 mon.c@1(probing) e3 *** Got Signal Terminated *** 2024-03-24T06:51:47.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:47 smithi155 podman[73472]: 2024-03-24 06:51:47.681819211 +0000 UTC m=+2.130314184 container died 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T06:51:53.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:52 smithi155 podman[73472]: 2024-03-24 06:51:52.590131019 +0000 UTC m=+7.038625987 container cleanup 9915a307e2878f96483159db9281df31ed9ffd5518b8444f0b00938d2f803fc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T06:51:53.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 24 06:51:52 smithi155 bash[73472]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mon-c 2024-03-24T06:51:53.986 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T06:51:54.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:54.048 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T06:51:54.048 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T06:51:54.048 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x 2024-03-24T06:51:54.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:54 smithi022 systemd[1]: Stopping Ceph mgr.x for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:55.077 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:51:54.661+0000 7ff476e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:51:55.077 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x[32231]: 2024-03-24T06:51:54.661+0000 7ff476e00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T06:51:55.077 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:54 smithi022 podman[103934]: 2024-03-24 06:51:54.756077011 +0000 UTC m=+0.267656735 container died c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T06:51:55.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:55 smithi022 podman[103934]: 2024-03-24 06:51:55.256733704 +0000 UTC m=+0.768313430 container cleanup c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T06:51:55.399 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:55 smithi022 bash[103934]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x 2024-03-24T06:51:55.823 INFO:journalctl@ceph.mgr.x.smithi022.stdout:Mar 24 06:51:55 smithi022 podman[103949]: 2024-03-24 06:51:55.617091844 +0000 UTC m=+0.837696135 container remove c73fd17ae3c8efa4cf27683d7be50ad103e840d1ea3024389043de2557180ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T06:51:55.858 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T06:51:55.965 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:55.965 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T06:51:55.966 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T06:51:55.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.y 2024-03-24T06:51:56.178 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:51:56 smithi143 systemd[1]: Stopping Ceph mgr.y for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:56.602 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:51:56 smithi143 podman[72589]: 2024-03-24 06:51:56.298608816 +0000 UTC m=+0.137132782 container died f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:51:56.906 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:51:56 smithi143 podman[72589]: 2024-03-24 06:51:56.703334911 +0000 UTC m=+0.541858879 container cleanup f2ffe76ce0a1e8ff5935ba3501d79d9e62b55296e4381f2e0aaed4779bffdda6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T06:51:56.906 INFO:journalctl@ceph.mgr.y.smithi143.stdout:Mar 24 06:51:56 smithi143 bash[72589]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-y 2024-03-24T06:51:57.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T06:51:57.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:57.131 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T06:51:57.132 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T06:51:57.132 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.z 2024-03-24T06:51:57.470 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:51:57 smithi155 systemd[1]: Stopping Ceph mgr.z for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:57.777 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:51:57 smithi155 podman[73604]: 2024-03-24 06:51:57.471256493 +0000 UTC m=+0.147206748 container died 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T06:51:58.278 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:51:58 smithi155 podman[73604]: 2024-03-24 06:51:58.059479093 +0000 UTC m=+0.735429356 container cleanup 3fc325da53bfaad6021109e7de241713df8e59b76810c7241cc89493772207eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T06:51:58.278 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 24 06:51:58 smithi155 bash[73604]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mgr-z 2024-03-24T06:51:58.467 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T06:51:58.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:51:58.506 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T06:51:58.507 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T06:51:58.507 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.0 2024-03-24T06:51:58.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:51:58 smithi022 systemd[1]: Stopping Ceph osd.0 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:51:59.399 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:51:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0[46316]: 2024-03-24T06:51:59.102+0000 7f9246a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:51:59.400 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:51:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0[46316]: 2024-03-24T06:51:59.102+0000 7f9246a00700 -1 osd.0 108 *** Got signal Terminated *** 2024-03-24T06:51:59.400 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:51:59 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0[46316]: 2024-03-24T06:51:59.102+0000 7f9246a00700 -1 osd.0 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:04 smithi022 podman[104067]: 2024-03-24 06:52:04.777661788 +0000 UTC m=+5.835296770 container died c0bdd3170b414ae4ad7e0f8ced807613048d00738f419966cba60aa2b3fe37c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:05.295 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:05 smithi022 podman[104067]: 2024-03-24 06:52:05.287637947 +0000 UTC m=+6.345272906 container cleanup c0bdd3170b414ae4ad7e0f8ced807613048d00738f419966cba60aa2b3fe37c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:52:05.295 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:05 smithi022 bash[104067]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0 2024-03-24T06:52:05.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:05 smithi022 podman[104084]: 2024-03-24 06:52:05.597531203 +0000 UTC m=+0.806741294 container remove c0bdd3170b414ae4ad7e0f8ced807613048d00738f419966cba60aa2b3fe37c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:52:06.827 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:06 smithi022 podman[104174]: 2024-03-24 06:52:06.409745104 +0000 UTC m=+0.044471823 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:06.828 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:06 smithi022 podman[104174]: 2024-03-24 06:52:06.540404372 +0000 UTC m=+0.175131125 container create 3f172ea2d696c25d7e8c2c8c20af69af5cfdd4683054861fc0ac019b489ad7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T06:52:06.828 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:06 smithi022 podman[104174]: 2024-03-24 06:52:06.827629042 +0000 UTC m=+0.462355798 container init 3f172ea2d696c25d7e8c2c8c20af69af5cfdd4683054861fc0ac019b489ad7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T06:52:07.149 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:06 smithi022 podman[104174]: 2024-03-24 06:52:06.839331736 +0000 UTC m=+0.474058512 container start 3f172ea2d696c25d7e8c2c8c20af69af5cfdd4683054861fc0ac019b489ad7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T06:52:07.150 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:06 smithi022 podman[104174]: 2024-03-24 06:52:06.951710247 +0000 UTC m=+0.586437046 container attach 3f172ea2d696c25d7e8c2c8c20af69af5cfdd4683054861fc0ac019b489ad7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T06:52:08.129 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 06:52:07 smithi022 podman[104174]: 2024-03-24 06:52:07.69603733 +0000 UTC m=+1.330764102 container died 3f172ea2d696c25d7e8c2c8c20af69af5cfdd4683054861fc0ac019b489ad7ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T06:52:08.549 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T06:52:08.684 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:08.684 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T06:52:08.684 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T06:52:08.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.1 2024-03-24T06:52:09.008 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:08 smithi143 systemd[1]: Stopping Ceph osd.1 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:09.428 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:09 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1[41155]: 2024-03-24T06:52:09.007+0000 7f71db896700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:09.428 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:09 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1[41155]: 2024-03-24T06:52:09.007+0000 7f71db896700 -1 osd.1 108 *** Got signal Terminated *** 2024-03-24T06:52:09.429 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:09 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1[41155]: 2024-03-24T06:52:09.007+0000 7f71db896700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:14.800 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:14 smithi143 podman[72715]: 2024-03-24 06:52:14.50007413 +0000 UTC m=+5.626607884 container died 098bde2ac34ee7c785526650be5ef4e034423a1e5cf9ca7baaf52ed1e7fb66bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T06:52:15.166 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:14 smithi143 podman[72715]: 2024-03-24 06:52:14.891086831 +0000 UTC m=+6.017620613 container cleanup 098bde2ac34ee7c785526650be5ef4e034423a1e5cf9ca7baaf52ed1e7fb66bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T06:52:15.167 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:14 smithi143 bash[72715]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1 2024-03-24T06:52:15.428 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72734]: 2024-03-24 06:52:15.167267446 +0000 UTC m=+0.670023290 container remove 098bde2ac34ee7c785526650be5ef4e034423a1e5cf9ca7baaf52ed1e7fb66bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:15.778 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72823]: 2024-03-24 06:52:15.362506339 +0000 UTC m=+0.017617519 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:15.779 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72823]: 2024-03-24 06:52:15.501670325 +0000 UTC m=+0.156781488 container create d91a6f1988fac42806374467ef3e882e33b9be7f8ad73f75442cd5871c2dca93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319) 2024-03-24T06:52:16.178 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72823]: 2024-03-24 06:52:15.779513972 +0000 UTC m=+0.434625143 container init d91a6f1988fac42806374467ef3e882e33b9be7f8ad73f75442cd5871c2dca93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T06:52:16.179 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72823]: 2024-03-24 06:52:15.782972591 +0000 UTC m=+0.438083772 container start d91a6f1988fac42806374467ef3e882e33b9be7f8ad73f75442cd5871c2dca93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T06:52:16.179 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:15 smithi143 podman[72823]: 2024-03-24 06:52:15.887909716 +0000 UTC m=+0.543020888 container attach d91a6f1988fac42806374467ef3e882e33b9be7f8ad73f75442cd5871c2dca93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T06:52:16.451 INFO:journalctl@ceph.osd.1.smithi143.stdout:Mar 24 06:52:16 smithi143 podman[72823]: 2024-03-24 06:52:16.186613361 +0000 UTC m=+0.841724539 container died d91a6f1988fac42806374467ef3e882e33b9be7f8ad73f75442cd5871c2dca93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T06:52:16.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T06:52:16.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:16.877 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T06:52:16.877 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T06:52:16.877 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.2 2024-03-24T06:52:17.196 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:16 smithi155 systemd[1]: Stopping Ceph osd.2 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:17.528 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2[41606]: 2024-03-24T06:52:17.196+0000 7f02a8aaa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:17.528 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2[41606]: 2024-03-24T06:52:17.196+0000 7f02a8aaa700 -1 osd.2 108 *** Got signal Terminated *** 2024-03-24T06:52:17.528 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2[41606]: 2024-03-24T06:52:17.196+0000 7f02a8aaa700 -1 osd.2 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:23.172 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:22 smithi155 podman[73728]: 2024-03-24 06:52:22.864745713 +0000 UTC m=+5.788529761 container died 0484b9372d44a272b8a39903b2418d23c1de085351c0ec1a08b82a114d867819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux ) 2024-03-24T06:52:23.523 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:23 smithi155 podman[73728]: 2024-03-24 06:52:23.305698638 +0000 UTC m=+6.229482680 container cleanup 0484b9372d44a272b8a39903b2418d23c1de085351c0ec1a08b82a114d867819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T06:52:23.523 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:23 smithi155 bash[73728]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2 2024-03-24T06:52:23.777 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:23 smithi155 podman[73756]: 2024-03-24 06:52:23.524099065 +0000 UTC m=+0.655419731 container remove 0484b9372d44a272b8a39903b2418d23c1de085351c0ec1a08b82a114d867819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T06:52:24.244 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:23 smithi155 podman[73846]: 2024-03-24 06:52:23.866389834 +0000 UTC m=+0.018048703 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:24.245 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:24 smithi155 podman[73846]: 2024-03-24 06:52:24.000034551 +0000 UTC m=+0.151693417 container create d25eb2d6c1420ac40b81fd37cbe73b4a1e6fcd06531f1df8983931bbbfefbe8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:52:24.527 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:24 smithi155 podman[73846]: 2024-03-24 06:52:24.244684595 +0000 UTC m=+0.396343466 container init d25eb2d6c1420ac40b81fd37cbe73b4a1e6fcd06531f1df8983931bbbfefbe8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T06:52:24.528 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:24 smithi155 podman[73846]: 2024-03-24 06:52:24.248462469 +0000 UTC m=+0.400121317 container start d25eb2d6c1420ac40b81fd37cbe73b4a1e6fcd06531f1df8983931bbbfefbe8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:52:24.528 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:24 smithi155 podman[73846]: 2024-03-24 06:52:24.361577686 +0000 UTC m=+0.513236549 container attach d25eb2d6c1420ac40b81fd37cbe73b4a1e6fcd06531f1df8983931bbbfefbe8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T06:52:24.997 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 24 06:52:24 smithi155 podman[73846]: 2024-03-24 06:52:24.65422876 +0000 UTC m=+0.805887618 container died d25eb2d6c1420ac40b81fd37cbe73b4a1e6fcd06531f1df8983931bbbfefbe8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:25.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T06:52:25.379 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:25.379 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T06:52:25.380 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T06:52:25.380 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.3 2024-03-24T06:52:25.899 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:25 smithi022 systemd[1]: Stopping Ceph osd.3 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:26.282 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:25 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3[53302]: 2024-03-24T06:52:25.932+0000 7ff5c8600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:26.283 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:25 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3[53302]: 2024-03-24T06:52:25.932+0000 7ff5c8600700 -1 osd.3 108 *** Got signal Terminated *** 2024-03-24T06:52:26.283 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:25 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3[53302]: 2024-03-24T06:52:25.932+0000 7ff5c8600700 -1 osd.3 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:32.117 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:31 smithi022 podman[104277]: 2024-03-24 06:52:31.786852414 +0000 UTC m=+6.006893587 container died afd041a2570053df74a8354dba34dc7369daea32d2db418485a4b23108b026ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T06:52:32.374 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:32 smithi022 podman[104277]: 2024-03-24 06:52:32.342094461 +0000 UTC m=+6.562135600 container cleanup afd041a2570053df74a8354dba34dc7369daea32d2db418485a4b23108b026ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T06:52:32.374 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:32 smithi022 bash[104277]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3 2024-03-24T06:52:32.899 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:32 smithi022 podman[104309]: 2024-03-24 06:52:32.635417698 +0000 UTC m=+0.831975745 container remove afd041a2570053df74a8354dba34dc7369daea32d2db418485a4b23108b026ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:52:33.899 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:33 smithi022 podman[104395]: 2024-03-24 06:52:33.430314282 +0000 UTC m=+0.032690564 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:33.899 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:33 smithi022 podman[104395]: 2024-03-24 06:52:33.544710669 +0000 UTC m=+0.147087008 container create 11e2d2bcfd28fc6e6b55be33c5f27797cc7f250399436b89124a98f065c43d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T06:52:34.400 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:33 smithi022 podman[104395]: 2024-03-24 06:52:33.959397148 +0000 UTC m=+0.561773500 container init 11e2d2bcfd28fc6e6b55be33c5f27797cc7f250399436b89124a98f065c43d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T06:52:34.400 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:33 smithi022 podman[104395]: 2024-03-24 06:52:33.972431414 +0000 UTC m=+0.574807706 container start 11e2d2bcfd28fc6e6b55be33c5f27797cc7f250399436b89124a98f065c43d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:34.400 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:34 smithi022 podman[104395]: 2024-03-24 06:52:34.064244636 +0000 UTC m=+0.666621025 container attach 11e2d2bcfd28fc6e6b55be33c5f27797cc7f250399436b89124a98f065c43d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T06:52:34.746 INFO:journalctl@ceph.osd.3.smithi022.stdout:Mar 24 06:52:34 smithi022 podman[104395]: 2024-03-24 06:52:34.424282429 +0000 UTC m=+1.026658788 container died 11e2d2bcfd28fc6e6b55be33c5f27797cc7f250399436b89124a98f065c43d7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T06:52:35.199 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T06:52:35.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:35.329 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T06:52:35.329 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T06:52:35.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.4 2024-03-24T06:52:35.678 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:35 smithi143 systemd[1]: Stopping Ceph osd.4 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:36.179 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4[47722]: 2024-03-24T06:52:35.701+0000 7f367d1a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:36.179 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4[47722]: 2024-03-24T06:52:35.701+0000 7f367d1a6700 -1 osd.4 108 *** Got signal Terminated *** 2024-03-24T06:52:36.179 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:35 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4[47722]: 2024-03-24T06:52:35.701+0000 7f367d1a6700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:41.679 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:41 smithi143 podman[72945]: 2024-03-24 06:52:41.398472476 +0000 UTC m=+5.873780545 container died 9421ea1fd3175cf26419b44050981e47744280120a81a0acd24c066e3a30b811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:52:41.961 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:41 smithi143 podman[72945]: 2024-03-24 06:52:41.777321362 +0000 UTC m=+6.252629409 container cleanup 9421ea1fd3175cf26419b44050981e47744280120a81a0acd24c066e3a30b811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:52:41.961 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:41 smithi143 bash[72945]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4 2024-03-24T06:52:42.389 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:41 smithi143 podman[72962]: 2024-03-24 06:52:41.962004439 +0000 UTC m=+0.561941607 container remove 9421ea1fd3175cf26419b44050981e47744280120a81a0acd24c066e3a30b811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:42.641 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.292611577 +0000 UTC m=+0.018133235 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:42.641 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.41289767 +0000 UTC m=+0.138419322 container create 71c65db7f55d8918d8dd160d2b2c25d0a44485d5e2480eff56a28a9174dabe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T06:52:42.641 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.641494866 +0000 UTC m=+0.367016520 container init 71c65db7f55d8918d8dd160d2b2c25d0a44485d5e2480eff56a28a9174dabe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T06:52:42.928 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.64454933 +0000 UTC m=+0.370070986 container start 71c65db7f55d8918d8dd160d2b2c25d0a44485d5e2480eff56a28a9174dabe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T06:52:42.928 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.741419353 +0000 UTC m=+0.466941006 container attach 71c65db7f55d8918d8dd160d2b2c25d0a44485d5e2480eff56a28a9174dabe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T06:52:43.220 INFO:journalctl@ceph.osd.4.smithi143.stdout:Mar 24 06:52:42 smithi143 podman[73052]: 2024-03-24 06:52:42.930654566 +0000 UTC m=+0.656176220 container died 71c65db7f55d8918d8dd160d2b2c25d0a44485d5e2480eff56a28a9174dabe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T06:52:43.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T06:52:43.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:43.565 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T06:52:43.565 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T06:52:43.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.5 2024-03-24T06:52:43.883 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:43 smithi155 systemd[1]: Stopping Ceph osd.5 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:44.278 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:43 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5[48599]: 2024-03-24T06:52:43.884+0000 7f207d187700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:44.278 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:43 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5[48599]: 2024-03-24T06:52:43.884+0000 7f207d187700 -1 osd.5 108 *** Got signal Terminated *** 2024-03-24T06:52:44.278 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:43 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5[48599]: 2024-03-24T06:52:43.884+0000 7f207d187700 -1 osd.5 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:49.893 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:49 smithi155 podman[73952]: 2024-03-24 06:52:49.544640853 +0000 UTC m=+5.785825951 container died 608c6ecff7285fe1044c16d7e14b4557ccf49a79a81c98e847607a9711c6df8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T06:52:50.143 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[73952]: 2024-03-24 06:52:50.000797682 +0000 UTC m=+6.241982788 container cleanup 608c6ecff7285fe1044c16d7e14b4557ccf49a79a81c98e847607a9711c6df8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T06:52:50.143 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 bash[73952]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5 2024-03-24T06:52:50.527 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[73969]: 2024-03-24 06:52:50.302139422 +0000 UTC m=+0.761332759 container remove 608c6ecff7285fe1044c16d7e14b4557ccf49a79a81c98e847607a9711c6df8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:52:50.945 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[74071]: 2024-03-24 06:52:50.570250695 +0000 UTC m=+0.017997736 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:52:50.946 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[74071]: 2024-03-24 06:52:50.695120928 +0000 UTC m=+0.142867956 container create cb74cc70a0bf12183533acf62d4c5a788a837a298e88c43230f5bbf06ba04cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T06:52:51.276 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[74071]: 2024-03-24 06:52:50.982214393 +0000 UTC m=+0.429961445 container init cb74cc70a0bf12183533acf62d4c5a788a837a298e88c43230f5bbf06ba04cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T06:52:51.277 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:50 smithi155 podman[74071]: 2024-03-24 06:52:50.985581248 +0000 UTC m=+0.433328276 container start cb74cc70a0bf12183533acf62d4c5a788a837a298e88c43230f5bbf06ba04cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T06:52:51.277 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:51 smithi155 podman[74071]: 2024-03-24 06:52:51.081089448 +0000 UTC m=+0.528836497 container attach cb74cc70a0bf12183533acf62d4c5a788a837a298e88c43230f5bbf06ba04cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:52:51.527 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 06:52:51 smithi155 podman[74071]: 2024-03-24 06:52:51.277182258 +0000 UTC m=+0.724929294 container died cb74cc70a0bf12183533acf62d4c5a788a837a298e88c43230f5bbf06ba04cda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:52:51.917 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T06:52:51.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:52:51.961 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T06:52:51.961 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T06:52:51.961 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.6 2024-03-24T06:52:52.399 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:52 smithi022 systemd[1]: Stopping Ceph osd.6 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:52:52.900 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6[60171]: 2024-03-24T06:52:52.527+0000 7f0f3f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:52:52.901 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6[60171]: 2024-03-24T06:52:52.527+0000 7f0f3f200700 -1 osd.6 108 *** Got signal Terminated *** 2024-03-24T06:52:52.901 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:52 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6[60171]: 2024-03-24T06:52:52.527+0000 7f0f3f200700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:52:58.613 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:58 smithi022 podman[104500]: 2024-03-24 06:52:58.269343726 +0000 UTC m=+5.853389275 container died b6779de5a45eb94c1d5bc83a79e399af8704cae9c71438603a8eb75aa55ca5d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T06:52:58.899 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:58 smithi022 podman[104500]: 2024-03-24 06:52:58.719905877 +0000 UTC m=+6.303951429 container cleanup b6779de5a45eb94c1d5bc83a79e399af8704cae9c71438603a8eb75aa55ca5d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:52:58.899 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:58 smithi022 bash[104500]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6 2024-03-24T06:52:59.399 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:58 smithi022 podman[104518]: 2024-03-24 06:52:58.929681421 +0000 UTC m=+0.633858483 container remove b6779de5a45eb94c1d5bc83a79e399af8704cae9c71438603a8eb75aa55ca5d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T06:53:00.118 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:59 smithi022 podman[104604]: 2024-03-24 06:52:59.722516257 +0000 UTC m=+0.042917845 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:00.118 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:52:59 smithi022 podman[104604]: 2024-03-24 06:52:59.855582786 +0000 UTC m=+0.175984395 container create 6e97bc91c93f4c56fb4b32b0311368b0c000a0dc4fa4c96a78dca9c37378bcf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:00.118 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:53:00 smithi022 podman[104604]: 2024-03-24 06:53:00.117504489 +0000 UTC m=+0.437906168 container init 6e97bc91c93f4c56fb4b32b0311368b0c000a0dc4fa4c96a78dca9c37378bcf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T06:53:00.399 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:53:00 smithi022 podman[104604]: 2024-03-24 06:53:00.129538954 +0000 UTC m=+0.449940586 container start 6e97bc91c93f4c56fb4b32b0311368b0c000a0dc4fa4c96a78dca9c37378bcf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T06:53:00.399 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:53:00 smithi022 podman[104604]: 2024-03-24 06:53:00.240916866 +0000 UTC m=+0.561318518 container attach 6e97bc91c93f4c56fb4b32b0311368b0c000a0dc4fa4c96a78dca9c37378bcf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T06:53:00.899 INFO:journalctl@ceph.osd.6.smithi022.stdout:Mar 24 06:53:00 smithi022 podman[104604]: 2024-03-24 06:53:00.577425444 +0000 UTC m=+0.897827126 container died 6e97bc91c93f4c56fb4b32b0311368b0c000a0dc4fa4c96a78dca9c37378bcf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:01.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T06:53:01.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:01.681 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T06:53:01.681 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T06:53:01.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.7 2024-03-24T06:53:01.968 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:01 smithi143 systemd[1]: Stopping Ceph osd.7 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:02.374 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:02 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7[54601]: 2024-03-24T06:53:02.007+0000 7f2d8ce63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:02.374 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:02 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7[54601]: 2024-03-24T06:53:02.007+0000 7f2d8ce63700 -1 osd.7 108 *** Got signal Terminated *** 2024-03-24T06:53:02.374 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:02 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7[54601]: 2024-03-24T06:53:02.007+0000 7f2d8ce63700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:53:06.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:06 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:06.038+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:06.428 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:06 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:06.126+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:07.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:07 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:07.060+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:07.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:07 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:07.139+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:07.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:07 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:07.251+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:07.924 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:07 smithi143 podman[73159]: 2024-03-24 06:53:07.607301497 +0000 UTC m=+5.730590734 container died 22f74fb3e4585ba302aa1c62321936f9ae8efedf81b171047d59b3d0606cd7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T06:53:08.179 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:08 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:08.155+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:08.179 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73159]: 2024-03-24 06:53:08.023058233 +0000 UTC m=+6.146347467 container cleanup 22f74fb3e4585ba302aa1c62321936f9ae8efedf81b171047d59b3d0606cd7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:53:08.179 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 bash[73159]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7 2024-03-24T06:53:08.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:08 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:08.047+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:08.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:08 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:08.260+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:08.628 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73188]: 2024-03-24 06:53:08.219378822 +0000 UTC m=+0.613367506 container remove 22f74fb3e4585ba302aa1c62321936f9ae8efedf81b171047d59b3d0606cd7c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T06:53:08.902 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73279]: 2024-03-24 06:53:08.532489814 +0000 UTC m=+0.017657942 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:08.902 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73279]: 2024-03-24 06:53:08.666978091 +0000 UTC m=+0.152146197 container create d84bbc4fa664c2fa2b13c56c342c76c397857aa1123924066bd3d8bf37eb9908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:53:09.178 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73279]: 2024-03-24 06:53:08.902536175 +0000 UTC m=+0.387704299 container init d84bbc4fa664c2fa2b13c56c342c76c397857aa1123924066bd3d8bf37eb9908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-24T06:53:09.178 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:08 smithi143 podman[73279]: 2024-03-24 06:53:08.906280052 +0000 UTC m=+0.391448160 container start d84bbc4fa664c2fa2b13c56c342c76c397857aa1123924066bd3d8bf37eb9908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T06:53:09.178 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:09 smithi143 podman[73279]: 2024-03-24 06:53:09.019799063 +0000 UTC m=+0.504967194 container attach d84bbc4fa664c2fa2b13c56c342c76c397857aa1123924066bd3d8bf37eb9908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_CLEAN=True) 2024-03-24T06:53:09.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:09 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:09.004+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:09.428 INFO:journalctl@ceph.osd.7.smithi143.stdout:Mar 24 06:53:09 smithi143 podman[73279]: 2024-03-24 06:53:09.192358468 +0000 UTC m=+0.677526582 container died d84bbc4fa664c2fa2b13c56c342c76c397857aa1123924066bd3d8bf37eb9908 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-24T06:53:09.428 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:09 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:09.176+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:09.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:09 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:09.216+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:09.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T06:53:09.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:09.884 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T06:53:09.884 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T06:53:09.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.8 2024-03-24T06:53:10.167 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:09 smithi155 systemd[1]: Stopping Ceph osd.8 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:10.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:09 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:09.972+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:10.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:10 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:10.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:10.528 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:10 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8[55486]: 2024-03-24T06:53:10.202+0000 7f3acdc49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:10.528 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:10 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8[55486]: 2024-03-24T06:53:10.202+0000 7f3acdc49700 -1 osd.8 108 *** Got signal Terminated *** 2024-03-24T06:53:10.528 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:10 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8[55486]: 2024-03-24T06:53:10.202+0000 7f3acdc49700 -1 osd.8 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:53:10.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:10 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:10.206+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:11.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:10 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:10.953+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:11.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:11 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:11.130+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:11.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:11 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:11.234+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:12.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:11 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:11.988+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:12.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:12 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:12.125+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:12.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:12 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:12.256+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:13.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:13 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:13.006+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:13.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:13 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:13.156+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:13.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:13 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:13.286+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:14.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:14.003+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:14.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:14 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:14.110+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:14.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:14 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:14.294+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:15.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:14 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:14.962+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:15.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:15 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:15.107+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:15.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:15 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:15.107+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:15.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:15 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:15.286+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:16.020 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:15 smithi155 podman[74176]: 2024-03-24 06:53:15.725984485 +0000 UTC m=+5.647497138 container died 10b2ed30cd8207163a8d62c903ef3fe34e4a49b622c8f731aaac6962bf66e949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:53:16.278 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74176]: 2024-03-24 06:53:16.11144049 +0000 UTC m=+6.032953126 container cleanup 10b2ed30cd8207163a8d62c903ef3fe34e4a49b622c8f731aaac6962bf66e949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T06:53:16.278 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 bash[74176]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8 2024-03-24T06:53:16.278 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:16 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:16.154+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:16.279 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:16 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:16.154+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:16.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:15 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:15.948+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:16.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:16 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:16.270+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:16.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:16 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:16.270+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:16.712 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74192]: 2024-03-24 06:53:16.304102508 +0000 UTC m=+0.575479171 container remove 10b2ed30cd8207163a8d62c903ef3fe34e4a49b622c8f731aaac6962bf66e949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T06:53:17.028 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74286]: 2024-03-24 06:53:16.713186589 +0000 UTC m=+0.103564241 container create 54e5056feab6a06e5cdc1ddd72adea152c4d251470aac3fa09a8952326c40396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:53:17.028 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74286]: 2024-03-24 06:53:16.627155373 +0000 UTC m=+0.017533039 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:17.028 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74286]: 2024-03-24 06:53:16.949353192 +0000 UTC m=+0.339730870 container init 54e5056feab6a06e5cdc1ddd72adea152c4d251470aac3fa09a8952326c40396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:17.028 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:16 smithi155 podman[74286]: 2024-03-24 06:53:16.952827037 +0000 UTC m=+0.343204705 container start 54e5056feab6a06e5cdc1ddd72adea152c4d251470aac3fa09a8952326c40396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:17.318 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:17 smithi155 podman[74286]: 2024-03-24 06:53:17.067281119 +0000 UTC m=+0.457658800 container attach 54e5056feab6a06e5cdc1ddd72adea152c4d251470aac3fa09a8952326c40396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0) 2024-03-24T06:53:17.319 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 24 06:53:17 smithi155 podman[74286]: 2024-03-24 06:53:17.255807247 +0000 UTC m=+0.646184909 container died 54e5056feab6a06e5cdc1ddd72adea152c4d251470aac3fa09a8952326c40396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T06:53:17.319 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:17.180+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:17.319 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:17.180+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:17.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:16 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:16.963+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:17.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:17 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:17.276+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:17.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:17 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:17.276+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:17.894 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T06:53:17.935 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:17.935 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T06:53:17.936 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T06:53:17.936 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.9 2024-03-24T06:53:18.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:17 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:17.932+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:18.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 systemd[1]: Stopping Ceph osd.9 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:18.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:18.194+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:18.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:18 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:18.194+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:18.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:18 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:18.290+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:18.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:18 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:18.290+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:18.900 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:18.544+0000 7f4fa2000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:18.900 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:18.544+0000 7f4fa2000700 -1 osd.9 108 *** Got signal Terminated *** 2024-03-24T06:53:18.900 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:18.544+0000 7f4fa2000700 -1 osd.9 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:53:19.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:18.936+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:19.400 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:18 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:18.936+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.607330+0000 front 2024-03-24T06:52:13.607383+0000 (oldest deadline 2024-03-24T06:53:18.907376+0000) 2024-03-24T06:53:19.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:19 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:19.200+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:19.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:19 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:19.200+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:19.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:19 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:19.307+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:19.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:19 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:19.307+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:20.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:19 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:19.918+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:20.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:19 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:19.918+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.607330+0000 front 2024-03-24T06:52:13.607383+0000 (oldest deadline 2024-03-24T06:53:18.907376+0000) 2024-03-24T06:53:20.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:20.199+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:20.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:20 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:20.199+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:20.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:20.304+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:20.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:20 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:20.304+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:21.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:20 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:20.921+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:21.399 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:20 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:20.921+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.607330+0000 front 2024-03-24T06:52:13.607383+0000 (oldest deadline 2024-03-24T06:53:18.907376+0000) 2024-03-24T06:53:21.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:21.200+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:21.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:21.200+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:21.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:21.327+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:21.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:21 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:21.327+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:22.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:21 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:21.885+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:22.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:21 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:21.885+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.607330+0000 front 2024-03-24T06:52:13.607383+0000 (oldest deadline 2024-03-24T06:53:18.907376+0000) 2024-03-24T06:53:22.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:22 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:22.204+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:22.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:22 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:22.204+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:22.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:22 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:22.320+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:22.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:22 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:22.320+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:23.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:22 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:22.848+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.299961+0000 front 2024-03-24T06:52:03.299511+0000 (oldest deadline 2024-03-24T06:53:05.599688+0000) 2024-03-24T06:53:23.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:22 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9[67037]: 2024-03-24T06:53:22.848+0000 7f4f98800700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.607330+0000 front 2024-03-24T06:52:13.607383+0000 (oldest deadline 2024-03-24T06:53:18.907376+0000) 2024-03-24T06:53:23.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:23 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:23.173+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:23.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:23 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:23.173+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:23.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:23.368+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:23.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:23 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:23.369+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:24.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:24 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:24.205+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:24.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:24 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:24.205+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:24.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:24 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:24.366+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:24.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:24 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:24.366+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:24.826 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:24 smithi022 podman[104708]: 2024-03-24 06:53:24.522784997 +0000 UTC m=+6.108657656 container died ac987208f9b0e54733f794124207a49c9866b6273ed38fc7f39ed13befe60610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T06:53:25.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:24 smithi022 podman[104708]: 2024-03-24 06:53:24.945724325 +0000 UTC m=+6.531597023 container cleanup ac987208f9b0e54733f794124207a49c9866b6273ed38fc7f39ed13befe60610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T06:53:25.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:24 smithi022 bash[104708]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9 2024-03-24T06:53:25.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:25 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:25.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:25.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:25 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:25.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:25.649 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:25 smithi022 podman[104727]: 2024-03-24 06:53:25.296832768 +0000 UTC m=+0.737401743 container remove ac987208f9b0e54733f794124207a49c9866b6273ed38fc7f39ed13befe60610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T06:53:25.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:25 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:25.341+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:25.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:25 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:25.341+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:26.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:26 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:26.201+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:26.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:26 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:26.201+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:26.538 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:26 smithi022 podman[104816]: 2024-03-24 06:53:26.072621297 +0000 UTC m=+0.037080212 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:26.538 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:26 smithi022 podman[104816]: 2024-03-24 06:53:26.21446063 +0000 UTC m=+0.178919551 container create 5e476dae2a44c39e4bcd63c00c0eb991743a26bb056bdc2fa5a6352b93d824f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS) 2024-03-24T06:53:26.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:26 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:26.342+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:26.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:26 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:26.342+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:26.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:26 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:26.342+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:26.866 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:26 smithi022 podman[104816]: 2024-03-24 06:53:26.60970572 +0000 UTC m=+0.574164645 container init 5e476dae2a44c39e4bcd63c00c0eb991743a26bb056bdc2fa5a6352b93d824f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T06:53:26.866 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:26 smithi022 podman[104816]: 2024-03-24 06:53:26.620533057 +0000 UTC m=+0.584991981 container start 5e476dae2a44c39e4bcd63c00c0eb991743a26bb056bdc2fa5a6352b93d824f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T06:53:26.866 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:26 smithi022 podman[104816]: 2024-03-24 06:53:26.747086339 +0000 UTC m=+0.711545287 container attach 5e476dae2a44c39e4bcd63c00c0eb991743a26bb056bdc2fa5a6352b93d824f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T06:53:27.149 INFO:journalctl@ceph.osd.9.smithi022.stdout:Mar 24 06:53:27 smithi022 podman[104816]: 2024-03-24 06:53:27.079002166 +0000 UTC m=+1.043461065 container died 5e476dae2a44c39e4bcd63c00c0eb991743a26bb056bdc2fa5a6352b93d824f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-24T06:53:27.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:27.224+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:27.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:27.224+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:27.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:27 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:27.224+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:27.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:27.343+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:27.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:27.343+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:27.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:27 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:27.343+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:27.753 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T06:53:27.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:27.861 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T06:53:27.861 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T06:53:27.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.10 2024-03-24T06:53:28.176 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:27 smithi143 systemd[1]: Stopping Ceph osd.10 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:28.176 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.175+0000 7ff268c40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:28.176 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.175+0000 7ff268c40700 -1 osd.10 108 *** Got signal Terminated *** 2024-03-24T06:53:28.176 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.175+0000 7ff268c40700 -1 osd.10 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:53:28.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.294+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:28.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.294+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:28.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:28 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:28.294+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:28.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:28.218+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:28.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:28.218+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:28.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:28.218+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:29.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:29 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:29.173+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:29.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:29 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:29.173+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:29.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:29 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:29.173+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:29.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:29 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:29.277+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:29.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:29 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:29.277+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:29.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:29 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:29.277+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:30.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:30 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:30.175+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:30.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:30 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:30.175+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:30.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:30 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:30.175+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:30.678 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:30 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:30.229+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:30.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:30 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:30.229+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:30.679 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:30 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:30.229+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:31.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:31 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:31.186+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:31.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:31 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:31.186+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:31.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:31 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:31.186+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:31.580 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:31 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:31.209+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:31.580 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:31 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:31.209+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:31.580 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:31 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:31.209+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:32.529 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:32 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:32.203+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:32.529 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:32 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:32.203+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:32.529 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:32 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:32.203+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:32.593 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:32 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:32.248+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:00.560114+0000 front 2024-03-24T06:52:00.560032+0000 (oldest deadline 2024-03-24T06:53:05.859898+0000) 2024-03-24T06:53:32.593 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:32 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:32.248+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:13.362362+0000 front 2024-03-24T06:52:13.362389+0000 (oldest deadline 2024-03-24T06:53:16.262519+0000) 2024-03-24T06:53:32.594 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:32 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10[61226]: 2024-03-24T06:53:32.248+0000 7ff265234700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:21.365056+0000 front 2024-03-24T06:52:21.364949+0000 (oldest deadline 2024-03-24T06:53:25.464847+0000) 2024-03-24T06:53:33.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:33 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:33.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:33.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:33 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:33.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:33.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:33 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:33.167+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:33.967 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:33 smithi143 podman[73385]: 2024-03-24 06:53:33.680095428 +0000 UTC m=+5.625314854 container died f042533963afe3e88566a7483d5bdeb57054e6726f659ce5f310b9ddb7cbda88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T06:53:34.310 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73385]: 2024-03-24 06:53:34.067336966 +0000 UTC m=+6.012556397 container cleanup f042533963afe3e88566a7483d5bdeb57054e6726f659ce5f310b9ddb7cbda88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T06:53:34.310 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 bash[73385]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10 2024-03-24T06:53:34.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:34 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:34.205+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:34.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:34 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:34.205+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:34.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:34 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:34.205+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:34.661 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73411]: 2024-03-24 06:53:34.310976004 +0000 UTC m=+0.627495842 container remove f042533963afe3e88566a7483d5bdeb57054e6726f659ce5f310b9ddb7cbda88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:53:34.928 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73501]: 2024-03-24 06:53:34.661629182 +0000 UTC m=+0.111034132 container create 6aea700babc73fdd79448f57daefa284e58554875f142359f9da0df9475f2439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T06:53:34.929 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73501]: 2024-03-24 06:53:34.568289361 +0000 UTC m=+0.017694321 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:34.929 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73501]: 2024-03-24 06:53:34.855425553 +0000 UTC m=+0.304830502 container init 6aea700babc73fdd79448f57daefa284e58554875f142359f9da0df9475f2439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T06:53:34.929 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73501]: 2024-03-24 06:53:34.858696015 +0000 UTC m=+0.308100960 container start 6aea700babc73fdd79448f57daefa284e58554875f142359f9da0df9475f2439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True) 2024-03-24T06:53:35.428 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:34 smithi143 podman[73501]: 2024-03-24 06:53:34.93017505 +0000 UTC m=+0.379580000 container attach 6aea700babc73fdd79448f57daefa284e58554875f142359f9da0df9475f2439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T06:53:35.429 INFO:journalctl@ceph.osd.10.smithi143.stdout:Mar 24 06:53:35 smithi143 podman[73501]: 2024-03-24 06:53:35.15118469 +0000 UTC m=+0.600589641 container died 6aea700babc73fdd79448f57daefa284e58554875f142359f9da0df9475f2439 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T06:53:35.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:35 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:35.183+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:35.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:35 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:35.183+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:35.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:35 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:35.183+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:35.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T06:53:35.759 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:35.760 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T06:53:35.760 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T06:53:35.760 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.11 2024-03-24T06:53:36.071 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:35 smithi155 systemd[1]: Stopping Ceph osd.11 for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:36.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.071+0000 7f575d2cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:36.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.071+0000 7f575d2cb700 -1 osd.11 108 *** Got signal Terminated *** 2024-03-24T06:53:36.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.071+0000 7f575d2cb700 -1 osd.11 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T06:53:36.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.219+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:36.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.219+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:36.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:36 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:36.219+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:37.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:37 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:37.197+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:37.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:37 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:37.197+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:37.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:37 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:37.197+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:38.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:38 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:38.177+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:38.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:38 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:38.177+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:38.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:38 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:38.177+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:39.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:39 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:39.147+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:39.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:39 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:39.147+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:39.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:39 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:39.147+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:40.527 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:40 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:40.102+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.22:6806 osd.0 since back 2024-03-24T06:52:03.313838+0000 front 2024-03-24T06:52:03.313739+0000 (oldest deadline 2024-03-24T06:53:06.813668+0000) 2024-03-24T06:53:40.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:40 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:40.102+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.1 since back 2024-03-24T06:52:12.714806+0000 front 2024-03-24T06:52:12.714747+0000 (oldest deadline 2024-03-24T06:53:15.014921+0000) 2024-03-24T06:53:40.528 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:40 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11[62576]: 2024-03-24T06:53:40.102+0000 7f57598bf700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-24T06:52:17.315808+0000 front 2024-03-24T06:52:17.315743+0000 (oldest deadline 2024-03-24T06:53:26.716460+0000) 2024-03-24T06:53:41.861 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:41 smithi155 podman[74405]: 2024-03-24 06:53:41.562619997 +0000 UTC m=+5.612343719 container died 143f11ab3ae370f521b00546fc8143acb85337a6e6e18ffc3bb1cf208a8fdf0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T06:53:41.862 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:41 smithi155 podman[74405]: 2024-03-24 06:53:41.862316646 +0000 UTC m=+5.912040359 container cleanup 143f11ab3ae370f521b00546fc8143acb85337a6e6e18ffc3bb1cf208a8fdf0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:53:42.277 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:41 smithi155 bash[74405]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11 2024-03-24T06:53:42.278 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74422]: 2024-03-24 06:53:42.097188791 +0000 UTC m=+0.534007564 container remove 143f11ab3ae370f521b00546fc8143acb85337a6e6e18ffc3bb1cf208a8fdf0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T06:53:42.724 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.316666561 +0000 UTC m=+0.017852084 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T06:53:42.724 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.414575982 +0000 UTC m=+0.115761490 container create 20d33aa492aaaa175d675f71af50b09792381e30f436daab488c60d31e8868fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-24T06:53:42.724 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.642201372 +0000 UTC m=+0.343386881 container init 20d33aa492aaaa175d675f71af50b09792381e30f436daab488c60d31e8868fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:42.725 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.64546508 +0000 UTC m=+0.346650593 container start 20d33aa492aaaa175d675f71af50b09792381e30f436daab488c60d31e8868fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:53:42.725 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.725059575 +0000 UTC m=+0.426245094 container attach 20d33aa492aaaa175d675f71af50b09792381e30f436daab488c60d31e8868fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:43.027 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 24 06:53:42 smithi155 podman[74511]: 2024-03-24 06:53:42.933491816 +0000 UTC m=+0.634677332 container died 20d33aa492aaaa175d675f71af50b09792381e30f436daab488c60d31e8868fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T06:53:43.556 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T06:53:43.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:43.596 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T06:53:43.596 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T06:53:43.596 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.a 2024-03-24T06:53:44.149 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:43 smithi022 systemd[1]: Stopping Ceph mds.a for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:44.649 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:44 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: 2024-03-24T06:53:44.186+0000 7f8a47400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:44.649 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:44 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a[73566]: 2024-03-24T06:53:44.186+0000 7f8a47400700 -1 mds.a *** got signal Terminated *** 2024-03-24T06:53:48.275 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:47 smithi022 podman[104931]: 2024-03-24 06:53:47.954711635 +0000 UTC m=+3.896315743 container died eddae7b6d838d7eed5a39f9382836d4f32b5b6cebf352dbfd4f994a3699377f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T06:53:48.649 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:48 smithi022 podman[104931]: 2024-03-24 06:53:48.387281794 +0000 UTC m=+4.328885896 container cleanup eddae7b6d838d7eed5a39f9382836d4f32b5b6cebf352dbfd4f994a3699377f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:48.650 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:48 smithi022 bash[104931]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a 2024-03-24T06:53:49.149 INFO:journalctl@ceph.mds.a.smithi022.stdout:Mar 24 06:53:48 smithi022 podman[104945]: 2024-03-24 06:53:48.679839481 +0000 UTC m=+0.694236944 container remove eddae7b6d838d7eed5a39f9382836d4f32b5b6cebf352dbfd4f994a3699377f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T06:53:49.193 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T06:53:49.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:49.322 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T06:53:49.322 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T06:53:49.322 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.d 2024-03-24T06:53:49.899 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:49 smithi022 systemd[1]: Stopping Ceph mds.d for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:50.399 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:49 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d[72539]: 2024-03-24T06:53:49.955+0000 7fa760400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:50.400 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:49 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d[72539]: 2024-03-24T06:53:49.955+0000 7fa760400700 -1 mds.d *** got signal Terminated *** 2024-03-24T06:53:52.349 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:52 smithi022 podman[105052]: 2024-03-24 06:53:52.027449479 +0000 UTC m=+2.193376876 container died 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T06:53:52.649 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:52 smithi022 podman[105052]: 2024-03-24 06:53:52.456976741 +0000 UTC m=+2.622904163 container cleanup 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T06:53:52.649 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:52 smithi022 bash[105052]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d 2024-03-24T06:53:53.149 INFO:journalctl@ceph.mds.d.smithi022.stdout:Mar 24 06:53:52 smithi022 podman[105066]: 2024-03-24 06:53:52.758235095 +0000 UTC m=+0.713376518 container remove 506cb8f9e4c6e0b7fbcd189d4c8658c54ef0f40c4a4620ec111aec7eb9c1555f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:53:53.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T06:53:53.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:53.396 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T06:53:53.396 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T06:53:53.396 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.g 2024-03-24T06:53:53.899 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:53 smithi022 systemd[1]: Stopping Ceph mds.g for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:54.399 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g[73185]: 2024-03-24T06:53:54.008+0000 7f269be00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:54.400 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:54 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g[73185]: 2024-03-24T06:53:54.008+0000 7f269be00700 -1 mds.g *** got signal Terminated *** 2024-03-24T06:53:55.382 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:55 smithi022 podman[105171]: 2024-03-24 06:53:55.124285807 +0000 UTC m=+1.244271999 container died e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T06:53:55.649 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:55 smithi022 podman[105171]: 2024-03-24 06:53:55.49396873 +0000 UTC m=+1.613954962 container cleanup e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:53:55.650 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:55 smithi022 bash[105171]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g 2024-03-24T06:53:56.149 INFO:journalctl@ceph.mds.g.smithi022.stdout:Mar 24 06:53:55 smithi022 podman[105185]: 2024-03-24 06:53:55.745087039 +0000 UTC m=+0.610130887 container remove e181e59c64ef4d0e08d95b9a059231f092c8ea4f767b1ad6ec43e046866ccbe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T06:53:56.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T06:53:56.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:53:56.422 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T06:53:56.422 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T06:53:56.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.j 2024-03-24T06:53:56.899 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:53:56 smithi022 systemd[1]: Stopping Ceph mds.j for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:53:57.399 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:53:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: 2024-03-24T06:53:57.062+0000 7f7279c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:53:57.399 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:53:57 smithi022 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j[72197]: 2024-03-24T06:53:57.062+0000 7f7279c00700 -1 mds.j *** got signal Terminated *** 2024-03-24T06:54:00.344 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:53:59 smithi022 podman[105290]: 2024-03-24 06:53:59.969622037 +0000 UTC m=+3.022663601 container died 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T06:54:00.648 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:54:00 smithi022 podman[105290]: 2024-03-24 06:54:00.438781532 +0000 UTC m=+3.491823090 container cleanup 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T06:54:00.648 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:54:00 smithi022 bash[105290]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j 2024-03-24T06:54:00.899 INFO:journalctl@ceph.mds.j.smithi022.stdout:Mar 24 06:54:00 smithi022 podman[105316]: 2024-03-24 06:54:00.64791328 +0000 UTC m=+0.660434191 container remove 492442409fbf0c574556014cb8677b1072ebc04ded0c0dd81026aea7a4f22bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T06:54:01.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T06:54:01.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:01.318 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T06:54:01.319 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T06:54:01.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.b 2024-03-24T06:54:01.608 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:01 smithi143 systemd[1]: Stopping Ceph mds.b for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:01.860 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:01 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: 2024-03-24T06:54:01.608+0000 7f2c8ae30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:01.860 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:01 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b[66613]: 2024-03-24T06:54:01.608+0000 7f2c8ae30700 -1 mds.b *** got signal Terminated *** 2024-03-24T06:54:04.117 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:03 smithi143 podman[73607]: 2024-03-24 06:54:03.854280305 +0000 UTC m=+2.338758580 container died 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T06:54:04.428 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:04 smithi143 podman[73607]: 2024-03-24 06:54:04.191452838 +0000 UTC m=+2.675931110 container cleanup 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T06:54:04.429 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:04 smithi143 bash[73607]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b 2024-03-24T06:54:04.429 INFO:journalctl@ceph.mds.b.smithi143.stdout:Mar 24 06:54:04 smithi143 podman[73633]: 2024-03-24 06:54:04.367443798 +0000 UTC m=+0.507301036 container remove 4434474f729dba6369db4d804e053d6648c188114b94ea34cce9a13c737ebeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-24T06:54:04.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T06:54:04.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:04.652 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T06:54:04.653 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T06:54:04.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.e 2024-03-24T06:54:04.928 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:04 smithi143 systemd[1]: Stopping Ceph mds.e for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:05.428 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:05 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e[65990]: 2024-03-24T06:54:05.052+0000 7feaef7e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:05.429 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:05 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e[65990]: 2024-03-24T06:54:05.052+0000 7feaef7e9700 -1 mds.e *** got signal Terminated *** 2024-03-24T06:54:08.112 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:07 smithi143 podman[73740]: 2024-03-24 06:54:07.860030788 +0000 UTC m=+2.992559446 container died 0f41d1e7b8eddea19bc49b512043cf54802572213bd728ce9c445d521920a3cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T06:54:08.429 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:08 smithi143 podman[73740]: 2024-03-24 06:54:08.211522551 +0000 UTC m=+3.344051201 container cleanup 0f41d1e7b8eddea19bc49b512043cf54802572213bd728ce9c445d521920a3cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T06:54:08.429 INFO:journalctl@ceph.mds.e.smithi143.stdout:Mar 24 06:54:08 smithi143 bash[73740]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-e 2024-03-24T06:54:08.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T06:54:08.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:08.667 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T06:54:08.668 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T06:54:08.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.h 2024-03-24T06:54:09.178 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:08 smithi143 systemd[1]: Stopping Ceph mds.h for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:09.179 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:08 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: 2024-03-24T06:54:08.973+0000 7f0d08fd9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:09.179 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:08 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h[66294]: 2024-03-24T06:54:08.973+0000 7f0d08fd9700 -1 mds.h *** got signal Terminated *** 2024-03-24T06:54:12.124 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:11 smithi143 podman[73864]: 2024-03-24 06:54:11.862831651 +0000 UTC m=+2.980435959 container died 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T06:54:12.429 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:12 smithi143 podman[73864]: 2024-03-24 06:54:12.198130898 +0000 UTC m=+3.315735183 container cleanup 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T06:54:12.429 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:12 smithi143 bash[73864]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h 2024-03-24T06:54:12.429 INFO:journalctl@ceph.mds.h.smithi143.stdout:Mar 24 06:54:12 smithi143 podman[73877]: 2024-03-24 06:54:12.365752991 +0000 UTC m=+0.500989535 container remove 086cde1d2cecd98e58e7400a1f76f0cf4d07736d5e076b24325aa67fd8373f1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-h, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T06:54:12.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T06:54:12.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:12.609 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T06:54:12.609 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T06:54:12.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.k 2024-03-24T06:54:12.849 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:12 smithi143 systemd[1]: Stopping Ceph mds.k for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:13.179 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:12 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: 2024-03-24T06:54:12.967+0000 7f1378839700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:13.179 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:12 smithi143 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k[65668]: 2024-03-24T06:54:12.967+0000 7f1378839700 -1 mds.k *** got signal Terminated *** 2024-03-24T06:54:16.179 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:15 smithi143 podman[73986]: 2024-03-24 06:54:15.860319104 +0000 UTC m=+3.039471133 container died d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T06:54:16.485 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:16 smithi143 podman[73986]: 2024-03-24 06:54:16.276092929 +0000 UTC m=+3.455244945 container cleanup d95ef13680ee5a2642585884f95b4cc6091755feae685f177677e5d99a809c55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T06:54:16.485 INFO:journalctl@ceph.mds.k.smithi143.stdout:Mar 24 06:54:16 smithi143 bash[73986]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-k 2024-03-24T06:54:16.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T06:54:16.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:16.738 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T06:54:16.738 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T06:54:16.738 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.c 2024-03-24T06:54:17.027 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:16 smithi155 systemd[1]: Stopping Ceph mds.c for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:17.528 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: 2024-03-24T06:54:17.027+0000 7fd4ed775700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:17.528 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:17 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c[67583]: 2024-03-24T06:54:17.027+0000 7fd4ed775700 -1 mds.c *** got signal Terminated *** 2024-03-24T06:54:20.170 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:19 smithi155 podman[74617]: 2024-03-24 06:54:19.863053396 +0000 UTC m=+2.934342108 container died b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T06:54:20.527 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:20 smithi155 podman[74617]: 2024-03-24 06:54:20.269199989 +0000 UTC m=+3.340488661 container cleanup b98fdcb700ea81f5e43dc34d5af38c014c68bba6344992a38ef6b96c9ac92234 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T06:54:20.528 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 24 06:54:20 smithi155 bash[74617]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-c 2024-03-24T06:54:20.691 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T06:54:20.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:20.729 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T06:54:20.730 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T06:54:20.730 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.f 2024-03-24T06:54:21.056 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:20 smithi155 systemd[1]: Stopping Ceph mds.f for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:21.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: 2024-03-24T06:54:21.055+0000 7fa132b44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:21.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:21 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f[67276]: 2024-03-24T06:54:21.055+0000 7fa132b44700 -1 mds.f *** got signal Terminated *** 2024-03-24T06:54:24.198 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:23 smithi155 podman[74751]: 2024-03-24 06:54:23.918081795 +0000 UTC m=+2.970616772 container died a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T06:54:24.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:24 smithi155 podman[74751]: 2024-03-24 06:54:24.297335307 +0000 UTC m=+3.349870272 container cleanup a19cae688fa8cfa1bf5ac8ea118f90658c53c8b928d91c176eb6ad500d8aea99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True) 2024-03-24T06:54:24.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 24 06:54:24 smithi155 bash[74751]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-f 2024-03-24T06:54:24.715 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T06:54:24.752 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:24.752 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T06:54:24.752 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T06:54:24.752 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.i 2024-03-24T06:54:25.128 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:24 smithi155 systemd[1]: Stopping Ceph mds.i for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:25.528 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:25 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: 2024-03-24T06:54:25.128+0000 7fcdac0e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:25.528 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:25 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i[67894]: 2024-03-24T06:54:25.128+0000 7fcdac0e4700 -1 mds.i *** got signal Terminated *** 2024-03-24T06:54:28.210 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:27 smithi155 podman[74873]: 2024-03-24 06:54:27.924814682 +0000 UTC m=+2.946158229 container died 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T06:54:28.527 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:28 smithi155 podman[74873]: 2024-03-24 06:54:28.210960959 +0000 UTC m=+3.232304499 container cleanup 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T06:54:28.528 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:28 smithi155 bash[74873]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i 2024-03-24T06:54:28.528 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 24 06:54:28 smithi155 podman[74885]: 2024-03-24 06:54:28.3790528 +0000 UTC m=+0.450227305 container remove 4039ab6cfecaa79144a7cf0ea40be406c23d1b0d310ab3d21b8dcfd12070489c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T06:54:28.607 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T06:54:28.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:28.641 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T06:54:28.642 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T06:54:28.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.l 2024-03-24T06:54:28.964 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:28 smithi155 systemd[1]: Stopping Ceph mds.l for 7bca48fe-e9a7-11ee-95cd-87774f69a715... 2024-03-24T06:54:29.277 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: 2024-03-24T06:54:28.964+0000 7f10aeb47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T06:54:29.278 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:28 smithi155 ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l[66965]: 2024-03-24T06:54:28.964+0000 7f10aeb47700 -1 mds.l *** got signal Terminated *** 2024-03-24T06:54:32.322 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:32 smithi155 podman[74994]: 2024-03-24 06:54:32.036957585 +0000 UTC m=+3.176231912 container died 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T06:54:32.323 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:32 smithi155 podman[74994]: 2024-03-24 06:54:32.322851822 +0000 UTC m=+3.462126148 container cleanup 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T06:54:32.731 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:32 smithi155 bash[74994]: ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l 2024-03-24T06:54:32.731 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 24 06:54:32 smithi155 podman[75008]: 2024-03-24 06:54:32.524007282 +0000 UTC m=+0.478754606 container remove 0c59592ec916078e899786fea630060c8b9393575b64f9043dd71c9617d952d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T06:54:32.748 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7bca48fe-e9a7-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T06:54:32.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T06:54:32.782 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T06:54:32.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T06:54:33.107 INFO:teuthology.orchestra.run.smithi022.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:54:38.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T06:54:38.646 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:54:41.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T06:54:41.529 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T06:54:44.275 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:54:44.385 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:54:44.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T06:54:44.448 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T06:54:44.449 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi022/crash 2024-03-24T06:54:44.451 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash -- . 2024-03-24T06:54:44.563 INFO:teuthology.orchestra.run.smithi022.stderr:tar: /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T06:54:44.563 INFO:teuthology.orchestra.run.smithi022.stderr:tar: Error is not recoverable: exiting now 2024-03-24T06:54:44.568 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi143/crash 2024-03-24T06:54:44.569 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash -- . 2024-03-24T06:54:44.616 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T06:54:44.616 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-03-24T06:54:44.618 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi155/crash 2024-03-24T06:54:44.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash -- . 2024-03-24T06:54:44.664 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T06:54:44.664 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-03-24T06:54:44.665 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T06:54:44.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T06:54:44.830 INFO:tasks.cephadm:Compressing logs... 2024-03-24T06:54:44.830 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T06:54:44.834 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T06:54:44.837 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T06:54:44.869 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T06:54:44.871 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T06:54:44.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T06:54:44.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T06:54:44.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T06:54:44.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T06:54:44.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T06:54:44.875 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log 2024-03-24T06:54:44.875 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T06:54:44.875 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T06:54:44.875 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T06:54:44.875 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.2.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T06:54:44.878 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.c.log: 92.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T06:54:44.879 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T06:54:44.880 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T06:54:44.880 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T06:54:44.880 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T06:54:44.880 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T06:54:44.883 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T06:54:44.883 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T06:54:44.883 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T06:54:44.883 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T06:54:44.883 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T06:54:44.884 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T06:54:44.884 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T06:54:44.884 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log 2024-03-24T06:54:44.884 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T06:54:44.884 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T06:54:44.888 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.b.log: 92.0% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T06:54:44.932 INFO:teuthology.orchestra.run.smithi022.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T06:54:44.933 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T06:54:44.940 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T06:54:44.942 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T06:54:44.943 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T06:54:44.945 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T06:54:44.946 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T06:54:44.947 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T06:54:44.948 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T06:54:44.949 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T06:54:44.950 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T06:54:44.951 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log 2024-03-24T06:54:44.952 INFO:teuthology.orchestra.run.smithi022.stderr: 94.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T06:54:44.952 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T06:54:44.953 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T06:54:44.954 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T06:54:44.955 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T06:54:44.956 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T06:54:44.957 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-24T06:54:44.957 INFO:teuthology.orchestra.run.smithi155.stderr: 83.4% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T06:54:44.959 INFO:teuthology.orchestra.run.smithi155.stderr: 86.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T06:54:44.961 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 94.4% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T06:54:44.962 INFO:teuthology.orchestra.run.smithi155.stderr: 90.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T06:54:44.974 INFO:teuthology.orchestra.run.smithi143.stderr: 90.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T06:54:44.980 INFO:teuthology.orchestra.run.smithi022.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-24T06:54:45.030 INFO:teuthology.orchestra.run.smithi022.stderr: 83.7% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T06:54:45.054 INFO:teuthology.orchestra.run.smithi022.stderr: 90.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T06:54:45.063 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T06:54:45.079 INFO:teuthology.orchestra.run.smithi143.stderr: 83.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T06:54:45.080 INFO:teuthology.orchestra.run.smithi143.stderr: 90.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T06:54:45.083 INFO:teuthology.orchestra.run.smithi143.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T06:54:45.130 INFO:teuthology.orchestra.run.smithi022.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T06:54:45.131 INFO:teuthology.orchestra.run.smithi022.stderr: 90.0% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T06:54:45.208 INFO:teuthology.orchestra.run.smithi155.stderr: 90.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T06:54:45.258 INFO:teuthology.orchestra.run.smithi143.stderr: 95.4% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T06:54:45.309 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T06:54:45.352 INFO:teuthology.orchestra.run.smithi155.stderr: 94.4% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T06:54:45.421 INFO:teuthology.orchestra.run.smithi143.stderr: 95.0% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T06:54:45.736 INFO:teuthology.orchestra.run.smithi143.stderr: 91.8% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T06:54:45.742 INFO:teuthology.orchestra.run.smithi022.stderr: 95.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T06:54:45.820 INFO:teuthology.orchestra.run.smithi022.stderr: 90.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T06:54:45.895 INFO:teuthology.orchestra.run.smithi143.stderr: 95.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T06:54:46.247 INFO:teuthology.orchestra.run.smithi155.stderr: 91.7% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T06:54:46.394 INFO:teuthology.orchestra.run.smithi155.stderr: 94.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T06:54:46.738 INFO:teuthology.orchestra.run.smithi155.stderr: 95.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T06:54:47.255 INFO:teuthology.orchestra.run.smithi155.stderr: 95.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T06:54:48.262 INFO:teuthology.orchestra.run.smithi143.stderr: 94.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T06:54:49.518 INFO:teuthology.orchestra.run.smithi022.stderr: 90.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T06:54:58.787 INFO:teuthology.orchestra.run.smithi022.stderr: 93.9% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T06:55:41.975 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T06:56:05.425 INFO:teuthology.orchestra.run.smithi143.stderr: 92.8% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T06:56:25.135 INFO:teuthology.orchestra.run.smithi143.stderr: 92.7% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T06:56:29.313 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T06:56:40.560 INFO:teuthology.orchestra.run.smithi143.stderr: 92.3% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T06:56:40.562 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-24T06:56:40.562 INFO:teuthology.orchestra.run.smithi143.stderr:real 1m55.710s 2024-03-24T06:56:40.562 INFO:teuthology.orchestra.run.smithi143.stderr:user 3m54.164s 2024-03-24T06:56:40.562 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m7.513s 2024-03-24T06:56:48.234 INFO:teuthology.orchestra.run.smithi155.stderr: 92.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T06:57:01.207 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T06:57:39.590 INFO:teuthology.orchestra.run.smithi155.stderr: 92.1% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T06:57:39.608 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-24T06:57:39.608 INFO:teuthology.orchestra.run.smithi155.stderr:real 2m54.748s 2024-03-24T06:57:39.609 INFO:teuthology.orchestra.run.smithi155.stderr:user 4m25.647s 2024-03-24T06:57:39.609 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m8.718s 2024-03-24T06:58:28.368 INFO:teuthology.orchestra.run.smithi022.stderr: 92.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T06:58:56.303 INFO:teuthology.orchestra.run.smithi022.stderr: 92.5% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T06:59:02.862 INFO:teuthology.orchestra.run.smithi022.stderr: 92.3% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T06:59:34.774 INFO:teuthology.orchestra.run.smithi022.stderr: 92.2% -- replaced with /var/log/ceph/7bca48fe-e9a7-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T06:59:34.778 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T06:59:34.778 INFO:teuthology.orchestra.run.smithi022.stderr:real 4m49.904s 2024-03-24T06:59:34.779 INFO:teuthology.orchestra.run.smithi022.stderr:user 8m26.887s 2024-03-24T06:59:34.779 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m52.012s 2024-03-24T06:59:34.781 INFO:tasks.cephadm:Archiving logs... 2024-03-24T06:59:34.781 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi022/log 2024-03-24T06:59:34.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T07:00:20.794 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi143/log 2024-03-24T07:00:20.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T07:00:55.924 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi155/log 2024-03-24T07:00:55.925 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T07:01:35.653 INFO:tasks.cephadm:Removing cluster... 2024-03-24T07:01:35.654 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force 2024-03-24T07:01:36.108 INFO:teuthology.orchestra.run.smithi022.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T07:01:37.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force 2024-03-24T07:01:37.882 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T07:01:38.584 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7bca48fe-e9a7-11ee-95cd-87774f69a715 --force 2024-03-24T07:01:38.853 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 7bca48fe-e9a7-11ee-95cd-87774f69a715 2024-03-24T07:01:39.427 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T07:01:39.427 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T07:01:39.490 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T07:01:39.509 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T07:01:39.525 INFO:tasks.cephadm:Teardown complete 2024-03-24T07:01:39.525 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T07:01:39.537 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T07:01:39.538 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T07:01:39.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T07:01:39.552 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T07:01:39.708 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T07:01:39.718 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T07:01:39.719 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T07:01:39.726 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T07:01:39.729 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T07:01:39.742 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-24T07:01:39.743 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T07:01:39.773 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -59us[ -59us] +/- 84ms 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -461us[ -451us] +/- 63ms 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +555us[ +555us] +/- 69ms 2024-03-24T07:01:39.788 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T07:01:39.794 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T07:01:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T07:01:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -47us[ -43us] +/- 84ms 2024-03-24T07:01:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -455us[ -451us] +/- 63ms 2024-03-24T07:01:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +567us[ +571us] +/- 69ms 2024-03-24T07:01:39.795 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T07:01:39.897 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T07:01:39.897 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T07:01:39.898 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -92us[ -92us] +/- 84ms 2024-03-24T07:01:39.898 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -489us[ -493us] +/- 63ms 2024-03-24T07:01:39.898 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +528us[ +528us] +/- 69ms 2024-03-24T07:01:39.898 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T07:01:39.901 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T07:01:39.911 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T07:01:39.911 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T07:01:39.925 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T07:01:40.236 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T07:01:40.335 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T07:01:40.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T07:01:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:40.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T07:01:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:40.568 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T07:01:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:40.598 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T07:01:40.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T07:01:40.619 INFO:teuthology.task.internal:Duration was 2612.537346 seconds 2024-03-24T07:01:40.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T07:01:40.630 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T07:01:40.630 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T07:01:40.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T07:01:40.641 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T07:01:40.694 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T07:01:40.701 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T07:01:40.828 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T07:01:41.259 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T07:01:41.260 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T07:01:41.260 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T07:01:41.344 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T07:01:41.344 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T07:01:41.381 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-03-24T07:01:41.381 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T07:01:41.409 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T07:01:41.409 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T07:01:41.417 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T07:01:41.423 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T07:01:41.524 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T07:01:41.525 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T07:01:41.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T07:01:41.532 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T07:01:42.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T07:01:42.095 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T07:01:42.095 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T07:01:42.211 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T07:01:42.240 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T07:01:42.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T07:01:42.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.305 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.332 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-03-24T07:01:42.376 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-03-24T07:01:42.437 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-24T07:01:42.505 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:42.558 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:42.572 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T07:01:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T07:01:42.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T07:01:42.599 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T07:01:42.600 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi022 2024-03-24T07:01:42.600 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T07:01:42.721 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi143 2024-03-24T07:01:42.721 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T07:01:42.761 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026/remote/smithi155 2024-03-24T07:01:42.761 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T07:01:42.799 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T07:01:42.799 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T07:01:42.807 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T07:01:42.810 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T07:01:42.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T07:01:42.874 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T07:01:42.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T07:01:42.927 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T07:01:42.928 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T07:01:42.935 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T07:01:42.938 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T07:01:42.950 INFO:teuthology.orchestra.run.smithi143.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 07:01 /home/ubuntu/cephtest 2024-03-24T07:01:42.952 INFO:teuthology.orchestra.run.smithi155.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 07:01 /home/ubuntu/cephtest 2024-03-24T07:01:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 07:01 /home/ubuntu/cephtest 2024-03-24T07:01:42.990 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T07:01:43.002 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T07:01:43.026 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T07:01:43.053 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-24T07:01:43.053 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T07:01:43.077 DEBUG:teuthology.nuke:shortname: smithi143 2024-03-24T07:01:43.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T07:01:43.082 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.520772', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMeF8n/SZuNgyZPs9XNwUg7rkA6uJdvep8icuSigElRcFLKOlLNXoAVa5l2pMWfh7LEbC34ir7PUfGKJAkySGVE='} 2024-03-24T07:01:43.100 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-24T07:01:43.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T07:01:43.104 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.521899', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGL7KZ8QT6pbke7ylbxXfXTJCLGVuszRe9QoGfwhyP9exiv911wQH3qxyX9WFRNUBy3v+I2/yeHuwvgIuWg0ew='} 2024-03-24T07:01:43.126 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619026', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 06:04:56.518033', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvQFjcEQjn+KQcODW9amDgNmp77l705RsazjDnv+bCR0hwxl3qwLlMi++O/VweM8c8l6R31x24acrcOB7Vy3Lw='} 2024-03-24T07:01:43.136 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-24T07:01:43.136 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T07:01:43.159 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-24T07:01:43.160 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:43.174 INFO:teuthology.orchestra.console.smithi143:Power off 2024-03-24T07:01:43.175 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T07:01:43.187 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T07:01:43.197 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-03-24T07:01:43.197 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:43.214 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-24T07:01:43.214 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T07:01:43.222 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T07:01:43.235 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-24T07:01:43.235 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:43.258 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-24T07:01:47.189 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:47.223 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:47.259 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:47.301 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T07:01:47.334 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T07:01:47.368 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-24T07:01:51.306 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:51.335 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:51.369 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:51.416 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T07:01:51.444 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T07:01:51.477 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-24T07:01:55.418 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:55.445 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:55.478 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T07:01:55.529 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-24T07:01:55.554 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-03-24T07:01:55.587 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-24T07:01:55.629 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-24T07:01:55.655 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-03-24T07:01:55.688 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-24T07:01:55.795 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} duration: 2612.537346124649 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi022 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7e59b48e9ef042e19595b23984ec83cd status: fail success: false 2024-03-24T07:01:55.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T07:01:55.892 INFO:teuthology.run:FAIL